Open data and data curation 11 Open data and data curation Hamish James Statistics New Zealand Loss of knowledge about the dataset is the key problem we face. Datasets are easy to break apart or put together. It is not enough to be able to find a dataset, you also need to be able to drill down into the dataset and examine individual records or variables. This number means nothing without context. For datasets, that context is provided by creating metadata for the dataset

Outline Setting the scene Open data How open data and data curation are related

Quick definitions data open data data curation information structured digital analogue unstructured data open data data curation

Defining data 14. This definition of data places no restrictions on the topics data may refer to, or on the purposes towards which data may be used. Data may be collected about anything. Some examples of data include sets of measurements captured by scientific instruments, responses extracted from questionnaires and forms, and structured catalogue entries about physical or digital objects, such as books and artworks. The broad scope of what may be treated as data is also expanding as modern information technology and management techniques provide ways of creating structure in previously unstructured information sources. With such a wide definition of data, readers should bear in mind that their understanding of the term may be different from that of others. 4 Data consists of sets of structured values that can be organised, analysed and manipulated by a software application or some other means of calculation. This includes data collected directly through surveys and administrative systems, as well as data created or compiled by aggregating or reanalysing other sources. A defining characteristic of data is that it is machine-readable.

Open data, data curation Open data is a philosophy based on the idea that that data is more valuable if more people can use it, and that technology has made the cost of sharing data negligble Data curation is a field of research and work focusing on the long-term management of data, built on the argument that the opportunity cost of losing data is high Open data highlights benefits Data curation worries about costs

data knowledge value

Focus of open data activities Data collected and held by governments Data collected or generated through publically funded research http://wiki.opengovdata.org/index.php?title=Open DataPrinciples

Reasons to make data open The underlying purposes of making publically funded data more accessible are to: inform decision making by government, businesses and communities increase transparency and accountability in government decision making assist informed participation by the public in government decision making promote economic development through the innovate application of data collected for one purpose to other tasks gain greater value from research data

Barriers to reuse of government data Agency culture (reluctance or hostility to data sharing) Funding constraints Ensuring data confidentiality Shared ownership Poor dissemination practices Last year we did what government departments do best, we produced a paper! Barrier to reuse of government held data The paper drew on a review of published international and national commentary and evidence along with a small number of New Zealand government agencies that are already making data available for reuse. Government agencies may be disinterested in, or actively opposed to the release of information and data. Although most evidence relates to access to official information or freedom of information laws, it seems likely that a culture of restricting access to information will also encompass restricting access to data. Data reuse initiatives can require significant funding. The costs of developing and maintaining data discovery and access systems are much easier to measure than the benefits that flow from data reuse, and this may make it difficult to construct strong business cases for investment. Concerns that allowing data reuse may breach privacy or confidentiality can make agencies reluctant to allow data to be reused. General legislation, such as the Privacy Act, and specific legislation, such as the Statistics Act, imposes requirements on government agencies to protect data. A lack of attention to how data is disseminated can greatly complicate reuse of the data. The adoption of standard machine-readable formats, consistent discovery metadata, and simple licensing arrangements will make it easier for others to reuse data. Barriers to the reuse of data can emerge where government agencies do not have sole ownership of the data they wish to make available. Data that is produced as a result of a public-private partnership can be difficult to make publicly available for re-use because of the potential threat to the commercial interests of the private partner. Data may have direct sale value, or may embody intellectual property of value. Maori could view a range of government-held datasets as holding customary knowledge and expect access to be restricted.

Open Government Data Principles Government data shall be considered open if it is made public in a way that complies with the principles below: Complete. All public data is made available. Public data is data that is not subject to valid privacy, security or privilege limitations. Primary. Data is as collected at the source, with the highest possible level of granularity, not in aggregate or modified forms. Timely. Data is made available as quickly as necessary to preserve the value of the data. Accessible. Data is available to the widest range of users for the widest range of purposes. Machine processable. Data is reasonably structured to allow automated processing. Non-discriminatory. Data is available to anyone, with no requirement of registration. Non-proprietary. Data is available in a format over which no entity has exclusive control. License-free. Data is not subject to any copyright, patent, trademark or trade secret regulation. Reasonable privacy, security and privilege restrictions may be allowed. OECD Principles and Guidelines for Access to Research Data from Public Funding

Characteristics of open data Free and open access to the data Freedom to redistribute the data Freedom to reuse the data No restriction of the above based on who someone is (e.g. their nationality) or their field of endeavour (e.g. commercial or non-commercial) c.f. http://www.okfn.org/about/

Creative Commons licence conditions Attribution Share-alike No derivative works NZGOAL (New Zealand Government Open Access and Licensing Framework) advocates the use of Creative Commons licences Non-commercial Creative Commons

Linked data Linked data uses semantic web approaches (especially RDF) to describe data and make it accessible to machines – a web of linked data RDF ‘triples’ are used to describe things Subject – predicate – object Hamish – is a – presenter

Linking Open Data dataset cloud

What is missing?

46 Loss of knowledge about the dataset is the key problem we face. 1717 46 Loss of knowledge about the dataset is the key problem we face. Datasets are easy to break apart or put together. It is not enough to be able to find a dataset, you also need to be able to drill down into the dataset and examine individual records or variables. This number means nothing without context. For datasets, that context is provided by creating metadata for the dataset

Data needs context 1818 Simple examples of metadata for a data value. The loss of one pieces of this metadata could render the data useless.

Examples “Which town or city in the UK has the highest proportion of students?" “Which town or city in the UK is home to one or more university campuses whose registered full or part time (non-distance) students divided by the local population gives the largest percentage?” http://digitalcuration.blogspot.com/2010/03/link ed-data-and-reality.html

re/use render explain Technology: Documentation: Hardware Standards Formats Software Documentation: Standards Meaning Interpretation

data knowledge value Technology to render data Documentation to explain

What is missing? Context Data is not self-describing Who provides the description? What does it cost to provide the description? How much of the description is held as tacit knowledge? Expert’s personal knowledge Rules and meaning encoded into the data and software

I'm not sure exactly what it was that you are thinking of I'm not sure exactly what it was that you are thinking of? However here are just a few cases where I had to do some extra work when the information was not available: Firstly in the Ag Economic Survey I had to trawl through correspondence in off-site boxes to 'cement' together a list of the codes and categories that related to the pre-1989 local govt areas used in the valuation rolls which was not in cars. (that took about 10 - 15 hours work spread over weeks. Problem was often you had parts of lists but not a list where you had codes against descriptions. At least this information was able to be used for the older Ag Production later. Farmtypes lists at the front of the Ag Production publications in the 1970's and 80' were not always what was in the actual data or published tables but by doing SAS frequency analysis I reconciled totals against the published tables to confirm the codes. Similar analysis was done for the breeds data some years against published tables. The Agriculture Production (Annual Census of Farms Year Ended 30 June 1979) study had no questionnaire or other proxy for linecodes, so we did 1980 and 1978 before and then did frequency analysis on all three and confidently confirmed most linecodes using published tables and the straddling years for reality checks. This took about 20 hours work just to do this. Then the work was as per other similar ingests. General comment is that when you are not familiar with data it takes a lot of analysis to try and 'validate' linecode and category descriptions. After some investment it gets easier after the initial investment of time if there are common variables across years but then suddenly they all change and you have to slow down and get familiar with a new set of linecodes. I have attached 1979 DDI for now.Let me know if you need anymore, cheers (See attached file: 5081_MD.xml)

Data curation Data curation involves: = open data = data curation Data management Adding value to data Data sharing for re-use Data preservation for later re-use http://www.dcc.ac.uk/news/what-makes-data-curation = open data = data curation

Digital Curation Centre

DDI Alliance

Open data brings benefits and risks more users highlights data curation failures justifies data curation costs pressure for more user support expands expert community increases risk of poor analysis

Complementary ideas Actively curated data will: Remain technologically accessible Be easier to understand (and therefore use) Data curation will benefit from data being made more open: Data that is in active use tends to remain usable Widely used data is better understood than isolated data

Thank you Contact details 29292929 Hamish James Manager, Information Management hamish.james@stats.govt.nz 04 931 4237