A Science Community Perspective Kerstin Lehnert Lamont-Doherty Earth Observatory Columbia University lehnert@ldeo.columbia.edu Thanks for helpful comments: Mark Ghiorso Ken Ferrier Al Hofmann Alexey Kaplan Roger Nielsen Mohan Ramamoorthy Tom Whittaker Data Quality A Science Community Perspective K. Lehnert, ESIP Panel on Data Quality 7/13/11

DQ & Science Science Technology Standards Norms Tools DQ is a shared norm which against the work of scholars is measured. Scientist produce data and use data that are the foundation of theories, hypothesis, and models. The quality of the data impacts the quality of the Before we can start thinking about the development and implementation of metadata schemas and protocols, science communities need to understand the norms Attacking large-scale problems in the Earth Sciences from climate change to convection of the Earth mantle requires scientists to use large datasets , many of which need to be compiled K. Lehnert, ESIP Panel on Data Quality 7/13/11

The Social Side of DQ “The reliability of knowledge about climate change depends on the commensurability of data in space and time.” From Paul N. Edwards: "A Vast Machine": Standards as Social Technology Science, vol. 304, 2004 DOI: 10.1126/science.1099290 Yet even these changes have not eliminated the importance of disciplined human beings for the successful implementation of standards. For example, in the late 1980s, the U.S. Weather Service replaced liquid-in-glass thermometers with digital electronic ones at thousands of stations in its Cooperative Station Network. The new thermometers displayed Fahrenheit in units of 0.1°. Observers were meant to round off the readings to the nearest degree, but about 10% of observers simply entered the entire figure. Many who did round off probably did so incorrectly (6). Furthermore, the new, more accurate instruments did not correlate exactly with the old ones. Network-wide, the new instrumentation altered the mean daily temperature range by −0.7°C and the average daily temperature by −0.1°C compared with the previous system (7). This example illustrates the complex combination of social and technical problems that affect the implementation of standards. The consequences for the detection of climatic change can be profound: The biases discovered in the U.S. Cooperative Station Network, although correctable, “are of the same magnitude as the changes of global and United States mean temperatures since the turn of the 20th century” (6). Matthew Maury's 1858 diagram of the global atmospheric circulation. K. Lehnert, ESIP Panel on Data Quality 7/13/11

Earth Science Data Diversity of data, many disciplines, mostly observational data and model outputs Big data from sensors and sensor networks, standardized data acquisition, scientist is not the producer of raw data, but synthesizes and generates derived products Small data that are generated in the lab or in the field, sometimes not even numerical, acquired by a wide range of methods, often personalized K. Lehnert, ESIP Panel on Data Quality 7/13/11

Error Budgets http://www.ssterrorbudget.org/ISSTST/White_Paper.html 7/13/11 K. Lehnert, ESIP Panel on Data Quality Error Budgets Diagram from White Paper on the SST Error Budget, produced by the U.S. SST Science Team http://www.ssterrorbudget.org/ISSTST/White_Paper.html

DQ: Instrument Errors “Most of the rapid decrease in globally integrated upper (0–750 m) ocean heat content anomalies (OHCA) between 2003 and 2005 reported by Lyman et al. [2006] appears to be an artifact resulting from the combination of two different instrument biases recently discovered in the in situ profile data.” K. Lehnert, ESIP Panel on Data Quality 7/13/11

DQ: Precision “Mantle Myths, Reservoirs, and Databases” Presentation by A. Hofmann at the Goldschmidt Conference 2008 K. Lehnert, ESIP Panel on Data Quality 7/13/11

What Defines DQ? “Knowing that I can trust the numbers.” “Data having an uncertainty that actually corresponds to the uncertainty stated in the the source.” “In one word, ‘completeness’.” (allows others to assess the validity of data, because then you can check for standards used, techniques, reproducibility, etc. Reproducibility, precision, … K. Lehnert, ESIP Panel on Data Quality 7/13/11

How Do You Evaluate DQ? ‘Analytical completeness’, including uncertainties, and metadata. Statistical tests, internal consistency. Rely on reputation of the investigator, either directly or by association. “Well, usually I don't, because that's a lot of work.” K. Lehnert, ESIP Panel on Data Quality 7/13/11

DQ Needs Carrots & Sticks Tools for DQ metadata management, e.g. capture during data acquisition Software for using DQ metadata in data analysis, synthesis, modeling Policies for and enforcement of data & metadata reporting Peer-review of data K. Lehnert, ESIP Panel on Data Quality 7/13/11

Data Publication Publication of data in repositories QC/QA at repository (completeness, consistency) Open Access Long-term archiving Link to scientific articles via unique identifiers Support for investigators to comply with agency policies K. Lehnert, ESIP Panel on Data Quality 7/13/11

Conclusions (I): Science Community Needs to define the disciplinary norms for DQ measures Needs to drive the implementation of disciplinary standards Policies for data reporting & publication Recommendations for data acquisition K. Lehnert, ESIP Panel on Data Quality 7/13/11

Conclusions (II): Technology Needs to translate disciplinary standards to technical standards Needs to provide software tools that facilitate DQ management (capture, communication, & assessment) K. Lehnert, ESIP Panel on Data Quality 7/13/11

Conclusion (III) Science and technology need to work closely to develop meaningful solutions for DQ management. The process needs to take into account the diversity of Earth Science disciplines and data types. K. Lehnert, ESIP Panel on Data Quality 7/13/11