Product Quality and Documentation – Recent Developments H. K. Ramapriyan Assistant Project Manager ESDIS Project, Code 423, NASA GFSC 1 Sumer ESIP Meeting July 8, 2014

Motivation Scientists (Providers/Dataset Producers) are motivated to provide high quality products and have a stake in ensuring that their data are not misused Users need to know quality of data they use Many ways to express quality Makes it difficult for both providers and users Data Centers are intermediaries Need to simplify providers’ job of supplying information Express information conveniently for users to access and understand 2

Background QA4EO Guidelines (2010) NASA “Making Earth System Data Records (ESDRs) for User in Research Environments (MEaSUREs)” Product Quality Checklist (2012) NOAA Climate Data Records (CDR) Maturity Matrix (Bates and Privette, 2012) Improving Data Quality Information for NASA Earth Observation Data (Lynnes et al, 2012) Obs4MIPS – Climate Model Intercomparison Project (CMIP5) (2012) Committee on Earth Observing Satellites (CEOS) Essential Climate Variables (ECV) inventory questions (2012) National Center for Atmospheric Research (NCAR) Community Contributions Pages (2013) CEOS Working Group on Information Systems and Services (WGISS) Metadata Quality Exploration Questionnaire (2013) Global Earth Observation System of Systems (GEOSS) Data Quality Guidelines (2013) EU FP7 project CORE-CLIMAX assessment of European capacity to produce ECV climate data records from satellite, in situ and reanalysis data – NOAA maturity matrix extended/revised (2013) ISO – Metadata Standard for Geographic Information Data Quality (2013) 3

Product Quality Checklist Result of about 2 years’ discussions in Metrics Planning and Reporting Working Group – MEaSUREs PI’s and DAACs represented Distinction between “Scientific Data Quality” and “Product Quality” Two separate checklists created – one for PI’s and another for DAACs to fill out Recommendation made to HQ and approved Adopted and used for MEaSUREs 2006 projects Included in Cooperative Agreements for MEaSUREs 2012 projects 4

Product Quality Checklist – PI’s 5 Project ESDR / EDSR Group Checklist Science Quality Level 1. Have the data been evaluated by external users? (Summarize results) 2. Is the data set complete as proposed? (Explain 'partial'). 3. Is the data set consistently processed as proposed? (Explain 'partial'). 4. Are uncertainties estimated and documented, including in their spatial or temporal dimension? 5. Have the data been validated, i.e. ‘assessed for uncertainties, to the extent possible by comparison with alternative measurements’? 6. Have differences between new products and any comparable existing products been documented? (Explain how, in what ways) 7. Have promised improvements in the new data compared to existing products been achieved? 8. Have the ESDR’s algorithm or analysis method, product description and product evaluation results been published in peer-reviewed literature? Documentation Quality Level 1. Is the data format well and completely described and/or is a commonly accepted appropriate standard format used? 2. Has data format description been provided to the DAAC? 3. Are the algorithm and processing steps described? 4. Have algorithm and processing steps description been provided to DAAC? 5. Is the metadata complete? 6. Is the documentation of the metadata complete? 7. Has documentation of the metadata been provided to the DAAC? Usage and Satisfaction 1. If project is distributing products, is the targeted community using the data? (Indicate trend) 2. If project is distributing products, is the broader community using the data? (Indicate trend) 3. If project is distributing products, are users satisfied with the data product? (Indicate trend)

Product Quality Checklist – DAACs 6 DAAC ESDR / ESDR Group Checklist Science Quality Level 1. Have differences between new products and any comparable existing products been documented? (Summarize results) Documentation Quality Level 1. Is the data format well and completely described and/or is a commonly accepted appropriate standard format used? 2. Are the algorithm and processing steps described? 3. Is the metadata complete? 4. Is the documentation of the metadata complete? Accessibility / Support Services Quality 1. Is it easy for users to discover the data? 2. Is it easy for users to access the data? 3. Are tools and services that enable reading and use of the data readily available? 4. Are there existing tools for analysis of this data set? 5. Can the users get help with discovery, access and use of the data? Usage and Satisfaction 1. For products distributed by DAAC, is the targeted community using the data? 2. For products distributed by DAAC, is the broader community using the data? (Indicate Trend) 3. For products distributed by DAAC, are users satisfied with the data product? (Indicate Trend)

NCAR Climate Data Guide - Community Contributions Pages What are the key strengths of this data set? What are the key limitations of this data set? What are the typical research applications of these data? What are examples from your work? What are some common mistakes that users encounter when processing or interpreting these data? What are the likely spurious (non-climatic) features, if any, of time series derived from these data? What corrections were applied to account for changes in observing systems, sampling methods or density, and satellite drift or degradation? Describe any conversion steps that are necessary or general strategies to compare these data with model output. What are some comparable data sets, if any? Why use this data set instead of another? How is uncertainty characterized in these data? Provide a summary statement about these data and their utility for climate research and model evaluation. 7

CEOS WGISS Metadata Quality Exploration Questionnaire Why did you choose this dataset for the survey? How does your organization define “fitness for purpose” for this dataset? What quality measures do you use to assess scientific quality? How are quality measures created initially? How do you store quality measures in your metadata? Are uncertainties estimated and documented, including in their spatial or temporal dimension? Have the data been validated, i.e. ‘assessed for uncertainties’, to the extent possible by comparison with alternative measurements’? Have the algorithm or analysis method, product description and product evaluation results been published in peer- reviewed literature? Is the data evaluated by external users? If so, how are the comments from external users captured? Any other relevant comments regarding Quality metadata. 8