A Data Quality Screening Service for Remote Sensing Data Christopher Lynnes, NASA/GSFC (P.I.) Edward Olsen, NASA/JPL Peter Fox, RPI Bruce Vollmer, NASA/GSFC Robert Wolfe, NASA/GSFC Contributions from R. Strub, T. Hearty, Y-I Won, M. Hegde, S. Zednik, P. West, N. Most, S. Ahmad, C. Praderas, K. Horrocks, I. Tchered, and A. Rezaiyan-Nojani Advancing Collaborative Connections for Earth System Science (ACCESS) Program Project Page:

Outline Why a Data Quality Screening Service? Making Quality Information Easier to Use via the Data Quality Screening Service (DQSS) DEMO DQSS Under the Hood DQSS Status DQSS Plans 2

Why a Data Quality Screening Service? 3

The quality of data can vary considerablyThe quality of data can vary considerably AIRS VariableBest (%) Good (%) Do Not Use (%) Total Precipitable Water38 24 Carbon Monoxide64729 Surface Temperature54451 Version 5 Level 2 Standard Retrieval Statistics 4

Quality schemes can be relatively simple…Quality schemes can be relatively simple… Total Column Precipitable WaterQual_H2O BestGoodDo Not Use kg/m 2 5

…or they can be more complicated…or they can be more complicated Hurricane Ike, viewed by the Atmospheric Infrared Sounder (AIRS) PBest : Maximum pressure for which quality value is Best in temperature profiles Air Temperature at 300 mbar 6

Quality flags are also sometimes packed together into bytes Cloud Mask Status Flag 0=Undetermined 1=Determined Cloud Mask Cloudiness Flag 0=Confident cloudy 1=Probably cloudy 2=Probably clear 3=Confident clear Day / Night Flag 0=Night 1=Day Sunglint Flag 0=Yes 1=No Snow/ Ice Flag 0=Yes 1=No Surface Type Flag 0=Ocean, deep lake/river 1=Coast, shallow lake/river 2=Desert 3=Land Big-endian arrangement for the Cloud_Mask_SDS variable in atmospheric products from Moderate Resolution Imaging Spectroradiometer (MODIS) 7

Current user scenarios...Current user scenarios... Nominal scenario Search for and download data Locate documentation on handling quality Read & understand documentation on quality Write custom routine to filter out bad pixels Equally likely scenario ( especially in user communities not familiar with satellite data ) Search for and download data Assume that quality has a negligible effect Repeat for each user 8

The effect of bad quality data is often not negligible Total Column Precipitable Water Quality BestGood Do Not Use kg/m 2 Hurricane Ike, 9/10/2008 9

Neglecting quality may introduce bias (a more subtle effect) AIRS Relative Humidity Comparison against Dropsonde with and without Applying PBest Quality Flag Filtering Boxed data points indicate AIRS RH data with dry bias > 20% From a study by Sun Wong (JPL) on specific humidity in the Atlantic Main Development Region for Tropical Storms 10

Percent of Biased Data in MODIS Aerosols Over Land Increases as Confidence Flag Decreases *Compliant data are within Aeronet Statistics derived from Hyer, E., J. Reid, and J. Zhang, 2010, An over-land aerosol optical depth data set for data assimilation by filtering, correction, and aggregation of MODIS Collection 5 optical depth retrievals, Atmos. Meas. Tech. Discuss., 3, 4091–

Making Quality Information Easier to Use via the Data Quality Screening Service (DQSS). 12

DQSS TeamDQSS Team P.I.: Christopher Lynnes Software Implementation: Goddard Earth Sciences Data and Information Services Center Implementation: Richard Strub Local Domain Experts (AIRS): Thomas Hearty and Bruce Vollmer AIRS Domain Expert: Edward Olsen, AIRS/JPL MODIS Implementation Implementation: Neal Most, Ali Rezaiyan, Cid Praderas, Karen Horrocks, Ivan Tchered Domain Experts: Robert Wolfe and Suraiya Ahmad Semantic Engineering: Tetherless World RPI Peter Fox, Stephan Zednik, Patrick West 13

The DQSS filters out bad pixels for the user Default user scenario Search for data Select science team recommendation for quality screening (filtering) Download screened data More advanced scenario Search for data Select custom quality screening parameters Download screened data 14

DQSS replaces bad-quality pixels with fill values Mask based on user criteria (Quality level < 2) Good quality data pixels retained Output file has the same format and structure as the input file (except for extra mask and original_data fields) Original data array (Total column precipitable water) 15

Visualizations help users see the effect of different quality filtersVisualizations help users see the effect of different quality filters 16

DQSS can encode the science team recommendations on quality screening AIRS Level 2 Standard Product Use only Best for data assimilation uses Use Best+Good for climatic studies MODIS Aerosols Use only VeryGood (highest value) over land Use Marginal+Good+VeryGood over ocean 17

Initial settings are based on Science Team recommendation. (Note: Good retains retrievals that Good or better). You can choose settings for all parameters at once......or variable by variable Or, users can select their own criteria...Or, users can select their own criteria... 18

DEMO (Search for AIRX2RET) 19

DQSS Under the HoodDQSS Under the Hood 20

DQSS FlowDQSS Flow user selection ancillary file Screener Quality Ontology data file quality mask screened data file End User data file w/ mask Masker Ontology Query 21

DQSS Ontology (The Whole Enchilada) 22

DQSS Ontology (Zoom)DQSS Ontology (Zoom) 23

DQSS StatusDQSS Status 24

Significant AccomplishmentsSignificant Accomplishments Release 1.0 of DQSS for AIRS Level 2 Standard Retrieval Technology Readiness Level (TRL) = 9 (for AIRS L2) Includes Quality Impact views Disclosure of Invention (NF 1679) filed Announced to AIRS Registered Data Product User Community (~770) Papers and Presentations Managing Data Quality for Collaborative Science workshop (peer- reviewed paper) Sounder Science meeting ESDSWG Poster A-Train User Workshop (part of GES DISC presentation) AGU: Ambiguity of Data Quality in Remote Sensing Data Ontology (v. 2.4) to accommodate both AIRS and MODIS L2 25

Current StatusCurrent Status DQSS is operational for AIRS L2 Standard Products DQSS is offered through the Mirador data search interface* at the GES DISC DQSS has been refactored to work in LAADS/MODAPS environment 2 1 / 2 month delay due to refactoring and (successful) audit of NPR compliance by NASAs Office of the Chief Engineer Schedule no longer has room for client-side screening But maybe...(to be continued) * 26

Metrics DQSS is included in web access logs sent to EOSDIS Metrics System (EMS) Tagged with Protocol DQSS EMS -> ESDS bridge implemented Metrics from EMS: Users: 50 Downloads:

DQSS RefactoringDQSS Refactoring GES DISCMODAPS/LAADS Synchronous vs. Asynchronous SynchronousAsynchronous batch post-process Screening machines Archive/Distributio n Processing minions (no outward connections allowed) 28

Refactored DQSS encapsulates ontology and screening parameters MODAPS Minions LAAD SWeb GUI DQSS Ontology Service data product screening options MODAPS Database screening parameters (XML) selected options screening parameters (XML) screening parameters (XML) Encapsulation enables reuse in other data centers with diverse environments. 29

DQSS PlansDQSS Plans 30

2010 Plans2010 Plans Adding more products MODIS L2 Water Vapor – 2/2011 MODIS L2 Clouds and Aerosols – 5/2011 Microwave Limb Sounder (MLS) L2 – 6/2011 Ozone Monitoring Instrument (OMI) L2 – 8/2011 Integrate with other services AIRS Variable Subsetting & NetCDF Reformatting – 3/2011 OPeNDAP – 6/2011 MODAPS Web Services – 10/11 Refactor for reusability & sustainability – 9/2011 Release – 11/2011 No longer planned: client-side screening Except...refactored DQSS may make this possible... 31

Combine DQSS with Other ServicesCombine DQSS with Other Services AIRS Subsetting, quality screening and reformatting to netCDF are currently mutually exclusive Combining them will also raise awareness of quality screening DQSS will also be more usable to communities more comfortable with netCDF (e.g., modeling community) OPeNDAP OPeNDAP, Inc. is working to support back-end calls to REST services Gateway is based on ACCESS-05 project (CEOP Satellite Data Server) Will allow access to DQSS via analysis and visualization clients (e.g., Panoply, IDV) May be reused for other back-end REST services 32

Refactoring for ReleaseRefactoring for Release Proposal schedule included refactoring for long-term sustainability A key element of successful technology infusion Current refactoring for MODIS environment... Gives us a head start May also give us an avenue to client-side capabilities 33

New Client Side ConceptNew Client Side Concept Ontology Web Form Data Provider Masker Screener dataset-specific instance info data files screening criteria XML File data files GES DISCEnd User 34

DQSS RecapDQSS Recap Screening satellite data can be difficult and time consuming for users The Data Quality Screening System provides an easy-to- use service The result should be: More attention to quality on users part More accurate handling of quality information… …With less user effort 35

Backup SlidesBackup Slides 36

DQSS Target Uses and UsersDQSS Target Uses and Users Routine Visual- ization Quick Recon. Machine- levelMetrics InterdisciplinaryXX EducationalXX Expert/Power? XX Applications X Algorithm Developers X 37

ESDSWG ActivitiesESDSWG Activities Contributions to all 3 TIWG subgroups Semantic Web: Contributed Use Case for Semantic Web ESIP Participation in ESIP Information Quality Cluster Services Interoperability and Orchestration Combining ESIP OpenSearch with servicecasting and datacasting standards Spearheading ESIP Federated OpenSearch cluster 5 servers (2 more soon); 4+ clients Now ESIP Discover cluster (see above) Processes and Strategies: Developed Use Cases for Decadal Survey Missions: DESDynI-Lidar, SMAP, and CLARREO 38

Projected TRL BreakdownProjected TRL Breakdown 39