Virtual Quality Screening Service (VQSS): Improving the application of quality information NASA funded Advancing Collaborative Connections for Earth System Science (ACCESS) Project Ed Armstrong, Thomas Huang, Zhangfan Xing, Christian Alacron, Toshio Chin, NASA JPL Siri Jodha Khalsa, NSIDC

 Optimal use of satellite-based earth science data records requires access to and understanding of the data quality information contained in those records. This can be a complex and time-consuming process, with metadata attributes, bit flags, ancillary variables all needed, possibly in combination, to ensure that the data meets scientific requirements. For example, quality screening of Level 3 data from the upcoming Soil Moisture Active Passive (SMAP) instrument can involve up to 26 unique bit states or conditions a user can filter for. For GHRSST L2P granules a minimum of 10 variables can be used to screen for SST data on a pixel-by-pixel basis. VQSS Introduction and Motivation

 The Virtualized Quality Screening Service (VQSS), a recently funded 2014 NASA ACCESS project, aims to address these issues and concerns by developing an infrastructure that will allow users to view and apply the quality information in SMAP and other products from the Group for High Resolution Sea Surface Temperature (GHRSST) Project. It leverages proven NASA components for data extraction, subsetting by value, and visualization using granule-based quality information.  Builds on existing PO.DAAC web services for data (and metadata) discovery, subsetting and extraction, and visualization.  See PO.DAAC web services homepage  VQSS.…

Implement web services and related infrastructure for quality screening of GHRSST and SMAP L2/L3 data. Allow users to explore meaning and effect of quality variables. Share and store URLs to explicitly extract and screen geophysical data from granules. How?  Expose data granules from GHRSST/SMAP mission through webification  Extend the PO.DAAC web services, also known as the EDGE as an external data service for granule searching  Provide a portal for the public exposure of VirtualQSS that will allows users to:  Search for SMAP granules using spatial-temporal constraints  Review quality,error estimates and others ancillary variables and information specific to a unique data type exposed through a semantic layer or lexicon  Apply quality flag filtering based on exact user specifications to granules  Subset and return filtered results as single granules and/or aggregates in preferred formats such as netCDF, HDF, JSON, and CSV  Store and share filtering queries with other scientists and the community Objectives of VirtualQSS

 A minimum 10 variables in a GHRSST granule could potentially be used to filter SST observations. More if additional experimental variables are provided.  quality_level (scalar flags)  l2p_flags (bit flags for various conditions)  wind (physical variable)  sses_bias (error statistic: bias in degC)  sses_sd (error statistic: standard deviation)  sea_ice_fraction (physical variable)  aerosol (physical variable)  dt_analysis (anomaly SST )  satellite_zenith_angle (instrument geometry)  solar_irradiance (physical variable)  …other experimental variables like diurnal warming SST/CHL_A/K490/Brightness Temperature depending on specific dataset variables GHRSST “quality” variables for L2P data

 Examples:  short l2p_flags(time, nj, ni); l2p_flags:flag_meanings = "microwave land ice lake river reserved_for_future_use no_retrieval N2_retrieval N3R_retrieval N3_retrieval D2_retrieval D3_retrieval cloud sun_glint cosmetic_fill l2p_flags:flag_masks = 1s, 2s, 4s, 8s, 16s, 32s, 64s, 128s, 256s, 512s, 1024s, 2048s, 4096s, 8192s, 16384s, 32768s ;  byte quality_level (time, nj, ni); quality_level:flag_meanings = "no_data bad_data worst_quality low_quality acceptable_quality best_quality"; quality_level:flag_values = 0b, 1b, 2b, 3b, 4b, 5b; CF metadata for describing quality

Example L3 flags for SMAP Also contains multiple variables for quality screening. Mostly in the form of bit flags. 26 unique bit states or conditions a user can filter for L3 data.

 Webification (w10n) is an enabling technology that simplifies use of large and complex science data, such as ones archived at PO.DAAC, using HTTP/HTTPS protocols with URLs comprised of well-defined parameters. Similar to OPeNDAP.  W10n abstracts an arbitrary data store as a tree, in which two types of entities exist: node and leaf.  Direct access to inner components of the node/leaf is via HTTP requests from either a web browser, script or similar client. Results of W10n calls return specified measurement arrays or metadata elements via subset by array value (v.s. subset by array index), according to supported output formats (JSON, HTML, netCDF) as specified in the URL request. Webification

More W10n SST query examples Example of webification requests for a MODIS Terra sea surface temperature granule. T L2_LAC_GHRSST_N- v01.nc.bz2/sea_surface_temperature[-130<lon<- 120,35<lat<45]?output=json T L2_LAC_GHRSST_N- v01.nc.bz2/sea_surface_temperature[quality_flag>=4]?output= nc T L2_LAC_GHRSST_N- v01.nc.bz2/sea_surface_temperature[quality_flag>=4,wind_spe ed>6,-130<lon<- 120,35<lat<45]?output=nc.4 Final request is for spatial subsetting with a specific quality flag levels chosen and a wind threshold of 6 m/s selected. Output is subsetted netCDF4.

Additional Webification visualization capabilities

VQSS Architecture Architecture of VQSS system. Webification is linked directly to the JPL PO.DAAC to serve its granules. Other data sources such as SMAP from NSIDC are served via “proxy.” VQSS will leverage and extend a granule and metadata discovery service at the PO.DAAC called EDGE to expose additional quality attributes and metadata for SMAP. A user interface will be developed to expose all these services to the science community.

Prototype Search Interface

 Allow users to search for relevant granules (e.g., GHRSST granules from a specific sensor, or soil moisture datasets from SMAP)  Expose quality information for each data type  Provide a mechanism to abstract the quality screening/subset request  Federate and balance w10n screening requests  Access returned results from the w10n servers in preferred output formats  Provide a method to save queries for future reference or sharing  Visualize results Portal

 OceanXtremes: Oceanographic Data-Intensive Anomaly Detection and Analytics Portal  PI: Thomas Huang/JPL, Co-I: Brian Wilson/JPL, George Chang/JPL, Ed Armstrong/JPL, and Toshio Chin/JPL  Mining and Utilizing Earth Science Dataset Metadata, Usage Metrics, and User Feedback to Improve Dataset Relevancy  PI: Chaowei (Phil) Yang/GMU, Co-I: Ed Armstrong/JPL, Thomas Huang/JPL, and David Moroni/JPL  A Service to Match Satellite and In-situ Marine Observations to Support Platform Intercomparisons, Cross-calibration, Validation, and Quality Control  PI: Shawn Smith/COAPS, Co-I: Thomas Huang/JPL, Vardis Tsontos/JPL, Ben Holt/JPL, Mark Bourassa/COAPS, and Steve Worley/NCAR 2014 NASA Advanced Information Systems Technology (AIST) proposals

 Working with large volume satellite data (Big Data) presents complex challenges with accessing relevant research information. This issue is only getting worse.  New tools and services are needed to discover, subset, extract, and visualize smaller volumes  The Virtual Quality Screening Service will offer a key contribution in this web services paradigm by providing the ability to apply any quality or ancillary variables as filters to the physical variable of interest  To be deployed on GHRSST and SMAP L2/L3/L4 granules in 2015 Summary

 Goal: Expose quality information in a user friendly fashion  OCSI search results return quality parameters  Build a lexicon  For example, user floats over variable or metadata attribute detailed information will be exposed  Menu choices of quality variables or quality levels  Ontology development  Formal dataset specific ontology linking variables with quality concepts including CF and ISO  May make it easier to extend to other datasets beyond GHRSST/SMAP Quality information exposure

 PO.DAAC SST data  GHRSST:  L2P MODIS_A, MODIS_T, VIIRS, AVHRR, AMSR-E, AMSR2  L3C AMSR2, AVHRR  L4 Global  NSIDC SMAP data  L2/3 Active, Passive and A/P datasets Future Integration plans

 Year 1:  Implement local w10n server for all GHRSST datasets residing in the PO.DAAC  Develop quality parameter lexicon adaptation for GHRSST data/metadata model  Enhance OCSI interface to include quality parameters and delegate to use ECHO for SMAP test data  Create W10n proxy implementation of NSDIC OPeNDAP servers.  Year 2:  Implement services in beta version of web portal.  Extend quality lexicon, granule searching, data filtering to support operational SMAP data.  Test and finalize portal.  Deploy portal and integrate with PO.DAAC Labs. Package software. Engage community for testing.  Package webification and other software for open source Milestones