A Data Quality Screening Service for Remote Sensing Data Christopher Lynnes, NASA/GSFC (P.I.) Edward Olsen, NASA/JPL Peter Fox, RPI Bruce Vollmer, NASA/GSFC.

Slides:



Advertisements
Similar presentations
Chapter 8 Software Prototyping.
Advertisements

Copyright © 2003 Pearson Education, Inc. Slide 1 Computer Systems Organization & Architecture Chapters 8-12 John D. Carpinelli.
Copyright © 2003 Pearson Education, Inc. Slide 7-1 Created by Cheryl M. Hughes The Web Wizards Guide to XML by Cheryl M. Hughes.
McGraw-Hill/Irwin Copyright © 2013 by The McGraw-Hill Companies, Inc. All rights reserved. Extended Learning Module D (Office 2007 Version) Decision Analysis.
Copyright © 2011, Elsevier Inc. All rights reserved. Chapter 6 Author: Julia Richards and R. Scott Hawley.
Author: Julia Richards and R. Scott Hawley
1 Copyright © 2013 Elsevier Inc. All rights reserved. Appendix 01.
Properties Use, share, or modify this drill on mathematic properties. There is too much material for a single class, so you’ll have to select for your.
UNITED NATIONS Shipment Details Report – January 2006.
By Rick Clements Software Testing 101 By Rick Clements
1 Hyades Command Routing Message flow and data translation.
18 Copyright © 2005, Oracle. All rights reserved. Distributing Modular Applications: Introduction to Web Services.
1 State Wildlife Action Plans Wiki: Business Transformation Tutorial Brand Niemann July 5, 2008
1 RA I Sub-Regional Training Seminar on CLIMAT&CLIMAT TEMP Reporting Casablanca, Morocco, 20 – 22 December 2005 Status of observing programmes in RA I.
Microsoft Access 2007 Advanced Level. © Cheltenham Courseware Pty. Ltd. Slide No 2 Forms Customisation.
Create an Application Title 1A - Adult Chapter 3.
Custom Statutory Programs Chapter 3. Customary Statutory Programs and Titles 3-2 Objectives Add Local Statutory Programs Create Customer Application For.
What gas makes up 78% of our atmosphere?
1 Click here to End Presentation Software: Installation and Updates Internet Download CD release NACIS Updates.
1 NatQuery 3/05 An End-User Perspective On Using NatQuery To Extract Data From ADABAS Presented by Treehouse Software, Inc.
Photo Slideshow Instructions (delete before presenting or this page will show when slideshow loops) 1.Set PowerPoint to work in Outline. View/Normal click.
Course Objectives After completing this course, you should be able to:
Software testing.
PP Test Review Sections 6-1 to 6-6
EU market situation for eggs and poultry Management Committee 20 October 2011.
PEPS Weekly Data Extracts User Guide September 2006.
© Paradigm Publishing, Inc Access 2010 Level 1 Unit 1Creating Tables and Queries Chapter 2Creating Relationships between Tables.
EIS Bridge Tool and Staging Tables September 1, 2009 Instructor: Way Poteat Slide: 1.
COMPUTER INTERFACES.
Copyright © 2012, Elsevier Inc. All rights Reserved. 1 Chapter 7 Modeling Structure with Blocks.
1 RA III - Regional Training Seminar on CLIMAT&CLIMAT TEMP Reporting Buenos Aires, Argentina, 25 – 27 October 2006 Status of observing programmes in RA.
Basel-ICU-Journal Challenge18/20/ Basel-ICU-Journal Challenge8/20/2014.
1..
CONTROL VISION Set-up. Step 1 Step 2 Step 3 Step 5 Step 4.
1 © 2004, Cisco Systems, Inc. All rights reserved. CCNA 1 v3.1 Module 10 Routing Fundamentals and Subnets.
SLP – Endless Possibilities What can SLP do for your school? Everything you need to know about SLP – past, present and future.
1 How Do I Order From.decimal? Rev 05/04/09 This instructional training document may be updated at anytime. Please visit and check the.
Macromedia Dreamweaver MX 2004 – Design Professional Dreamweaver GETTING STARTED WITH.
Model and Relationships 6 M 1 M M M M M M M M M M M M M M M M
1 Using one or more of your senses to gather information.
Analyzing Genes and Genomes
To the Assignments – Work in Progress Online Training Course
©Brooks/Cole, 2001 Chapter 12 Derived Types-- Enumerated, Structure and Union.
Chapter 12 Working with Forms Principles of Web Design, 4 th Edition.
Essential Cell Biology
Intracellular Compartments and Transport
PSSA Preparation.
Chapter 11 Creating Framed Layouts Principles of Web Design, 4 th Edition.
Essential Cell Biology
Earth Science Data: Why So Chris Lynnes.
Energy Generation in Mitochondria and Chlorplasts
Murach’s OS/390 and z/OS JCLChapter 16, Slide 1 © 2002, Mike Murach & Associates, Inc.
© Paradigm Publishing, Inc Access 2010 Level 2 Unit 2Advanced Reports, Access Tools, and Customizing Access Chapter 8Integrating Access Data.
Introduction to ikhlas ikhlas is an affordable and effective Online Accounting Solution that is currently available in Brunei.
Introduction Peter Dolog dolog [at] cs [dot] aau [dot] dk Intelligent Web and Information Systems September 9, 2010.
South Dakota Library Network MetaLib User Interface South Dakota Library Network 1200 University, Unit 9672 Spearfish, SD © South Dakota.
TCP/IP Protocol Suite 1 Chapter 18 Upon completion you will be able to: Remote Login: Telnet Understand how TELNET works Understand the role of NVT in.
Chapter 9: Using Classes and Objects. Understanding Class Concepts Types of classes – Classes that are only application programs with a Main() method.
A Data Quality Screening Service for Remote Sensing Data Christopher Lynnes, NASA/GSFC (P.I.) Edward Olsen, NASA/JPL Peter Fox, RPI Bruce Vollmer, NASA/GSFC.
Data Quality Screening Service Christopher Lynnes, Bruce Vollmer, Richard Strub, Thomas Hearty Goddard Earth Sciences Data and Information Sciences Center.
Data Quality Screening Service Christopher Lynnes, Richard Strub, Thomas Hearty, Bruce Vollmer Goddard Earth Sciences Data and Information Sciences Center.
Community Based Initiatives GES DISC UWG 2011 Meeting.
A Data Quality Screening Service for Remote Sensing Data Christopher Lynnes, NASA/GSFC Edward Olsen, NASA/JPL Peter Fox, RPI Bruce Vollmer, NASA/GSFC Robert.
Data Quality of AIRS Level 2 Standard Products* Not all L2 data values are of equal quality quality varies for the same parameter in different retrievals.
Federated Space-Time Query for Earth Science Data Using OpenSearch Conventions ESIP Federated Search Cluster Chris Lynnes Bruce Beaumont Ruth Duerr Hook.
Ambiguity of Quality in Remote Sensing Data Christopher Lynnes, NASA/GSFC Greg Leptoukh, NASA/GSFC Funded by : NASA’s Advancing Collaborative Connections.
Experiences Developing a Semantic Representation of Product Quality, Bias, and Uncertainty for a Satellite Data Product Patrick West 1, Gregory Leptoukh.
GES DISC Services Push Harder? Be Careful? Change Direction? What about adding ______?
1 2.5 DISTRIBUTED DATA INTEGRATION WTF-CEOP (WGISS Test Facility for CEOP) May 2007 Yonsook Enloe (NASA/SGT) Chris Lynnes (NASA)
Presentation transcript:

A Data Quality Screening Service for Remote Sensing Data Christopher Lynnes, NASA/GSFC (P.I.) Edward Olsen, NASA/JPL Peter Fox, RPI Bruce Vollmer, NASA/GSFC Robert Wolfe, NASA/GSFC Contributions from R. Strub, T. Hearty, Y-I Won, M. Hegde, S. Zednik, P. West, N. Most, S. Ahmad, C. Praderas, K. Horrocks, I. Tchered, and A. Rezaiyan-Nojani Advancing Collaborative Connections for Earth System Science (ACCESS) Program Project Page:

Outline Why a Data Quality Screening Service? Making Quality Information Easier to Use via the Data Quality Screening Service (DQSS) DEMO DQSS Under the Hood DQSS Status DQSS Plans 2

Why a Data Quality Screening Service? 3

The quality of data can vary considerablyThe quality of data can vary considerably AIRS VariableBest (%) Good (%) Do Not Use (%) Total Precipitable Water38 24 Carbon Monoxide64729 Surface Temperature54451 Version 5 Level 2 Standard Retrieval Statistics 4

Quality schemes can be relatively simple…Quality schemes can be relatively simple… Total Column Precipitable WaterQual_H2O BestGoodDo Not Use kg/m 2 5

…or they can be more complicated…or they can be more complicated Hurricane Ike, viewed by the Atmospheric Infrared Sounder (AIRS) PBest : Maximum pressure for which quality value is Best in temperature profiles Air Temperature at 300 mbar 6

Quality flags are also sometimes packed together into bytes Cloud Mask Status Flag 0=Undetermined 1=Determined Cloud Mask Cloudiness Flag 0=Confident cloudy 1=Probably cloudy 2=Probably clear 3=Confident clear Day / Night Flag 0=Night 1=Day Sunglint Flag 0=Yes 1=No Snow/ Ice Flag 0=Yes 1=No Surface Type Flag 0=Ocean, deep lake/river 1=Coast, shallow lake/river 2=Desert 3=Land Big-endian arrangement for the Cloud_Mask_SDS variable in atmospheric products from Moderate Resolution Imaging Spectroradiometer (MODIS) 7

Current user scenarios...Current user scenarios... Nominal scenario Search for and download data Locate documentation on handling quality Read & understand documentation on quality Write custom routine to filter out bad pixels Equally likely scenario ( especially in user communities not familiar with satellite data ) Search for and download data Assume that quality has a negligible effect Repeat for each user 8

The effect of bad quality data is often not negligible Total Column Precipitable Water Quality BestGood Do Not Use kg/m 2 Hurricane Ike, 9/10/2008 9

Neglecting quality may introduce bias (a more subtle effect) AIRS Relative Humidity Comparison against Dropsonde with and without Applying PBest Quality Flag Filtering Boxed data points indicate AIRS RH data with dry bias > 20% From a study by Sun Wong (JPL) on specific humidity in the Atlantic Main Development Region for Tropical Storms 10

Percent of Biased Data in MODIS Aerosols Over Land Increases as Confidence Flag Decreases *Compliant data are within Aeronet Statistics derived from Hyer, E., J. Reid, and J. Zhang, 2010, An over-land aerosol optical depth data set for data assimilation by filtering, correction, and aggregation of MODIS Collection 5 optical depth retrievals, Atmos. Meas. Tech. Discuss., 3, 4091–

Making Quality Information Easier to Use via the Data Quality Screening Service (DQSS). 12

DQSS TeamDQSS Team P.I.: Christopher Lynnes Software Implementation: Goddard Earth Sciences Data and Information Services Center Implementation: Richard Strub Local Domain Experts (AIRS): Thomas Hearty and Bruce Vollmer AIRS Domain Expert: Edward Olsen, AIRS/JPL MODIS Implementation Implementation: Neal Most, Ali Rezaiyan, Cid Praderas, Karen Horrocks, Ivan Tchered Domain Experts: Robert Wolfe and Suraiya Ahmad Semantic Engineering: Tetherless World RPI Peter Fox, Stephan Zednik, Patrick West 13

The DQSS filters out bad pixels for the user Default user scenario Search for data Select science team recommendation for quality screening (filtering) Download screened data More advanced scenario Search for data Select custom quality screening parameters Download screened data 14

DQSS replaces bad-quality pixels with fill values Mask based on user criteria (Quality level < 2) Good quality data pixels retained Output file has the same format and structure as the input file (except for extra mask and original_data fields) Original data array (Total column precipitable water) 15

Visualizations help users see the effect of different quality filtersVisualizations help users see the effect of different quality filters 16

DQSS can encode the science team recommendations on quality screening AIRS Level 2 Standard Product Use only Best for data assimilation uses Use Best+Good for climatic studies MODIS Aerosols Use only VeryGood (highest value) over land Use Marginal+Good+VeryGood over ocean 17

Initial settings are based on Science Team recommendation. (Note: Good retains retrievals that Good or better). You can choose settings for all parameters at once......or variable by variable Or, users can select their own criteria...Or, users can select their own criteria... 18

DEMO (Search for AIRX2RET) 19

DQSS Under the HoodDQSS Under the Hood 20

DQSS FlowDQSS Flow user selection ancillary file Screener Quality Ontology data file quality mask screened data file End User data file w/ mask Masker Ontology Query 21

DQSS Ontology (The Whole Enchilada) 22

DQSS Ontology (Zoom)DQSS Ontology (Zoom) 23

DQSS StatusDQSS Status 24

Significant AccomplishmentsSignificant Accomplishments Release 1.0 of DQSS for AIRS Level 2 Standard Retrieval Technology Readiness Level (TRL) = 9 (for AIRS L2) Includes Quality Impact views Disclosure of Invention (NF 1679) filed Announced to AIRS Registered Data Product User Community (~770) Papers and Presentations Managing Data Quality for Collaborative Science workshop (peer- reviewed paper) Sounder Science meeting ESDSWG Poster A-Train User Workshop (part of GES DISC presentation) AGU: Ambiguity of Data Quality in Remote Sensing Data Ontology (v. 2.4) to accommodate both AIRS and MODIS L2 25

Current StatusCurrent Status DQSS is operational for AIRS L2 Standard Products DQSS is offered through the Mirador data search interface* at the GES DISC DQSS has been refactored to work in LAADS/MODAPS environment 2 1 / 2 month delay due to refactoring and (successful) audit of NPR compliance by NASAs Office of the Chief Engineer Schedule no longer has room for client-side screening But maybe...(to be continued) * 26

Metrics DQSS is included in web access logs sent to EOSDIS Metrics System (EMS) Tagged with Protocol DQSS EMS -> ESDS bridge implemented Metrics from EMS: Users: 50 Downloads:

DQSS RefactoringDQSS Refactoring GES DISCMODAPS/LAADS Synchronous vs. Asynchronous SynchronousAsynchronous batch post-process Screening machines Archive/Distributio n Processing minions (no outward connections allowed) 28

Refactored DQSS encapsulates ontology and screening parameters MODAPS Minions LAAD SWeb GUI DQSS Ontology Service data product screening options MODAPS Database screening parameters (XML) selected options screening parameters (XML) screening parameters (XML) Encapsulation enables reuse in other data centers with diverse environments. 29

DQSS PlansDQSS Plans 30

2010 Plans2010 Plans Adding more products MODIS L2 Water Vapor – 2/2011 MODIS L2 Clouds and Aerosols – 5/2011 Microwave Limb Sounder (MLS) L2 – 6/2011 Ozone Monitoring Instrument (OMI) L2 – 8/2011 Integrate with other services AIRS Variable Subsetting & NetCDF Reformatting – 3/2011 OPeNDAP – 6/2011 MODAPS Web Services – 10/11 Refactor for reusability & sustainability – 9/2011 Release – 11/2011 No longer planned: client-side screening Except...refactored DQSS may make this possible... 31

Combine DQSS with Other ServicesCombine DQSS with Other Services AIRS Subsetting, quality screening and reformatting to netCDF are currently mutually exclusive Combining them will also raise awareness of quality screening DQSS will also be more usable to communities more comfortable with netCDF (e.g., modeling community) OPeNDAP OPeNDAP, Inc. is working to support back-end calls to REST services Gateway is based on ACCESS-05 project (CEOP Satellite Data Server) Will allow access to DQSS via analysis and visualization clients (e.g., Panoply, IDV) May be reused for other back-end REST services 32

Refactoring for ReleaseRefactoring for Release Proposal schedule included refactoring for long-term sustainability A key element of successful technology infusion Current refactoring for MODIS environment... Gives us a head start May also give us an avenue to client-side capabilities 33

New Client Side ConceptNew Client Side Concept Ontology Web Form Data Provider Masker Screener dataset-specific instance info data files screening criteria XML File data files GES DISCEnd User 34

DQSS RecapDQSS Recap Screening satellite data can be difficult and time consuming for users The Data Quality Screening System provides an easy-to- use service The result should be: More attention to quality on users part More accurate handling of quality information… …With less user effort 35

Backup SlidesBackup Slides 36

DQSS Target Uses and UsersDQSS Target Uses and Users Routine Visual- ization Quick Recon. Machine- levelMetrics InterdisciplinaryXX EducationalXX Expert/Power? XX Applications X Algorithm Developers X 37

ESDSWG ActivitiesESDSWG Activities Contributions to all 3 TIWG subgroups Semantic Web: Contributed Use Case for Semantic Web ESIP Participation in ESIP Information Quality Cluster Services Interoperability and Orchestration Combining ESIP OpenSearch with servicecasting and datacasting standards Spearheading ESIP Federated OpenSearch cluster 5 servers (2 more soon); 4+ clients Now ESIP Discover cluster (see above) Processes and Strategies: Developed Use Cases for Decadal Survey Missions: DESDynI-Lidar, SMAP, and CLARREO 38

Projected TRL BreakdownProjected TRL Breakdown 39