GES DISC Services Push Harder? Be Careful? Change Direction? What about adding ______?

Discovery ServicesDiscovery Services  Mirador  Development scaled back to sustaining engineering level  External Search (in Test mode TS1) External Search (in Test mode TS1)  Technically successful, but...  Usability-challenged  Start and stop date/time  Total number of hits  Uniform sort order  Duplicates  Usability: Simplicity vs. Features (esp. Services)  Mirador Usability Sounding Board?  mail list for queries on usability quandaries

Data ServicesData Services

Number of Users* - March 2011Number of Users* - March 2011 *OK, not really. It’s the number of distinct IP addresses

Number of Users*: Sep 2010 – Apr 2011Number of Users*: Sep 2010 – Apr 2011

Data Quality Screening ServiceData Quality Screening Service

The quality of AIRS data varies considerably AIRS ParameterBest (%) Good (%) Do Not Use (%) Total Precipitable Water38 24 Carbon Monoxide64729 Surface Temperature54451 Version 5 Level 2 Standard Retrieval Statistics

Quality Schemes can be complicatedQuality Schemes can be complicated Hurricane Ike, viewed by the Atmospheric Infrared Sounder (AIRS) PBest : Maximum pressure for which quality value is “Best” in temperature profiles Air Temperature at 300 mbar

Current user scenarios...Current user scenarios...  Nominal scenario  Search for and download data  Locate documentation on handling quality  Read & understand documentation on quality  Write custom routine to filter out bad pixels  Equally likely scenario ( especially in user communities not familiar with satellite data )  Search for and download data  Assume that quality has a negligible effect Repeat for each user

The effect of bad quality data is often not negligible Total Column Precipitable Water Quality BestGood Do Not Use kg/m 2 Hurricane Ike, 9/10/2008

DQSS replaces bad-quality pixels with fill values Mask based on user criteria (Quality level < 2) Good quality data pixels retained Output file has the same format and structure as the input file (except for extra mask and original_data fields) Original data array (Total column precipitable water)

DQSS DemoDQSS Demo

DQSS Status + PlansDQSS Status + Plans  Operational for AIRS L2 Standard Retrieval  Nearly operational for MODIS Water Vapor  Next: MODIS Aerosols, MLS Water Vapor  Next: ???  Also, OPeNDAP Gateway nearly reader to front-end DQSS  Allow OPeNDAP access to DQSS-served data.

OPeNDAP*  Remote access to data: no need to download  Access at fine granularity  Variable  Array regions  Stride  Present HDF data as netCDF/CF  Enhances Tool Usability  Reformatting: ASCII, netCDF *OPeNDAP = OpenSource Project for a Network Data Access Protocol

Who Uses OPeNDAP?Who Uses OPeNDAP?  Industrial-strength scripters looking for subsets  Thick client users  GrADS, Panoply, IDV, McIDAS-V, Ferret  Internal Systems  Giovanni  MapServer  Simple Subset Wizard

OPeNDAP DemoOPeNDAP Demo

OGC* Standards - WMSOGC* Standards - WMS  Web Map Service (WMS)  URL request: returns map image  Implemented with open-source MapServer  Giovanni also supports WMS  Consumers:  AIRS NRT page AIRS NRT page  Google Earth  GIS programs  IDV  Giovanni *OGC = Open Geospatial Consortium

OGC - WCSOGC - WCS  Returns “coverages”: data variables in NetCDF/CF1  Used by other systems  DataFed  Giovanni  Atmospheric Composition Portal  Simple Subset Wizard

Subsetting  Semi-custom tools for some products  Reuse HSE libraries from UAH  Reuse Lats4D from A. DaSilva  Usually HDF in -> HDF out  Implemented as REST* URLs  Subsetting at time of download  Subsets are implemented as user requests come in  Areas where we should proactively develop subsetters?

~100 Subsettable Datasets~100 Subsettable Datasets  AIRS Radiances (channel), L2 Retrievals (variable), L3 (spatial+variable via SSW)  MLS L2 (spatial+variable)  TOMS L3, OMI L2-L3 (spatial+variable), OMI L2  TRMM L3 (spatial+variable)  Models (spatial+variable)  Did we miss any (that shouldn’t be missed)?  Should all SSW subsets be offered in Mirador?

Format ConversionFormat Conversion  Custom code for some L3 and L2 datasets  HDF -> netCDF/CF  Improves usability in tools  Moving toward external tools where possible  OPeNDAP  Lats4d: based on GrADS

Simple Subset WizardSimple Subset Wizard  Desired: “Just give me the data from time 1 to time 2 for this spatial box”.  Current: “search for granules, view granules, select granules, select subset option, re-enter spatial box...”  ESDIS-funded technology infusion effort  DEMO DEMO

Giovanni EvolutionGiovanni Evolution

G3 Evolution to Agile Giovanni (G4)G3 Evolution to Agile Giovanni (G4)  Factors driving evolution  G3 architecture was never completed  No workflow engine  Cost of adding significant features is too high  Architecture is too brittle

Key G4 GoalsKey G4 Goals  Reduce cost and time to add new features  Improve performance over G3  Support external maintenance of external data

Evolution PlanEvolution Plan  Implement new projects in Agile Giovanni (G4)  Aerostat ACCESS project  Point data in database, bias corrections  Year of Tropical Convection (YOTC) Year of Tropical Convection (YOTC)  Level 2 data  Community-based Giovanni  Externally maintained portals and data  Implement G4 features to meet existing G3 functionality  Migrate G3 instances to G4 portals

Roads Not TakenRoads Not Taken  Giovanni 3 enhancements  ISO Metadata  Document architecture  Mirador features and usability revamp  Persistent locators  Unique identifiers Not  Giovanni Evolution  DQSS  Atmospheric Composition Portal  Simple Subset Wizard  Community-based Initiatives  Mirador External Search  Expanding data services Taken

Backup SlidesBackup Slides

Agile Giovanni Architectural FeaturesAgile Giovanni Architectural Features  Model-view-controller  Semantic Web underpinnings  Variable-centric, not dataset-centric  Code reuse: Kepler, YUI, JCache, MapServer