ESIP Federation Meeting July 2008 Data and Metadata Standards for GHRSST: present to future ESIP Federation Meeting July 2008 Univ. of New Hampshire, Durham, NH Ed Armstrong Jet Propulsion Lab/CalTech Physical Oceanography DAAC Pasadena, California

ESIP Federation Meeting July 2008 Data throughput… Gb Day -1

ESIP Federation Meeting July 2008 User Base Power Users: NAVO UK Met Office NASA JPL Canada Meteorological Agency Danish Meteorological Institute

ESIP Federation Meeting July 2008 Product Specifications netCDF 3.6 Self describing format – Popular in the ocean modeling and NWP community Direct access – Improves usability: L2P products can have over 20 arrays Portable – GHRSST is an international project with data produced on a variety of platforms “bundled” with CF/COARDS compliant metadata Limitations: – No internal compression. Externally compressed (bzip2 or gzip) – No chunking

ESIP Federation Meeting July 2008 Metadata Specifications NASA Directory Interchange (DIF) format Compact, well defined user history, simple to implement A granule level XML metadata (File Record [FR]) for each file made by producer – typical entries are dynamic such as time and position – e.g., A product level XML metadata (Dataset Description [DSD]) – typical entries are more static contacts, document links, and summary statements – e.g., Some “custom” tags added. For example, These are stored as separate databases at the JPL GDAC Master Metadata Repository (MMR) – mySQL relational database for search, query, and assembly

ESIP Federation Meeting July 2008 Metadata Specifications FGDC conversion All GHRSST products are transferred to NOAA Longterm Stewardship and Reanalysis Facility (LTSRF) after a period of 30 day NOAA has requirement for FGDC compliant metadata for its products. No data ingested without metadata. A conversion of the FR and DSD records to FGDC is performed on file-by-file basis – Mappings from DIF to FGDC done using XSLT style sheets – For L2P data multiple files of same product type in same FGDC record. Up to 288 files per FGDC record.

ESIP Federation Meeting July 2008 Mapping Anomaly Some mappings are not clean. FR record: Technical Contact Ed Armstrong Jet Propulsion Laboratory, 4800 Oak Grove Dr, Pasadena, CA USA mapped FGDC record: Jet Propulsion Laboratory, 4800 Oak Grove Dr, Pasadena, CA USA Jet Propulsion Laboratory, 4800 Oak Grove Dr, Pasadena, CA USA Jet Propulsion Laboratory, 4800 Oak Grove Dr, Pasadena, CA USA Jet Propulsion Laboratory, 4800 Oak Grove Dr, Pasadena, CA USA Jet Propulsion Laboratory, 4800 Oak Grove Dr, Pasadena, CA USA

ESIP Federation Meeting July 2008 Measurements of SST (netCDF CF1.3) Depth 10  m 1 mm 1 m 10 m SST skin – SST 10m (K) 1. Night-time (or strong winds) profile in red 2. Day time situation, strong solar radiation and light winds sea_surface_skin_temperature sea_surface_subskin_temperature sea_water_temperature (Attribute:depth) Sea_surface_foundation_temperature Night-time (or strong winds) profile in red surface_temperature

ESIP Federation Meeting July 2008 CF 1.3 Specification for SST standard names surface_temperature (SSTint) sea_water_temperature (SSTdepth) sea_surface_skin_temperature; K sea_surface_subskin_temperature; K sea_surface_foundation_temperature: K

ESIP Federation Meeting July 2008 Future product development NetCDF version 4 Benefits –Internal compression and chunking (from HDF storage model) –Support for Parallel I/O –Backward compatibility with v3.6 –The lure of “new stuff” Advantages must be weighted against a format that has proven metrics and use –Operationally it is difficult to implement a new file format –The user may (will) complain. New software and libraries to install etc. Evaluated this summer, with a decision by next GHRSST Science Team Meeting June 2009 for adoption

ESIP Federation Meeting July 2008 Future metadata requirements European Space Agency will require metadata for next generation Sentinel satellite derived products to be ISO compliant. This directly impacts GHRSST. Also required for using GHRSST products in WMO Integrated Global Observing Systems (WIGOS) systems (a GEOSS component) What to change in our metadata production system ? We have an investment in something that works. We can’t break everything and start from scratch. The way forward is likely to generate mappings to ISO similar to FGDC conversion. –Currently reviewing our metadata architecture (FR and DSD XML records) to see what needs to be added/modified –Review must be completed in the next months. A written protocol by the end of the year. –What part of ISO to conform to?? North American profile…Remote Sensing extensions. –ISO metadata embedded in netCDF v4 GHRSST file ??

ESIP Federation Meeting July 2008 Summary GHRSST has demonstrated significant achievements with regard to data and metadata standards on a relatively modest budget Although focused on SST discipline, is serves as a model for a working GEOSS Supporting Science and Applications through well thought out and implemented best practices One of the challenges will be to evolve with new technologies wrt to format and metadata standards ISO –Relatively straightforward once the mappings are determined netCDF v4 –More challenging. More investigation needed. Others: Search, Discovery, Imaging –THREEDS: data cataloguing –JPL Datacasting: RSS-like data subscription –Swath granule subset –ECHO metadata inventory –OPeNDAP –WMS POET

ESIP Federation Meeting July 2008 Backups

ESIP Federation Meeting July 2008 (A)ATSR reduces uncertainty in global average SST (John Kennedy, Nick Rayner)

ESIP Federation Meeting July 2008 GDAC overview

ESIP Federation Meeting July 2008 Polar Orbiting infrared has high accuracy & spatial resolution Geostationary infrared has high temporal resolution Microwave Polar orbiting has all-weather capability GHRSST-PP L4 products (A. Harris)

ESIP Federation Meeting July 2008 SST system quality control and uncertainty estimation Analysis Products netCDF format Input SST data of Varied format & with no errors Data merging and analysis SST Obs. Products netCDF format Add ancillary QC SSES& dynamic flags The GHRSST-PP Strategy SST Observations Applications L3 L1P Radiance data

ESIP Federation Meeting July 2008 GDS-v2.0 Rev 0.1 June 2008 Generate L2P data products (L2Pc -> L2P) Generate L4 analysed data products (e.g., SSTfnd) based on L2P data input. Satellite SST data Ingest & QC input data streams Auxiliary data Referenc e data fields Generate L3P data products Referenc e data fields Analysed L4 SST data products Rolling and long-term archive system Derive Single Sensor error statistics (SSES) Satellite and in situ MDB Single Sensor Error Statistics Generate MDB records MDB/SSES system MDB recor ds Diagnostics, metrics and GDS data product validation HR-DDS archive GDAC &RDAC L4 SST HR- DDS data granules L2P HR- DDS data granules Extract HR- DDS L2P granules User data access through LAS/DODS HR- DDS server Extract HR-DDS L4 SST granules HR-DDS system In situ data CDR SST data products Re-Analysis f L2P/L3P/L4 L3P SST data products Referenc e data fields L2P/L2Pc SST data products Metrics definition s GMPE SST ensemble products Generate GMPE products Referenc e data fields Main data flow HR-DDS data flow MDB/SSES data flow RAN data flow GMPE data flow Referenc e data fields Auxiliary data GMPE system RAN system