Edward Armstrong, Jorge Vazquez, Andrew Bingham, Thomas Huang, Chris Finch, Charles Thompson, Tim Stough, Tim McKnight, and Cynthia Chen JPL PO.DAAC / GDAC California Institute of Technology 10 th GHRSST Science Team Meeting Santa Rosa, CA 1 June 2009

Introduction The Global Data Assembly Center (GDAC) Ingest and maintain interfaces to 32 GHRSST L2P/L3P/L4 datastreams from 10 RDACS. ~25 G/day > 5000 files/day Realtime quality checking for data and metadata granules Create FGDC metadata for daily collection granules Distribution via FTP/OPeNDAP/POET Maintain interfaces to the LTSRF for 30 day old data and metadata exchange GDAC web portal Work with RDAC data providers on new or updated products 1 June th GHRSST Science Team Meeting, Santa Rosa, CA

Continued Maintain discovery and access interfaces. MMR has > 1.5 million records User and application support MODIS L2P production in collaboration with the OBPG and RSMAS Ancillary L2P filling Datacasting and other value added tools, services, and products 1 June th GHRSST Science Team Meeting, Santa Rosa, CA

New GHRSST products Level 4 ABOM GAMSSA (28 km) and RAMSSA (9km) New version of AVHRR_OI, and AVHRR_AMSR_OI Level 2P NEODAAS regional L2P for AVHRR_17L, _18L UPA L2P for AATSR (ATS_NR_2P) Totals (32 active) 18 L2P 2 L3P 12 L4 Product table URL 1 June th GHRSST Science Team Meeting, Santa Rosa, CA

Operations GDAC uptime 2 separate 2 day outages (down 100 hours; < 1%) Scheduled infrastructure maintenance RAID failure Some data provider outages (3 day outage of MODIS L2P) Latency No statistics compiled But no serious issues reported from users Coordination with LTSRF Close communication maintained between operational centers Have dealt with hiccups on a case-by-case basis 1 June th GHRSST Science Team Meeting, Santa Rosa, CA

PO.DAAC Data Management and Archive System (DMAS) New ingest manager, new database, new discovery……NEW !! From the perspective of the RDACs nothing will change Improvements in scalability, throughput, load balancing, discovery, database storage, product validation, and metrics capturing. Ease of maintenance and operator supervision GHRSST datastreams have been tested with new system Fully operational by summer June th GHRSST Science Team Meeting, Santa Rosa, CA

PO.DAAC Data Management and Archive System Service-oriented system Common metadata data model Metadata extraction and translation Data discovery Parallel and load-balanced data ingestion Dynamic deployment of ingest engines Storage management Validation State-driven Secure Portable Scalable

GHRSST Migration Adaptation MMR validation and translation Data file validation Scans local/remote locations for new data Integration with back-end RDAC cluster Inventory Full migration from existing MySQL database Port to use the new data model FGDC and Index generators Website Fully Operational End of June ’09

GDAC Distribution Statistics Number of distinct monthly users through April 2009 FTP/OPeNDAP/POET 1 June th GHRSST Science Team Meeting, Santa Rosa, CA

Distribution Statistics Number of files distributed 1 June th GHRSST Science Team Meeting, Santa Rosa, CA

Distribution Statistics Volume distribution 1 June th GHRSST Science Team Meeting, Santa Rosa, CA

Product distribution – L2P By distinct number of users May 2007 – May 2009 aa 1 June th GHRSST Science Team Meeting, Santa Rosa, CA

Product distribution – L4 By distinct number of users May 2007 – May June th GHRSST Science Team Meeting, Santa Rosa, CA

XML TAG summary Current metadata shortcomings, two examples Limited data provenance, especially for L4 products No L2P orbit characterization… ISO roadmap Crosswalk from current GHRSST metadata model to ISO completed Preliminary UML diagrams based on ISO 19115, , and completed Available: Advisory-Group-(XML-TAG).htmlhttp:// Advisory-Group-(XML-TAG).html WMO metadata model compatibility 1 June th GHRSST Science Team Meeting, Santa Rosa, CA

GDAC value added projects NASA MEaSURES proposal funded in collaboration with Univ. of Miami to generate global ultra high resolution blended products based on OI wavelet basis functions and motion compensation (PI Mike Chin, Arthur Mariano, Jorge Vazquez, Ed Armstrong).

10 th GHRSST Science Team Meeting, Santa Rosa, CA GDAC value added projects NASA Research to Operations proposal funded to improve existing SPORT SST composite products for the US east and west coasts (PI Jorge Vazquez and Gary Jedlovec). Collaboration with SPORT who already has customers in the National Weather Service and contacts in the oil and gas industry. Integration of GHRSST products for fisheries and marine ecosystem decision support and marine GIS (Dale Kiefer, USC and Ed Armstrong) Eddy detection (Charles Dong, UCLA) Seasonal SST fronts (Sara Purca, IMAEPE, Peru)

SST gradient and frontal probability maps Based on 4 km AVHRR Pathfinder and GHRSST NCDC AVHRR_OI data 1 June th GHRSST Science Team Meeting, Santa Rosa, CA

SST gradient and frontal probability maps Based on 4 km AVHRR Pathfinder and GHRSST NCDC AVHRR_OI data 1 June th GHRSST Science Team Meeting, Santa Rosa, CA

Virtual Ocean Data Center 1 June th GHRSST Science Team Meeting, Santa Rosa, CA NASA ACCESS Virtual Ocean Data Center (VODC) proposal funded to create infrastructure for transparent data and metadata harvesting and distribution using Object Orientated Data Technology (OODT) middleware (PI Rob Raskin, co-Is Chris Mattman, Sean McClease, Ed Armstrong). OODT runner up in NASA 2003 Software of the Year award “one-stop” Internet presence for searching, and extracting data from oceanographic data sources.

Search Architecture CAM-20

Search for Content (regular search page) CAM-21

10 th GHRSST Science Team Meeting, Santa Rosa, CA JPL Datacasting Model: an RSS-like feed for satellite data streams Client can subscribe to feed: “aware” on new data as soon as it is available Provider can package feed with metadata that consumer can filter on (e.g., location, cloud free pixels etc.) Datacasting is now operational on some GHRSST L2P data Version 2.0 client available

L2 subsetting Orbital L2 subsetting for regions of interest Based on NSIDC backtrack algorithm for granule query Prototyped on GHRSST L2P data Fall June th GHRSST Science Team Meeting, Santa Rosa, CA

GHRSST L4 validation 1 June th GHRSST Science Team Meeting, Santa Rosa, CA

Summary GDAC meeting all its fundamental operational and distribution requirements New data management architecture (DMAS) will make it easier to expand to the increasing volumes of GHRSST datastreams Low downtime Users reporting no or little issues with access Improved tools and services on the horizon Discovery improvement with VODC and “new” MMR Implement L2 subsetting More datacasting streams and browse images Improvements Metrics capturing (“casting” and “harvesting”) ISO metadata implementation Product documentation Applications and products UHR SST L4 products Supporting directly or indirectly a number of user applications 1 June th GHRSST Science Team Meeting, Santa Rosa, CA