IOOS Data Management Integration Standards Plans in the Northeast Eric Bridger (GMRI) Sep. 26 2012.

Slides:

Advertisements

Similar presentations

1 NASA CEOP Status & Demo CEOS WGISS-25 Sanya, China February 27, 2008 Yonsook Enloe.

Advertisements

ECOOP Data Management System (T2.2/WP2) Declan Dunne 13 th February 2008, Athens.

The Model Output Interoperability Experiment in the Gulf of Maine: A Success Story Made Possible By CF, NcML, NetCDF-Java and THREDDS Rich Signell (USGS,

NERACOOS: Unlocking the power of data Riley Young Morse | Gulf of Maine Research Institute.

DMAC ST and the Activities of the IOOS PO Derrick Snowden DMAC Steering Team

Integrating NOAA’s Unified Access Framework in GEOSS: Making Earth Observation data easier to access and use Matt Austin NOAA Technology Planning and Integration.

Crossing the Digital Divide

Building the GCOOS: Integration of Data Dr. Matthew Howard GCOOS-RA Annual Meeting of the Parties & Board of Directors February 2009 Orlando, FL.

The SouthEast Coastal Ocean Observing SECOORA Meeting Regional Association (SECOORA) June 11-12, 2007 SECOORA Data Management and Communications SECOORA.

Integrated Ocean Observing System Data Management and Communications March 2004 The US Integrated Ocean Observing System (IOOS) Plan for Data Management.

Data Discovery 1.Data discovery systems 2.TDS and metadata 3.NcISO services.

Steve Rutz NOAA/NESDIS National Oceanographic Data Center NODC Observing Systems Team Leader June 21, 2011.

Unidata TDS Workshop THREDDS Data Server Overview October 2014.

Introduction Downloading and sifting through large volumes of data stored in differing formats can be a time-consuming and sometimes frustrating process.

The use of standard OGC web services in integrating distributed model, satellite and in-situ datasets Alastair Gemmell Jon Blower Keith Haines Environmental.

A Super-Regional Modeling Testbed for Improving Forecasts of Environmental Processes for the U.S. Atlantic and Gulf of Mexico Coasts Cyberinfrastructure.

From Data Integration to Integrated Solutions Tom Shyka, GMRI Paul Currier, NH DES Water Quality Workshop MACOORA - NERACOOS -SECOORA IOOS, USGS, EPA,

Quick Unidata Overview NetCDF Workshop 25 October 2012 Russ Rew.

Implementation of Model Data Interoperability for IOOS: Successes and Lessons Learned Rich Signell USGS Woods Hole, MA / NOAA Silver Spring USA Model Data.

Observing System Monitoring Center Integrating data and information across observing system networks.

Bringing it All Together: NODC’s Geoportal Server as an Integration Tool for Interoperable Data Services Kenneth S. Casey, Ph.D. YuanJie Li NOAA National.

WP 9 (former Task 1b of WP 1): Data infrastructure Robert Huber UNI-HB Esonet 2nd all regions workshop, Paris

Unidata’s TDS Workshop TDS Overview – Part II October 2012.

2 3 ROMS/COAWST NcML file 4 5 Exploiting IOOS: A Distributed, Standards-Based Framework and Software Stack for Searching, Accessing, Analyzing and.

Unidata TDS Workshop TDS Overview – Part I XX-XX October 2014.

THREDDS Data Server Ethan Davis GEOSS Climate Workshop 23 September 2011.

IOOS DMAC Meeting Regional Association Discussion IOOS PO May 2015.

U.S. Department of the Interior U.S. Geological Survey Management of Oceanographic time-series data at the Woods Hole Coastal and Marine Science Center.

Mid-Course Review: NetCDF in the Current Proposal Period Russ Rew

Cyberinfrastructure - Collaborative Portal for the IOOS Super-Regional Modeling Testbed Sara Graves, Manil Maskey, Ken Keiser Information Technology &

Glider data (*.*bd files) via Iridium or rsync from WHOI, Umaine or UMaryland Disk NetCDF/Acquire WMO ID IOOS Glider DAC ERDDAP OPeNDAP sequence or download.geoJson,.json.nc,.ncCF,.ncCFMA.csv,.tsv,

Enhancements to a Community Toolset for Ocean Model Data Interoperability: Unstructured grids, NCTOOLBOX, and Distributed Search Rich Signell (USGS), Woods.

Tools in Support of a National DMAC Derrick Snowden NERACOOS/ODP Annual Meeting 26 Sep 2012.

Accomplishments and Remaining Challenges: THREDDS Data Server and Common Data Model Ethan Davis Unidata Policy Committee Meeting May 2011.

DMAC: Infrastructure to enable delivery of IOOS ® information Presentation to the IOOS Advisory Committee Derrick Snowden, System Architect, U.S. IOOS.

IOOS Model Data Interoperability Design ROMS POM WW3 WRF ECOM NcML Common Data Model OPeNDAP+CF WCS NetCDF Subset THREDDS Data Server Standardized (CF)

1 NOAA IOOS Program Data Integration Framework (DIF) Project Overview Adapted from a brief to the NOAA Data Management Committee August 6, 2008 by Jeff.

NODC ↔ Data Consumers Steve Rutz NOAA/NESDIS National Oceanographic Data Center NODC Observing Systems Team Leader June 21, 2011.

Opendap dev - meeting, Boulder, Feb 2007 OPeNDAP infrastructure in European Operational Oceanography T Loubrieu (IFREMER) T Jolibois (CLS)

IOOS Modeling Testbed Cyberinfrastructure Rich Signell, USGS, Woods Hole, MA IOOS-RA-Briefing, Feb 14, 2012.

IOOS Coastal Ocean Modeling Testbed (COMT) Cyberinfrastructure Oceans 12 Becky Baltes, IOOS Liz Smith, SURA Rich Signell, USGS Eoin Howlett, Kyle Wilcox,

Unidata TDS Workshop THREDDS Data Server Overview

Recent developments with the THREDDS Data Server (TDS) and related Tools: covering TDS, NCML, WCS, forecast aggregation and not including stuff covered.

IOOS Data Services with the THREDDS Data Server Rich Signell USGS, Woods Hole IOOS DMAC Workshop Silver Spring Sep 10, 2013 Rich Signell USGS, Woods Hole.

Unidata’s TDS Workshop TDS Overview – Part I July 2011.

Kevin O’Brien University of Washington/JISAO NOAA/PMEL The Observing System Monitoring Center Steve Hankin, PMEL Ted Habermann, NGDC David Neufeld, NGDC.

TEMPLATE DESIGN © OOSTethys - Open Source Software for the Global Earth Observing Systems of Systems E. Bridger 1, L.

1 Implementing QA/QC Standards for In Situ Ocean Sensors Using OGC-Sensor Web Enablement a.k.a. QARTOD to OGC a.k.a. Q2O Project Status Brief to NOAA IOOS.

The Unified Access Framework for Gridded Data … the 1 st year focus of NOAA’s Global Earth Observation Integrated Data Environment (GEO-IDE) Steve Hankin,

SECOORA Geo Tools Presentation Dan Ramage, University of South Carolina

Emodnet Central Portal Ifremer inputs Excuse us for not beeing present today! Have a fruitful meeting!

Chemical lot - HOW: Infrastructure set up based on SeaDataNet V1 efficient distributed Marine Data Management Infrastructure; Principle of “ADOPTED AND.

1 of 37 Next Steps SOS DescribeSensor implementation (SensorML) Testing/evaluation/refinement of existing work Build or borrow needed components … Reference.

Jeff de La Beaujardière, PhD NOAA Sr System Architect, IOOS Program Office DAARWG Meeting Silver Spring, Integrated Ocean Observing System (IOOS)

U.S. INTEGRATED OCEAN OBSERVING SYSTEM DATA ACCESS SERVICES Coastal Geotools: Advances in Federal Data Access Services Hassan Moustahfid, Chris Duncombe.

The Research Data Archive at NCAR: A System Designed to Handle Diverse Datasets Bob Dattore and Steven Worley National Center for Atmospheric Research.

ERDDAP The Next Generation of Data Servers Bob Simons DOC / NOAA / NMFS / SWFSC / ERD Monterey, CA Disclaimer: The opinions expressed.

DOC / NOAA / NMFS / SWFSC / ERD

Catalog-driven workflows using CSW Rich Signell, USGS, Woods Hole, MA, USA Filipe Fernandes, SECOORA, Brazil Kyle Wilcox, Axiom Data Science, Wickford,

Access to IOOS Data Relevant to OOI Kathleen Bailey NOAA/NOS/IOOS January 6, 2016.

Rich Signell Roland Viger Curtis Price USGS Community for Data Integration Feb 15, 2012.

1 2.5 DISTRIBUTED DATA INTEGRATION WTF-CEOP (WGISS Test Facility for CEOP) May 2007 Yonsook Enloe (NASA/SGT) Chris Lynnes (NASA)

Interoperability Day Introduction Standards-based Web Services Interfaces to Existing Atmospheric/Oceanographic Data Systems Ben Domenico Unidata Program.

NOAA IOOS SOS Implementations in 2008 Jeff de La Beaujardière, PhD NOAA IOOS Program DIF Sr Systems Architect.

Update on Unidata Technologies for Data Access Russ Rew

NOAA EDMC Ocean Observatories Initiative Cyberinfrastructure Karen Stocks OOI CI Data Curator University of California, San Diego Ocean Observatories.

Unidata Infrastructure for Data Services Russ Rew GO-ESSP Workshop, LLNL

The Arctic Observing Network and its Data Management Challenges Florence Fetterer (NSIDC/CIRES/CU), James A. Moore (NCAR/EOL), and the CADIS team Photo.

IOOS National Glider Data Assembly Center

Robert Dattore and Steven Worley

Presentation transcript:

IOOS Data Management Integration Standards Plans in the Northeast Eric Bridger (GMRI) Sep

Summary Interoperable Data Sharing History Current NERACOOS DMAC – DAC / SOA Rethinking Data Integration Framework Future Directions ( In-situ obs and Models )

Long History of Interoperable Data Sharing ODP est ODP GCMD Metadata 2005 Early SOS, CF Compliance (OOSTethys) 2006 GOM Model inter-operability 2007 EPA EN Web Services Workshop 2008 SWE/DIF SOS 2009 IOOS Model Testbed ISO Metadata Workshop 2011

NERACOOS DAC Issues Dates to 2001 – UMO NetCDF => Postgres Data Aggregation Center not a true SOA Numerous diverse ingest methods – Locally many OOSTethsy Perl ASCII (UNH, SmartBay, UConn) – NDBC ASCII ingest – never switched to SOS – RSS (CDIP) DAC / Data Provider discrepancies – UMO Post Recovery data New data sources not easily integrated. Missing data. UConn, URI Sustainability

Rethinking NERACOOS DMAC IOOS Convergence of OGC Standards – approach New IOOS SOS SWE Profile – OGC SWE, CF, Vocabularies at MMI, mapped SOS to UCAR NetCDF Common Data Model, SensorML metadata – Discrete Sampling Geometries implemented in the NetCDF Java library ( IOOS Modeling Testbed) IOOS Funded Tools to assist RA’s – Java based, ncSOS, Custom 52North SOS – JS / Python Clients coming SOA and DAC – SOA - Data Providers should control their data and metadata – Still a role for DAC

THREDDS is the glue Thematic Real-Time Environmental Distributed Data Services THREDDS plugin architecture – ncML, ncWMS (familiar with these, CF 1.6) – OpenDAP – access to NetCDF or subsets – ncISO – ISO Metadata (2011 NECODP Workshop) – New ncSOS (for NetCDF time series files) – Handle in-situ obs and models with same protocols.

Problems Solved IOOS new SOS Profile Compliance NODC Archiving CF 1.6 via ncML In-situ obs (real-time and archival) – SOS, OpenDAP, NetCDF and subsets, WMS ISO 19115/2 Metdata via ncISO used by IOOS Catalog, GEOSS, NGDC Geoportal Catalog. Data providers control data and metadata Post Recovery files QA/QC, recalibrated

Access to historic QA/QC’d long time series (back to 2001 controlled by data provider “The first time is the hardest time” A model for dealing with diverse data ingests in the NERACOOS Region –Data Aggregation Center There will remain a need for a DAC 52 North could serve this role Will work with existing PostGIS DB Views Comes with ingest modules (NDBC, NERRS, CO- OPS) ASCII ingest could continue Future

Regional Data Integration Framework Infrastructure will enable data portal access ERDDAP - NetCDF based fast access to historic data. Multiple output formats: – TDS WMS, ISO, SOS, OpenDAP – ERDDAP GeoJSON, HTML, CSV, REST API Meeting Archive obligations to NODC Innovative Products

Cloud Web Services NERACOOS Cloud experience Cost Savings, Scalability, Shared Resources. Binary Image w Linux, Apache, PostGIS, PHP, TDS and plugins, NetCDF, latest Java lib, etc. Point and click Web Services ?

Model Forecast / Observation Viewer elobs elobs Operational since 2009 www neracoos org / datatools/forecast/modelobs Web Services Application