NOAA Integrated Ocean Observing System (IOOS) Data Integration Framework Richard Crout NOAA NWS National Data Buoy Center GCOOS Board of Directors Meeting Corpus Christi, TX 19 August 2008

Outline NOAA’s Integrated Ocean Observing System Data Integrated Framework –Challenge –Solution - DIF –Goals and Objectives –Approach –Coordination –Web Services –Status –Validation and Verification Regional Implementation Future Challenges

NOAA IOOS … Is modernizing the way NOAA collects, shares, and uses ocean information Is increasing data interoperability across NOAA and the regions and efficiency of operations Is expediting access to data for improved decision making  To achieve these objectives, the Program focuses its contributions in two main areas: 1.Developing the IOOS Data Integration Framework (NOAA DIF-to-National DMAC) 2.Managing the regional IOOS partnership

NOAA IOOS Program Focus Standards –Develop and maintain a process to document, coordinate, evaluate and accept data standards Regions –Invest in Regional Associations efforts that contribute to the definition and implementation of IOOS Data Integration Framework (DIF) –Conduct a limited scope implementation of IOOS within NOAA as a proof of principle and to gather lessons to inform the larger IOOS effort

Data Integration Framework (DIF) NOAA IOOS Data Integration Framework Systems Engineering & Standards Development Integrated Ecosystem Assessment Month 36 Integration of 5 IOOS Core Variables ProductEnhancement Test & Evaluation Verification &Validation Integrated information services for NOAA programs Identify observation gaps Validated enhanced data products NOAA’s Data Integration Framework Enhanced decision support through: NOAA MISSION OBJECTIVES NOAA 5 Core Variables Months 0-12Month 18Month 24 Harmful Algal Bloom Model Hurricane Intensity Model Coastal Inundation Model TemperatureSalinitySea LevelCurrentsColor Harmful Algal Bloom Model Integrated Ecosystem Assessment Hurricane Intensity Model Coastal Inundation Model External sources of 5 Core Variables (consistent with NOAA standards) TemperatureSalinitySea LevelCurrentsColor Systems Engineering: Data Access & Exchange REGIONAL COASTAL NEEDS Future State: Regional-coastal data integration for Regional scale data and information products and services Integrated Variable Ingest for Select Data Products Test & EvaluationBenchmarked Product Improvements for Operational Use Data Standards

IOOS Functional Areas Observing Systems Data Management and Communications (DMAC) Modeling and Analysis = Provider (observing system, data archive) = Customer (decision support tool, analysis tool, generic client) = Data Discovery, Access, Transport (encodings, metadata, catalog services, access services, conversion services)

Integration Challenge multiple, unique, independent data requests 10 distinct data requests 10 algorithms to convert to model format 10 different load functions Data provider changes have ripple effects Inefficient Costly processing Time consuming Stovepipe highlight

Integration Solution DIF content and transport standards/services

What is the Data Integrated Framework? Developing the IOOS Data Integration Framework First spiral of IOOS development –Core IOOS project Limited scope –DMAC functions Common data-sharing infrastructure Agreed-upon standards Aggregation services Web services –Implement at selected provider & customer sites –Focus on several core variables Standardize on small number of services & encodings –Adopt, adapt, expand, develop Evaluate in FY 2010 (Graphic by i3 Aerospace Technologies Pty Ltd – used with permission)

NOAA IOOS Data Integrated Framework Validate premise: data integration to measure improvements System engineering approach –Functional Requirements from identified customers –Concept of Operations Document –Design Document in progress Architectural principles –Service oriented architecture –Distributed services operated by data stewards –Adopt existing standards and specifications Goals and Objectives

Documentation Systems Engineering Approach NOAA IOOS Program Data Integration Framework (DIF) Master Project Plan Version 1.0 September 12, 2007 NOAA IOOS Program Data Integration Framework (DIF) Functional Requirements Document Version 1.0 September 17, 2007 NOAA IOOS Program Data Integration Framework (DIF) DIF As-Is Baseline Systems Document Version 1.0 September 19, 2007 NOAA IOOS Program Data Integration Framework (DIF) Concept of Operations Version 1.0 April 25, 2008 Master Project Plan Version 1.0 9/12/07 Functional Requirements Document Version 1.0 9/17/07 DIF As-Is Baseline Systems Document Version 1.0 9/19/07 Concept of Operations Version 1.0 4/25/08

NOAA IOOS DIF Office of National Environmental Satellite, Data, and Information Services (NESDIS) CoastWatch National Climate Data Center (NCDC) National Ocean Data Center (NODC) National Geophysical Data Center (NGDC) National Coastal Data Development Center (NCDDC) Office of Satellite Data Processing and Distribution (OSDPD) National Marine Fisheries Service Southwest Fisheries Science Center (SWFSC) National Ocean Service Coastal Services Center (CSC) National Centers for Coastal Ocean Science (NCCOS) Center for Operational Oceanographic Products and Services (CO-OPS) National Weather Service National Data Buoy Center (NDBC) National Centers for Environmental Prediction (NCEP) Oceanic and Atmospheric Research Pacific Marine Environmental Laboratory (PMEL)

IOOS Data Integration Framework Coordination Functional Requirements (completed) NESDIS/NCDDCNOS/CSC NWS/NDBCNMFS/SWFSC NESDIS/NGDCNWS/NCEP OAR/PMELNOS/CO-OPS Data Standards (completed) NESDIS/NCDDC NESDIS/NGDC OAR/PMEL NOS/CSC Integrated Products Team Chair – Charles Alexander NOAA IOOS Program National Weather Service (6) National Environmental Satellite Data and Information Service (6 + 1) National Marine Fisheries Service (2) National Ocean Service (9 + 3) Oceanic and Atmospheric Research (1 + 1) CONOPS (completed) NESDIS/ORA NMFS/SWFSC NWS/NDBC NOS/CSDL NOS/CSCNWS/NCEP NWS/EMC Web Services and Data Encodings Chair - Jeff de La Beaujardiere NOAA IOOS Program NESDIS (5)NWS (2) NOS (4)OAR (1) NMFS (1)RA’s (5) Image Matters Design NOS NWS NESDIS NOS NMFS METADATA NESDIS/NGDC NESDIS/NCDDC Harmful Algal Blooms NOS/NCCOS NOS/CO-OPS NWS/NDBC Coastal Inundation NWS/TPC NOS/CO-OPS

DIF Implementation Schedule Developing the National IOOS Data Integration Framework May Jun Jul Aug Sep Oct Nov Dec Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec Jan Feb Schema and IntegrationFinal Report May Jun Jul Aug Sep Oct Nov Dec Schema and Integration 5/12/2008 Deliver XML Schema NOAA IOOS Program 6/12/2008 Integrated Currents CO-OPS & NDBC 6/26/2008 DCS Color NCDDC 8/18/2008 Integrated Sea Level Temperature, Salinity, Winds, & Waves CO-OPS & NDBC 9/25/2008 Integrated Color CoastWatch 9/30/2008 Integrated HF Radar NDBC Global and Coastal Point Obs OSMC 10/31/2008 Integrated PORTS Currents (profile) CO-OPS 11/30/2008 WMS Services Available NDBC 12/31/2008 PORTS Survey Currents (profile) CO-OPS Global Profile Obs OSMC OSMC – Observing System Monitoring Center XML – Entensible Markup Language DCS – Data Content Standard WMS – Web Map Standard

IOOS Data Integration Framework Data Sources Gridded DataImagery Data KULA01 CWAO CREX++ T A008 D EUREKA //// //// In-Situ Data Need for three services! Open Geospatial Consortium (OSG) provides them.

16 Recommended Web Services and Data Encodings In-situ data (buoys, piers, towed sensors) Gridded data (model outputs, satellite) OGC Sensor Observation Service (SOS) OpenDAP and/or OGC Web Coverage Service (WCS) XML based on OGC Observations and Measurements (O&M) NetCDF using Climate and Forecast (CF) conventions Images of data OGC Web Map Service (WMS) GeoTIFF, PNG etc. -possibly with standardized styles OGC: Open Geospatial Consortium XML: Extensible Markup Language OPeNDAP: Open-source Project for a Network Data Access Protocol NetCDF: Network Common Data Format GeoTIFF: Geospatial Tagged Image File Format PNG: Portable Network Graphics Data TypeWeb ServiceEncoding

17 XML Encoding of In-Situ Data (SOS GetObservation Result) XML Extensible Markup Language Generic method for structuring text data OGC GML Open Geospatial Consortium Geography Markup Language XML that can represent any geospatial feature OGC O&M Observations and Measurements Model GML that describes the act of measuring real-world phenomena and the result of the measurement specialized by Data providers currently beta-testing this specification (

IOOS Data Integration Framework Sensor Observation Service NDBC is implementing SOS on a map interface page [ Data may be retrieved as: Comma separated values (CSV) NetCDF XML Data available: Sea level Currents Salinity Sea Temperature Winds & Waves

IOOS Data Integration Framework Sensor Observation Service – GetObservation for ocean current vertical profile

DIF Data Provider Status Expected by end CY2008 Sensor Observation Service NDBC DAC - buoys, C-MAN, VOS, NOS NERRS, etc. IOOS DAC - Regional observations TAO DAC - Tropical Atmosphere Ocean DART DAC - Deep-Ocean Assessment and Reporting of Tsunamis National Water Level Observation Network PORTS® NDBC CO-OPS User Requests Sensor Observation Service High-Frequency Radar (HFR) surface currents Web Coverage Service & OPeNDAP Data Content & Encoding Standards Web Coverage Service & OPeNDAP Satellite Ocean Color (Aqua MODIS, Terra MODIS, SeaWiFS) CoastWatch Data Content & Encoding Standards Data Integrated from One Provider NDBC Data Assembly Centers (DACs) IOOS Variables: temperature, salinity, currents, water level, winds, waves, ocean color (chlorophyll)

21 Additional Providers & Services IOOS Variables + other variables t.b.d. Single User Request Registry/Catalog Service Data Integration Service NDBC SOS WCS/OPeNDAP CO-OPS SOS Data Archives (NODC?) SOS CoastWatch WCS/OPeNDAP Region 1 SOS Region 2 WCS/OPeNDAP Agency 2 SOS Agency 1 WCS/OPeNDAP Data Integrated from Multiple Providers

Service Function Valid HTTP response on success and error Server load test Service Content XML is well-formed (obeys generic XML rules) XML is valid (obeys schema rules) XML matches Data Record Definition Service Operations Service can receive data queries Service can send data queries Data Validation Data returned matches query parameters Data complies with Abstract Data Content Standard Data is easily combined/integrated o expressions of geographic position are consistent o expressions of temporal position are consistent o expressions of measured parameters are consistent Controlling Documents OGC SOS Specification OGC O&M Specification Abstract Data Content Standard XML schema XML dictionaries Feature Type Record Definitions Verification and Validation

DIF Service Layers Target for FY2010 Data Providers NOAA Regional Partners Other Services Integration Translation Registry/Catalog Sensor Observation Service Web Coverage Service OpenDAP Web Map Service Access Services NOAA DSTs Customers COTS/ GIS General Public Other

Pathway to a National DMAC Developing the National IOOS Data Integration Framework

NOAA IOOS DIF Accomplishments 7 IOOS variables integrated by end of 2008 Focus on major NOAA data providers for greatest impact –Example: 83% of NOAA’s ocean temperature integrated and accessible in initial DIF implementation DIF Integrated Products Team (IPT) developing effective, cross-NOAA integration solution DIF implementation software and protocols will be made available to regional partners –NDBC provides 39 Regional temperature observations in August 2008 –Regions to begin implementation in FY09 Developing the IOOS Data Integration Framework

Regional Implementation

Regional Potential (GCOOS, SECOORA, Federal, Industry)

NOAA IOOS Data Integrated Framework Very short schedule driven by externals NDBC and CO-OPS are breaking new ground –Limited IOOS-specific reference material –Simultaneous implementation, testing, revision –Sensor metadata descriptions (SensorML) not ready Process needed for working with OGC, SSML, etc. to revise/harmonize/promote in FY09 Need for better documentation, tools, converters, clients, reference implementation, test suites Need more customers/users Future Challenges

Questions?

Back-up Slides

NOAA Coordination Integrated Products Team National Weather Service National Env. Satellite Data & Informtn. Service National Marine Fisheries Service National Ocean Service Oceanic and Atmospheric Research Chris TaylorWalter SmithHendrick Tolman Steve BaigAvichal MehraBill Burnett Ken CaseyKent HughesLewis McCulloch Julie BoschJennifer FryPhil Keegstra Tess BrandonKen McDonaldPaul Digiacomo Ted Haberman Roy MendelssohnJim SargentDave Foley Katie FisherJohn UlmerDaniel Martin Andrea HardyRebecca LoveKatie Fisher Shelley TomlinsonWhitley SaumweberRich Patchen John DandyJosh Pederson Jack Harlan Anne Ball (CSC/Ocean.US) Steve HankinGustavo GoniDerrick Snowden Chair, Charles Alexander – NOAA IOOS Program 11

Temperature Integration Build FY08 FY09 (w/ funding) Under investigation 83% 100% % of NOAA Integrated

33 Analyze options and determine optimal approach Define requirements National DMAC Development Path (based on NOAA, NASA DoD Decision Processes) Initial Concept Studies & Work of Others Functional & Technical Requirements Analysis of Alternatives (AoA) Economic Analysis of Preferred Alternative Capability Development Strategy NOAA Data Management Projects NSF Ocean Observing Initiative Regional Coastal Ocean Observing Systems Analysis of Existing Approaches Industry RFI Data Integration Framework (DIF) Use Cases Implementation Decision (~late FY2010) Federal SOA projects High-Level Functional Requirements Concept of Operations 26

IOOS Data Integration Framework Web Map Service (WMS) for Images of Data Map is a georeferenced “picture” of data GetCapabilities operation results in a “table of contents” in standardized format GetMap operation results in image of data customized according to: Variables of interest User-specified bounding box User-specified time Image size File format PNG, GetTIFF, JPEG, GIF May contain layers of data

IOOS Data Integration Framework Web Coverage Service for Gridded Data Coverage is an array of gridded data values Simplified viewpoint for this discussion, can be more complex GetCapabilities operation results in a “table of contents” GetCoverage operation results in a data file containing header and array(s) of numbers customized for: Variable of interest User-specified bounding box User-specified time File format netCDF, HDF, floating-point TIFF DescribeCoverage operation results in metadata about a specific dataset

IOOS Data Integration Framework Sensor Observation Service for in-situ Data Sampling feature discrete locations of measurements Point, vertical, or horizontal profile, trajectory … and time series or collections thereof GetCapabilities operation results in a “table of contents” GetObservation operation results in an XML data file containing observation values for desired: Variables of interest Bounding box or named geographic feature of interest or a single sensor Time DescribeSensor operation results in XML providing detailed information about a specific sensor

IOOS Data Integration Framework Sensor Observation Service Sensor Ocean Service (SOS) One of a family of standards and specifications that constitute the Open Geospatial Consortium (OGC), Inc. Sensor Web Enablement (SWE) activity An application programming interface (API) for providing sensor and observation data via extensible markup language (XML). Supports: GetCapabilities – This function returns metadata about this service DescribeSensor – The function returns detailed sensor characteristics GetObservation – This function returns observation data of the specified type for the stations within the specified station and date/time range. A bounding box will be added in the future. Currently, can get: Most recent observation Observation at a specific time Observation for a time range

IOOS Data Integration Framework Sensor Observation Service – GetObservation for ocean current vertical profile

Integration Challenge multiple, unique, independent data requests 10 distinct data requests 10 algorithms to convert to model format 10 different load functions Data provider changes have ripple effects Inefficient Costly processing Time consuming Stovepipe highlight

Integration Solution DIF content and transport standards/services DIF content and transport standards/services

Documentation: Systems Engineering Approach NOAA IOOS Program Data Integration Framework (DIF) Master Project Plan Version 1.0 September 12, 2007 NOAA IOOS Program Data Integration Framework (DIF) Functional Requirements Document Version 1.0 September 17, 2007 NOAA IOOS Program Data Integration Framework (DIF) DIF As-Is Baseline Systems Document Version 1.0 September 19, 2007 NOAA IOOS Program Data Integration Framework (DIF) Concept of Operations Version 1.0 April 25, 2008 Master Project Plan Version 1.0 9/12/07 Functional Requirements Document Version 1.0 9/17/07 DIF As-Is Baseline Systems Document Version 1.0 9/19/07 Concept of Operations Version 1.0 4/25/08 16

NOAA IOOS Data Integration Framework Systems Engineering & Standards Development Data Integration Framework (DIF) Integrated Ecosystem Assessment Month 36 Integration of 5 IOOS Core Variables ProductEnhancement Test & Evaluation Verification &Validation Integrated information services for NOAA programs Identify observation gaps Validated enhanced data products NOAA’s Data Integration Framework Enhanced decision support through: NOAA MISSION OBJECTIVES NOAA 5 Core Variables Months 0-12Month 18Month 24 Harmful Algal Bloom Model Hurricane Intensity Model Coastal Inundation Model TemperatureSalinitySea LevelCurrentsColor Harmful Algal Bloom Model Integrated Ecosystem Assessment Hurricane Intensity Model Coastal Inundation Model External sources of 5 Core Variables (consistent with NOAA standards) TemperatureSalinitySea LevelCurrentsColor Systems Engineering: Data Access & Exchange REGIONAL COASTAL NEEDS Future State: Regional-coastal data integration for Regional scale data and information products and services Integrated Variable Ingest for Select Data Products Test & EvaluationBenchmarked Product Improvements for Operational Use Data Standards