1. NOAA Integrated Ocean Observing System (IOOS) Data Integration Framework Richard Crout NOAA NWS National Data Buoy Center GCOOS Board of Directors Meeting Corpus Christi, TX 19 August 2008

2. Outline NOAA’s Integrated Ocean Observing System Data Integrated Framework –Challenge –Solution - DIF –Goals and Objectives –Approach –Coordination –Web Services –Status –Validation and Verification Regional Implementation Future Challenges

3. NOAA IOOS … Is modernizing the way NOAA collects, shares, and uses ocean information Is increasing data interoperability across NOAA and the regions and efficiency of operations Is expediting access to data for improved decision making  To achieve these objectives, the Program focuses its contributions in two main areas: 1.Developing the IOOS Data Integration Framework (NOAA DIF-to-National DMAC) 2.Managing the regional IOOS partnership

4. NOAA IOOS Program Focus Standards –Develop and maintain a process to document, coordinate, evaluate and accept data standards Regions –Invest in Regional Associations efforts that contribute to the definition and implementation of IOOS Data Integration Framework (DIF) –Conduct a limited scope implementation of IOOS within NOAA as a proof of principle and to gather lessons to inform the larger IOOS effort

5. Data Integration Framework (DIF) 5-4-12 NOAA IOOS Data Integration Framework Systems Engineering & Standards Development Integrated Ecosystem Assessment Month 36 Integration of 5 IOOS Core Variables ProductEnhancement Test & Evaluation Verification &Validation Integrated information services for NOAA programs Identify observation gaps Validated enhanced data products NOAA’s Data Integration Framework Enhanced decision support through: NOAA MISSION OBJECTIVES NOAA 5 Core Variables Months 0-12Month 18Month 24 Harmful Algal Bloom Model Hurricane Intensity Model Coastal Inundation Model TemperatureSalinitySea LevelCurrentsColor Harmful Algal Bloom Model Integrated Ecosystem Assessment Hurricane Intensity Model Coastal Inundation Model External sources of 5 Core Variables (consistent with NOAA standards) TemperatureSalinitySea LevelCurrentsColor Systems Engineering: Data Access & Exchange REGIONAL COASTAL NEEDS Future State: Regional-coastal data integration for Regional scale data and information products and services Integrated Variable Ingest for Select Data Products Test & EvaluationBenchmarked Product Improvements for Operational Use Data Standards

6. IOOS Functional Areas Observing Systems Data Management and Communications (DMAC) Modeling and Analysis = Provider (observing system, data archive) = Customer (decision support tool, analysis tool, generic client) = Data Discovery, Access, Transport (encodings, metadata, catalog services, access services, conversion services)

7. Integration Challenge multiple, unique, independent data requests 10 distinct data requests 10 algorithms to convert to model format 10 different load functions Data provider changes have ripple effects Inefficient Costly processing Time consuming Stovepipe highlight

8. Integration Solution DIF content and transport standards/services

9. What is the Data Integrated Framework? Developing the IOOS Data Integration Framework First spiral of IOOS development –Core IOOS project Limited scope –DMAC functions Common data-sharing infrastructure Agreed-upon standards Aggregation services Web services –Implement at selected provider & customer sites –Focus on several core variables Standardize on small number of services & encodings –Adopt, adapt, expand, develop Evaluate in FY 2010 (Graphic by i3 Aerospace Technologies Pty Ltd – used with permission)

10. NOAA IOOS Data Integrated Framework Validate premise: data integration to measure improvements System engineering approach –Functional Requirements from identified customers –Concept of Operations Document –Design Document in progress Architectural principles –Service oriented architecture –Distributed services operated by data stewards –Adopt existing standards and specifications Goals and Objectives

11. Documentation Systems Engineering Approach NOAA IOOS Program Data Integration Framework (DIF) Master Project Plan Version 1.0 September 12, 2007 NOAA IOOS Program Data Integration Framework (DIF) Functional Requirements Document Version 1.0 September 17, 2007 NOAA IOOS Program Data Integration Framework (DIF) DIF As-Is Baseline Systems Document Version 1.0 September 19, 2007 NOAA IOOS Program Data Integration Framework (DIF) Concept of Operations Version 1.0 April 25, 2008 Master Project Plan Version 1.0 9/12/07 Functional Requirements Document Version 1.0 9/17/07 DIF As-Is Baseline Systems Document Version 1.0 9/19/07 Concept of Operations Version 1.0 4/25/08

12. NOAA IOOS DIF Office of National Environmental Satellite, Data, and Information Services (NESDIS) CoastWatch National Climate Data Center (NCDC) National Ocean Data Center (NODC) National Geophysical Data Center (NGDC) National Coastal Data Development Center (NCDDC) Office of Satellite Data Processing and Distribution (OSDPD) National Marine Fisheries Service Southwest Fisheries Science Center (SWFSC) National Ocean Service Coastal Services Center (CSC) National Centers for Coastal Ocean Science (NCCOS) Center for Operational Oceanographic Products and Services (CO-OPS) National Weather Service National Data Buoy Center (NDBC) National Centers for Environmental Prediction (NCEP) Oceanic and Atmospheric Research Pacific Marine Environmental Laboratory (PMEL)

13. IOOS Data Integration Framework Coordination Functional Requirements (completed) NESDIS/NCDDCNOS/CSC NWS/NDBCNMFS/SWFSC NESDIS/NGDCNWS/NCEP OAR/PMELNOS/CO-OPS Data Standards (completed) NESDIS/NCDDC NESDIS/NGDC OAR/PMEL NOS/CSC Integrated Products Team Chair – Charles Alexander NOAA IOOS Program National Weather Service (6) National Environmental Satellite Data and Information Service (6 + 1) National Marine Fisheries Service (2) National Ocean Service (9 + 3) Oceanic and Atmospheric Research (1 + 1) CONOPS (completed) NESDIS/ORA NMFS/SWFSC NWS/NDBC NOS/CSDL NOS/CSCNWS/NCEP NWS/EMC Web Services and Data Encodings Chair - Jeff de La Beaujardiere NOAA IOOS Program NESDIS (5)NWS (2) NOS (4)OAR (1) NMFS (1)RA’s (5) Image Matters Design NOS NWS NESDIS NOS NMFS METADATA NESDIS/NGDC NESDIS/NCDDC Harmful Algal Blooms NOS/NCCOS NOS/CO-OPS NWS/NDBC Coastal Inundation NWS/TPC NOS/CO-OPS

14. DIF Implementation Schedule Developing the National IOOS Data Integration Framework May Jun Jul Aug Sep Oct Nov Dec Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec Jan Feb 2008 2009 2010 Schema and IntegrationFinal Report May Jun Jul Aug Sep Oct Nov Dec Schema and Integration 5/12/2008 Deliver XML Schema NOAA IOOS Program 6/12/2008 Integrated Currents CO-OPS & NDBC 6/26/2008 DCS Color NCDDC 8/18/2008 Integrated Sea Level Temperature, Salinity, Winds, & Waves CO-OPS & NDBC 9/25/2008 Integrated Color CoastWatch 9/30/2008 Integrated HF Radar NDBC Global and Coastal Point Obs OSMC 10/31/2008 Integrated PORTS Currents (profile) CO-OPS 11/30/2008 WMS Services Available NDBC 12/31/2008 PORTS Survey Currents (profile) CO-OPS Global Profile Obs OSMC OSMC – Observing System Monitoring Center XML – Entensible Markup Language DCS – Data Content Standard WMS – Web Map Standard

15. IOOS Data Integration Framework Data Sources Gridded DataImagery Data KULA01 CWAO 051800 CREX++ T000101 A008 D09040++ 71 917 EUREKA 7598 -08593 00010 18 1998 04 29 23 18 061 019 //// //// 375 0082 0000 400 10137 030 0000 200 10000 030 0001 002 09687 037 0002 002 09366 033 0004 002 08831 037 0005 200 08500 036 0007 002 08013 043 0007 002 07881 047 0008 002 07646 037 0009 002 07442 042 0011 200 07000 031 0012 002 06849 027 0013 002 06710 036 0015 002 06291 029 0022 200 05000 028 0025 002 04557 027 0029 002 04065 024 0029 200 04000 020 0032 002 03626 025 0038 002 03000 020 0040 002 02890 021 0040 002 02829 065 0041 002 02726 105 0043 002 02576 118 0044 200 02500 135 0048 002 02218 165 0049 002 02147 161 0050 002 02104 171 0051 002 02031 153 0051 002 02010 159 0051 200 02000 171 0052 002 01941 188 0054 002 01854 198 0056 002 01744 187 0056 002 01717 194 0057 002 01683 191 0058 002 01640 161 0058 002 01623 159 0059 002 01585 168 0059 002 01576 185 0060 002 01545 197 0061 002 01500 202 In-Situ Data Need for three services! Open Geospatial Consortium (OSG) provides them.

16. 16 Recommended Web Services and Data Encodings In-situ data (buoys, piers, towed sensors) Gridded data (model outputs, satellite) OGC Sensor Observation Service (SOS) OpenDAP and/or OGC Web Coverage Service (WCS) XML based on OGC Observations and Measurements (O&M) NetCDF using Climate and Forecast (CF) conventions Images of data OGC Web Map Service (WMS) GeoTIFF, PNG etc. -possibly with standardized styles OGC: Open Geospatial Consortium XML: Extensible Markup Language OPeNDAP: Open-source Project for a Network Data Access Protocol NetCDF: Network Common Data Format GeoTIFF: Geospatial Tagged Image File Format PNG: Portable Network Graphics Data TypeWeb ServiceEncoding

17. 17 XML Encoding of In-Situ Data (SOS GetObservation Result) XML Extensible Markup Language Generic method for structuring text data OGC GML Open Geospatial Consortium Geography Markup Language XML that can represent any geospatial feature OGC O&M Observations and Measurements Model GML that describes the act of measuring real-world phenomena and the result of the measurement specialized by Data providers currently beta-testing this specification (http://www.csc.noaa.gov/ioos/schema/IOOS-DIF/)http://www.csc.noaa.gov/ioos/schema/IOOS-DIF/

18. IOOS Data Integration Framework Sensor Observation Service NDBC is implementing SOS on a map interface page [ http://sdf.ndbc.noaa.gov/] Data may be retrieved as: Comma separated values (CSV) NetCDF XML Data available: Sea level Currents Salinity Sea Temperature Winds & Waves

19. IOOS Data Integration Framework Sensor Observation Service – GetObservation for ocean current vertical profile

20. DIF Data Provider Status Expected by end CY2008 Sensor Observation Service NDBC DAC - buoys, C-MAN, VOS, NOS NERRS, etc. IOOS DAC - Regional observations TAO DAC - Tropical Atmosphere Ocean DART DAC - Deep-Ocean Assessment and Reporting of Tsunamis National Water Level Observation Network PORTS® NDBC CO-OPS User Requests Sensor Observation Service High-Frequency Radar (HFR) surface currents Web Coverage Service & OPeNDAP Data Content & Encoding Standards Web Coverage Service & OPeNDAP Satellite Ocean Color (Aqua MODIS, Terra MODIS, SeaWiFS) CoastWatch Data Content & Encoding Standards Data Integrated from One Provider NDBC Data Assembly Centers (DACs) IOOS Variables: temperature, salinity, currents, water level, winds, waves, ocean color (chlorophyll)

21. 21 Additional Providers & Services IOOS Variables + other variables t.b.d. Single User Request Registry/Catalog Service Data Integration Service NDBC SOS WCS/OPeNDAP CO-OPS SOS Data Archives (NODC?) SOS CoastWatch WCS/OPeNDAP Region 1 SOS Region 2 WCS/OPeNDAP Agency 2 SOS Agency 1 WCS/OPeNDAP Data Integrated from Multiple Providers

22. Service Function Valid HTTP response on success and error Server load test Service Content XML is well-formed (obeys generic XML rules) XML is valid (obeys schema rules) XML matches Data Record Definition Service Operations Service can receive data queries Service can send data queries Data Validation Data returned matches query parameters Data complies with Abstract Data Content Standard Data is easily combined/integrated o expressions of geographic position are consistent o expressions of temporal position are consistent o expressions of measured parameters are consistent Controlling Documents OGC SOS Specification OGC O&M Specification Abstract Data Content Standard XML schema XML dictionaries Feature Type Record Definitions Verification and Validation

23. DIF Service Layers Target for FY2010 Data Providers NOAA Regional Partners Other Services Integration Translation Registry/Catalog Sensor Observation Service Web Coverage Service OpenDAP Web Map Service Access Services NOAA DSTs Customers COTS/ GIS General Public Other

24. Pathway to a National DMAC Developing the National IOOS Data Integration Framework

25. NOAA IOOS DIF Accomplishments 7 IOOS variables integrated by end of 2008 Focus on major NOAA data providers for greatest impact –Example: 83% of NOAA’s ocean temperature integrated and accessible in initial DIF implementation DIF Integrated Products Team (IPT) developing effective, cross-NOAA integration solution DIF implementation software and protocols will be made available to regional partners –NDBC provides 39 Regional temperature observations in August 2008 –Regions to begin implementation in FY09 Developing the IOOS Data Integration Framework

26. Regional Implementation

27. Regional Potential (GCOOS, SECOORA, Federal, Industry)

28. NOAA IOOS Data Integrated Framework Very short schedule driven by externals NDBC and CO-OPS are breaking new ground –Limited IOOS-specific reference material –Simultaneous implementation, testing, revision –Sensor metadata descriptions (SensorML) not ready Process needed for working with OGC, SSML, etc. to revise/harmonize/promote in FY09 Need for better documentation, tools, converters, clients, reference implementation, test suites Need more customers/users Future Challenges

29. Questions? http://ioos.noaa.gov

30. Back-up Slides

31. NOAA Coordination Integrated Products Team National Weather Service National Env. Satellite Data & Informtn. Service National Marine Fisheries Service National Ocean Service Oceanic and Atmospheric Research Chris TaylorWalter SmithHendrick Tolman Steve BaigAvichal MehraBill Burnett Ken CaseyKent HughesLewis McCulloch Julie BoschJennifer FryPhil Keegstra Tess BrandonKen McDonaldPaul Digiacomo Ted Haberman Roy MendelssohnJim SargentDave Foley Katie FisherJohn UlmerDaniel Martin Andrea HardyRebecca LoveKatie Fisher Shelley TomlinsonWhitley SaumweberRich Patchen John DandyJosh Pederson Jack Harlan Anne Ball (CSC/Ocean.US) Steve HankinGustavo GoniDerrick Snowden Chair, Charles Alexander – NOAA IOOS Program 11

32. Temperature Integration Build FY08 FY09 (w/ funding) Under investigation 83% 100% % of NOAA Integrated

33. 33 Analyze options and determine optimal approach Define requirements National DMAC Development Path (based on NOAA, NASA DoD Decision Processes) Initial Concept Studies & Work of Others Functional & Technical Requirements Analysis of Alternatives (AoA) Economic Analysis of Preferred Alternative Capability Development Strategy NOAA Data Management Projects NSF Ocean Observing Initiative Regional Coastal Ocean Observing Systems Analysis of Existing Approaches Industry RFI Data Integration Framework (DIF) Use Cases Implementation Decision (~late FY2010) Federal SOA projects High-Level Functional Requirements Concept of Operations 26

34. IOOS Data Integration Framework Web Map Service (WMS) for Images of Data Map is a georeferenced “picture” of data GetCapabilities operation results in a “table of contents” in standardized format GetMap operation results in image of data customized according to: Variables of interest User-specified bounding box User-specified time Image size File format PNG, GetTIFF, JPEG, GIF May contain layers of data

35. IOOS Data Integration Framework Web Coverage Service for Gridded Data Coverage is an array of gridded data values Simplified viewpoint for this discussion, can be more complex GetCapabilities operation results in a “table of contents” GetCoverage operation results in a data file containing header and array(s) of numbers customized for: Variable of interest User-specified bounding box User-specified time File format netCDF, HDF, floating-point TIFF DescribeCoverage operation results in metadata about a specific dataset

36. IOOS Data Integration Framework Sensor Observation Service for in-situ Data Sampling feature discrete locations of measurements Point, vertical, or horizontal profile, trajectory … and time series or collections thereof GetCapabilities operation results in a “table of contents” GetObservation operation results in an XML data file containing observation values for desired: Variables of interest Bounding box or named geographic feature of interest or a single sensor Time DescribeSensor operation results in XML providing detailed information about a specific sensor

37. IOOS Data Integration Framework Sensor Observation Service Sensor Ocean Service (SOS) One of a family of standards and specifications that constitute the Open Geospatial Consortium (OGC), Inc. Sensor Web Enablement (SWE) activity An application programming interface (API) for providing sensor and observation data via extensible markup language (XML). Supports: GetCapabilities – This function returns metadata about this service DescribeSensor – The function returns detailed sensor characteristics GetObservation – This function returns observation data of the specified type for the stations within the specified station and date/time range. A bounding box will be added in the future. Currently, can get: Most recent observation Observation at a specific time Observation for a time range

38. IOOS Data Integration Framework Sensor Observation Service – GetObservation for ocean current vertical profile

40. Integration Challenge multiple, unique, independent data requests 10 distinct data requests 10 algorithms to convert to model format 10 different load functions Data provider changes have ripple effects Inefficient Costly processing Time consuming Stovepipe highlight

41. Integration Solution DIF content and transport standards/services DIF content and transport standards/services

42. Documentation: Systems Engineering Approach NOAA IOOS Program Data Integration Framework (DIF) Master Project Plan Version 1.0 September 12, 2007 NOAA IOOS Program Data Integration Framework (DIF) Functional Requirements Document Version 1.0 September 17, 2007 NOAA IOOS Program Data Integration Framework (DIF) DIF As-Is Baseline Systems Document Version 1.0 September 19, 2007 NOAA IOOS Program Data Integration Framework (DIF) Concept of Operations Version 1.0 April 25, 2008 Master Project Plan Version 1.0 9/12/07 Functional Requirements Document Version 1.0 9/17/07 DIF As-Is Baseline Systems Document Version 1.0 9/19/07 Concept of Operations Version 1.0 4/25/08 16

43. NOAA IOOS Data Integration Framework Systems Engineering & Standards Development Data Integration Framework (DIF) Integrated Ecosystem Assessment Month 36 Integration of 5 IOOS Core Variables ProductEnhancement Test & Evaluation Verification &Validation Integrated information services for NOAA programs Identify observation gaps Validated enhanced data products NOAA’s Data Integration Framework Enhanced decision support through: NOAA MISSION OBJECTIVES NOAA 5 Core Variables Months 0-12Month 18Month 24 Harmful Algal Bloom Model Hurricane Intensity Model Coastal Inundation Model TemperatureSalinitySea LevelCurrentsColor Harmful Algal Bloom Model Integrated Ecosystem Assessment Hurricane Intensity Model Coastal Inundation Model External sources of 5 Core Variables (consistent with NOAA standards) TemperatureSalinitySea LevelCurrentsColor Systems Engineering: Data Access & Exchange REGIONAL COASTAL NEEDS Future State: Regional-coastal data integration for Regional scale data and information products and services Integrated Variable Ingest for Select Data Products Test & EvaluationBenchmarked Product Improvements for Operational Use Data Standards