1. Integrated Ocean Observing System Data Management and Communications Sept 2003 The US Integrated Ocean Observing System (IOOS) Plan for Data Management and Communications (DMAC) IOOS DMAC Steering Committee

2. Sept 2003 Integrated Ocean Observing System Data Management and Communications The variety and complexity of ocean sciences observations creates great data management challenges

3. Sept 2003 Integrated Ocean Observing System Data Management and Communications IOOS: One system….seven goals –Detect and Predict Change –Mitigate natural hazards –Improve safety and efficiency of marine ops –Ensure national security –Reduce public health risks –Protect and restore marine ecosystems –Sustain marine resources

4. Sept 2003 Integrated Ocean Observing System Data Management and Communications “Achieving these goals depends upon the development of a system that efficiently links ocean observations to data management and analysis for timely delivery of environmental data and information” We need a Data Management and Communications Subsystem (DMAC)

5. June 2003 IOOS Data Management and Communications Subsystem Users Archive Centers Modeling On-line Browse ProductsForecasts Maps Regional Data Management Systems International Data Management Systems Terrestrial and Atmospheric Data Management Systems Metadata, Data Discovery and Data Transport Standards and Protocols Satellites Ships Primary Data Assembly & QC Hand Measurements Moorings Floats

6. June 2003 IOOS Data Management and Communications Subsystem Users Archive Centers Modeling On-line Browse Products Forecasts Maps Regional Data Management Systems International Data Management Systems Terrestrial and Atmospheric Data Management Systems Metadata, Data Discovery and Data Transport Standards and Protocols Satellites Ships Primary Data Assembly & QC Hand Measurements Moorings Floats

7. Sept 2003 Integrated Ocean Observing System Data Management and Communications JuneFeb DMAC Steering Committee Convened Expert team papers 1 st Draft Plan Publish DMAC Plan DMAC Timeline – Creating the Plan Mar Airlie workshop SepDec Requirements, schedules, & costs May “Expert” review Sept 20022003 --Metadata --Transport --Archive --Products June public review mtg

8. Sept 2003 Integrated Ocean Observing System Data Management and Communications IOOS-DMAC history 2000 - Ocean.US established; planning begins for development of a sustained and Integrated Ocean Observation System (IOOS) for US 2000 - Ocean.US established; planning begins for development of a sustained and Integrated Ocean Observation System (IOOS) for US March 2002 - Ocean.US National Workshop on US IOOS March 2002 - Ocean.US National Workshop on US IOOS June 2002 - The Data Management and Communications Steering Committee (DMAC-SC) convened by Ocean.US June 2002 - The Data Management and Communications Steering Committee (DMAC-SC) convened by Ocean.US May 2003 - external “expert” review of DMAC Plan May 2003 - external “expert” review of DMAC Plan June 2003 – public review meeting hosted by GoMOOS June 2003 – public review meeting hosted by GoMOOS September 2003 – Plan published September 2003 – Plan published

9. Sept 2003 Integrated Ocean Observing System Data Management and Communications What makes this a difficult problem? Like the classic “trilemma” … We can’t have all three

10. Sept 2003 Integrated Ocean Observing System Data Management and Communications …competing characteristics rule out the most straightforward solutions

11. Sept 2003 Integrated Ocean Observing System Data Management and Communications No simple data standard can be designed that all ocean data providers will utilize

12. Sept 2003 Integrated Ocean Observing System Data Management and Communications No management structure exists to enforce adoption of a complex, data management solution

13. Sept 2003 Integrated Ocean Observing System Data Management and Communications Data cannot be centrally managed at a single location

14. Sept 2003 Integrated Ocean Observing System Data Management and Communications The DMAC Plan is a framework for interoperability among independent, heterogeneous systems, large and small Participating organizations will manage their data independently Participating organizations will manage their data independently

15. Sept 2003 Integrated Ocean Observing System Data Management and Communications The value added by DMAC will be integration -- that many types of ocean data from different suppliers will be available in a uniform way Guiding Principles

16. Sept 2003 Integrated Ocean Observing System Data Management and Communications The methods utilized by data suppliers to deliver data to their “customers” need not change (FTP, subscription services, etc.) Guiding Principles

17. Sept 2003 Integrated Ocean Observing System Data Management and Communications However, a data provider may choose to utilize DMAC middleware as a cost effective way to reach a wide range of users. Guiding Principles

18. Sept 2003 Integrated Ocean Observing System Data Management and Communications Easy for users to find and access all data Easy for users to find and access all data Driven by user needs Driven by user needs Preserves all irreplaceable data Preserves all irreplaceable data Reliable, sustained, efficient operations Reliable, sustained, efficient operations Open design. Open standards process Open design. Open standards process Federated components acting as a whole

19. Sept 2003 Integrated Ocean Observing System Data Management and Communications DMAC requires participation from –All levels of management –Scientists –Technical staff Why doesn’t DMAC exist today? The greatest challenges are in the area of community outreach and organizational behavior rather than in technology!

20. Sept 2003 Integrated Ocean Observing System Data Management and Communications The DMAC Plan An evolutionary path An evolutionary path –A software engineering strategy –An inter-agency/international outlook Three parts Three parts –Part I – overview for general readers –Part II – formal requirements, milestones, tasks for technical planners –Part III – in-depth appendices on special topics

21. Sept 2003 Integrated Ocean Observing System Data Management and Communications The DMAC Plan Technology components Technology components –Data Discovery (metadata) –Data Transport –On-line Browse –Archive –Metrics, feedback and fault correction

22. Sept 2003 Integrated Ocean Observing System Data Management and Communications The DMAC Plan Data Discovery (metadata) Data Discovery (metadata) –Utilize FGDC (& future ISO) –A partnership of current and new metadata repositories –Metadata WG to design the strategy for distributed search –A Web Portal at which users can search Embed in other Web pages Embed in other Web pages Machine-to-machine accessible Machine-to-machine accessible –Seamless segue to DMAC On-Line Browse and Data Transport

23. Sept 2003 Integrated Ocean Observing System Data Management and Communications The DMAC Plan Data Transport (access) Data Transport (access) –“Web Services”  standardize the interface to the data –Adopt current successes OPeNDAP (a.k.a. “DODS” or “NVODS”) OPeNDAP (a.k.a. “DODS” or “NVODS”) Discipline neutral Discipline neutral Potentially interoperable with GIS components Potentially interoperable with GIS components Adapts to current and future computer applications Adapts to current and future computer applications –WG must define comprehensive semantic data model(s)

24. Sept 2003 Integrated Ocean Observing System Data Management and Communications OPeNDAP: distributed access to data and metadata

25. Sept 2003 Integrated Ocean Observing System Data Management and Communications The DMAC Plan On-line Browse On-line Browse –Uniform access to all IOOS data from Web browsers maps (“GIS”) & basic scientific graphics maps (“GIS”) & basic scientific graphics readable tables readable tables –Target audience is IOOS scientific managers (also useful to many others) –Utilize Data Transport –Live Access Server (LAS) is an initial “pre- operational” component

26. Sept 2003 Integrated Ocean Observing System Data Management and Communications The DMAC Plan Archive Archive –Build upon partnership of National and Regional archive centers –Archive WG to be formed establish DMAC archive policies establish DMAC archive policies ensure that all data have an archive plan ensure that all data have an archive plan –Utilize DMAC Metadata and Transport standards –(new) Operational data plus QC to be archived (in coordination with data assembly centers such as US GODAE server and NOAA/NDBC)

27. Sept 2003 Integrated Ocean Observing System Data Management and Communications The DMAC Plan Data products (*) (IOOS Modeling and Analysis Subsystem) Data products (*) (IOOS Modeling and Analysis Subsystem) State estimation State estimation Forecasts Forecasts Information products (*) Information products (*) Value-added products for user groups provided by the private sector, academe, government Value-added products for user groups provided by the private sector, academe, government * Not part of the DMAC Plan

28. Sept 2003 Integrated Ocean Observing System Data Management and Communications The DMAC Plan Feedback and Fault Correction Feedback and Fault Correction –DMAC Web portal will invite user feedback –DMAC to maintain a “Help desk” –Performance metrics gathered automatically –Oversight by the DMAC Steering Committee Continual, active outreach to users Continual, active outreach to users

29. Sept 2003 Integrated Ocean Observing System Data Management and Communications Implementation of IOOS has already begun New funding possible by FY06 New funding possible by FY06 Key pieces of DMAC must be in place by that time Key pieces of DMAC must be in place by that time –Interim metadata standards (incl. semantic data model) –Distributed data discovery architecture –Voids to be filled in the data transport toolkit –Initial responsibilities clear among Archive centers Must fund these developments using current mechanisms! Must fund these developments using current mechanisms! (See details in Part I, Section 4 of DMAC Plan) Part I, Section 4Part I, Section 4

30. Sept 2003 Integrated Ocean Observing System Data Management and Communications DMAC Plan published DMAC Phased ImplementationPhased Implementation Sept 2003 ‘04‘05‘06‘07 Immediate Priorities Metadata Standards (W.G.) GIS&OBIS Data Transport integration Archive strategy (W.G.) Software engineering oversight ‘08‘09‘10 $81M$2.3M Five year implementation Plan Standards and Tool Development ($30M) Hardware & Software for networking, data assemb. & archive ($37M) Pilot projects ($14M) minimally functioning initial system IOC

31. Sept 2003 Integrated Ocean Observing System Data Management and Communications NVODS and DMAC Use of OPeNDAP (NVODS) was not a guaranteed outcome of the DMAC planning process Use of OPeNDAP (NVODS) was not a guaranteed outcome of the DMAC planning process –It was selected for DMAC Data Transport because the NVODS project anticipated the DMAC analysis of the IOOS “trilemma” – the need for a “framework for interoperability”. trilemma

32. Sept 2003 Integrated Ocean Observing System Data Management and Communications NVODS and DMAC OPeNDAP must be made ready for the FY06 rapid expansion of IOOS. GIS compatibility GIS compatibility OBIS (biological) compatibility OBIS (biological) compatibility delimited files (ASCII) delimited files (ASCII) OPeNDAP Development Kit (ODK) – an easily adaptable server core OPeNDAP Development Kit (ODK) – an easily adaptable server core

33. Sept 2003 Integrated Ocean Observing System Data Management and Communications NVODS and DMAC A semantic data model A semantic data model –need leadership from NVODS A stable, published spec (v4 -- XML) A stable, published spec (v4 -- XML) Expanded resource control framework Expanded resource control framework –Restrict data volumes, cpu cycles, privs, … Expanded “web service” infrastructure Expanded “web service” infrastructure –Query resource limits

34. Sept 2003 Integrated Ocean Observing System Data Management and Communications NVODS and DMAC Improved Sequence data support “pilot”  “pre-op” Improved Sequence data support “pilot”  “pre-op” –additional client options perl, python, “translation”, jdbc (?), … perl, python, “translation”, jdbc (?), … –usability conventions (metadata) min/max ranges, size guidelines, … min/max ranges, size guidelines, … –much more hands-on experience further forms of aggregation further forms of aggregation –in-situ collections of profiles and time series, 5D and 6D forecast ensembles

35. Sept 2003 Integrated Ocean Observing System Data Management and Communications NVODS and DMAC Hard work remains to be done, but with a sustained effort NVODS/OPeNDAP can succeed NVODS/OPeNDAP can succeed

36. Sept 2003 Integrated Ocean Observing System Data Management and Communications Questions? Click Click for immediate actions recommended for IOOS data providers

37. Sept 2003 Integrated Ocean Observing System Data Management and Communications Concrete actions to contribute data into the IOOS/DMAC framework today Data discovery Data discovery 1.Produce FGDC metadata 2.Submit it to national search sites (NASA/GCMD, NOAA/NCDDC & CSC) 3.Create a home page and a search portal to your data holdings Interoperable data access Interoperable data access On-line browse On-line browse

38. Sept 2003 Integrated Ocean Observing System Data Management and Communications Concrete actions … Data discovery Data discovery Interoperable data access Interoperable data access 1.Gridded data: – Install OPeNDAP (*) servers 2.Time series: – Consider partnership with NOAA/NDBC (**) 3.In-situ data: – Use RDBMS or standard formats – Contribute to development of pilot IOOS/DMAC components – “DRDS”, “A.S.” On-line browse On-line browse (*) http://www.unidata.ucar.edu/packages/dods/ http://www.unidata.ucar.edu/packages/dods/ (**) Dave.Gilhousen@noaa.gov, (228)-688-2840

39. Sept 2003 Integrated Ocean Observing System Data Management and Communications Concrete actions … Data discovery Data discovery Interoperable data access Interoperable data access On-line browse On-line browse 1.Install Live Access Server (*) utilizing DMAC/OPeNDAP data transport 2.Create custom Web sites that demonstrate regional data integration (*) http://www.ferret.noaa.gov/LAS/

40. Sept 2003 Integrated Ocean Observing System Data Management and Communications Questions?

41. Sept 2003 Integrated Ocean Observing System Data Management and Communications

42. Sept 2003 Integrated Ocean Observing System Data Management and Communications

43. Sept 2003 Integrated Ocean Observing System Data Management and Communications

44. Sept 2003 Integrated Ocean Observing System Data Management and Communications