Integrating ocean climate observations Perspectives for consideration by the COSC Steve Hankin, NOAA/PMEL

Oct '10 OCO Annual Review 2 What do we mean by integrated data management, anyway? iTunes ® as a metaphor... (songs  datasets)

Oct '10 OCO Annual Review 3 Mel Briscoe, ONR program manager, speaking about integrated data access: “I’m not worried about the research community. They will figure out a way to get the data they need. ” A comment on the culture of scientists …

Oct '10 OCO Annual Review 4 Informal guidance to the DMAC Steering Committee (Ocean.US 2002): “The financial industry has managed a successful integration of its data. What can we learn from them in order to integrate ocean data?” Lets look at this from a cultural perspective. (irony aside )

Oct '10 OCO Annual Review 5 Imagine a world of finance organized as the world of ocean science is … Many small, independent banks Bank managers often cede financial data management planning to department heads: –loan dept, mortgage dept, checking/savings dept., investments, …

Oct '10 OCO Annual Review 6 What habits for exchanging financial data would evolve? Ad hoc means would dominate initially (Excel spreadsheets and ASCII tables) Some departments would develop protocols for exchange with counterparts at other banks

Oct '10 OCO Annual Review 7 How would we improve integration of data in such a world? Difficult. Departments have developed systems that work adequately Making changes would be disruptive to achieving short term financial (read “scientific”) goals. Ref: The Innovator’s Dilemma, Clayton M. Christensen

Oct '10 OCO Annual Review 8 How banking reality made integration easier A big bank chain needs integration internally –Creates an IT divisions to achieve it Integration between big chains becomes easier: –have a “culture of integration” management sees gains technical IT staff share outlooks users rely upon it

Oct '10 OCO Annual Review 9 Technology alone is not the solution; it is just one part of a strategy.

Oct '10 OCO Annual Review 10 What is the state of our “culture of integration” today? If we’re serious about developing “Climate Services” then integration is no longer optional.

Oct '10 OCO Annual Review 11 The state of the ocean/climate “culture of integration” Weak linkages across data life cycle: –realtime delivery systems –scientific data assembly centers –archives Weak linkages across data types –platforms: ships v. profilers v. drifters v. moorings, … –modeling v. obs, … –“science data” v. end user data (geo- political, commerce, …)

Oct '10 OCO Annual Review 12 A cross-platform culture of integration is developing … Argo and OceanSites are converging on interoperable formats using netCDF-CF But clear opportunities remain untapped: –Argo successes have not become the basis of a strategy for other profile data -- XBTs, CTDs, …. –OceanSites successes have not become a basis for tide gauges –drifters, cruise tracks, gliders, CPR, etc. do not exploit opportunities to share standards

Oct '10 OCO Annual Review 13 Assertion: The senior scientists largely define the culture of science communities –advise program managers –influence colleagues

Oct '10 OCO Annual Review Applause for adding a data integration hat on the COSC ! (Ken Casey) 1. Be a voice for data integration in strategic discussions (e.g. NOAA strategic plan, NOAA matrix management) 2. Coach management in creating a “culture of integration” 3. Promote specific integration goals for metadata and data access How can the COSC help to foster a “culture of integration”?

Oct '10 OCO Annual Review Coach management on creating a “culture of integration” Funding opportunities that encourage scientists and software developers to work together Give greater weight to contributions of data and data systems – not just of publications “… the need for new partnerships and collaborations among domain scientists […] and data scientists [as well as] infrastructure development to support digital data; and the need for sustainable economic models to support [it]” NSB, 2007

Oct '10 OCO Annual Review Metadata integration RT metadata is incomplete by its nature (QC, fall rate, …) RT metadata is incomplete by its nature (QC, fall rate, …)  strive for “good enuf” RT metadata (BUFR 2012) Metadata servers are here to stay Metadata servers are here to stay  the sensor’s ID is the “=“ that creates a solvable equation

Oct '10 OCO Annual Review Metadata integration Define the services Service(ID)  theMetadata Define the services Service(ID)  theMetadata See OceanSites examples. Extend the success to Argo, tide gauges, XBTs, drifters, … See OceanSites examples. Extend the success to Argo, tide gauges, XBTs, drifters, … For help with the metadata alphabet soup (ISO 19115, SensorML, FGDC, CSW, ) look to IT partners: NDBC and Ted Habermann. For help with the metadata alphabet soup (ISO 19115, SensorML, FGDC, CSW, ) look to IT partners: NDBC and Ted Habermann.

Oct '10 OCO Annual Review Integrated Data Access Evolving …. Don’t look for a single, dominant service any time soon. Evolving …. Don’t look for a single, dominant service any time soon. Support multiple access techniques. Support multiple access techniques. Prioritize based upon which services are Prioritize based upon which services are –popular with users AND –illustrate a clear vision

Oct '10 OCO Annual Review Integrated Data Access A clear priority: netCDF-CF-OPeNDAP-THREDDS A clear priority: netCDF-CF-OPeNDAP-THREDDS OceanSites and Argo converging. OceanSites and Argo converging. Still needing plans: tide gauges, XBT’s, drifters, hydrographic,… Still needing plans: tide gauges, XBT’s, drifters, hydrographic,… Manage the progress ! Manage the progress ! For help look to OSMC and … For help look to OSMC and …

Oct '10 OCO Annual Review 20 Oct NOAA/UAF-grid The NOAA UAF project (Unified Access Framework) : OCO data assembly centers: make the data available UAF: connect it to the scientists’ favorite “iTunes” (w/ volunteer energy)

Oct '10 OCO Annual Review 21 Wrap up Building a “culture of integration” will take time Building a “culture of integration” will take time –the priorities scientists set and their willingness to use their influence is the key Immediate data management priorities: Immediate data management priorities: –complete the OceanSites & Argo convergence on netCDF-CF-DAP with rich metadata –continue similarly to tide gauges, XBTs, drifters, hydro, … –Migrate RT to BUFR by 2012 with ironclad IDs.

Oct '10 OCO Annual Review 22 Conclusion “Management of data should have a status in a science organization that is as fundamental and unquestioned as management of personnel and management of budgets.“ “NOAA is, at its foundation, an environmental information generating organization …” NGSP 2010

Oct '10 OCO Annual Review 23 Thank you