DMAC Data Integration What is it really? Why does it seem frozen in place? How do we get it moving? Steve Hankin (NOAA/PMEL) DMAC = Data Management and Communications subsystem of the US Integrated Ocean Observing System (IIOS) []

June '07 OCO Annual Review 2 Part 1. A Short Digression (begging your indulgence …) What’s new in the Observing System Monitoring Center (OSMC)

June '07 OCO Annual Review 3

June '07 OCO Annual Review 4

June '07 OCO Annual Review 5 under the hood … Metadata feeds from NOAAPort & GODAE GODAE QC fields to be added next … A feed from NCEP ? Goal: –Compare QC strategies. –Compare GTS filters and feeds.

June '07 OCO Annual Review 6 Part 2. DMAC Data Integration (DMAC = Data Management and Communications subsystem of IOOS) Just what is DMAC “data integration” ? (and what is it not ?) Start with a taxonomy thru examples … What is it really? Why does it seem frozen in place? How do we get it moving?

June '07 OCO Annual Review 7 What is “DMAC integration”? Case study 1: Numerical Weather Prediction Consider FNMOC: Pull observations from GTS Convert disparate formats to single format Apply global QC  An “integrated” data product for assimilation.

June '07 OCO Annual Review 8 Surely this is “integration” … but it is only available to one project –Integration for a narrowly focused purpose Call this “project integration” Note: GODAE Server distributes the FNMOC product … a step in the right direction … a step in the right direction

June '07 OCO Annual Review 9 What is “DMAC integration”? Case study 2: Web Theme Pages Pull together images, documents & links Put care into presentation. “Friendly”

June '07 OCO Annual Review 10 Surely this is “integration” NO! This is a useful service. But it is not data integration.

June '07 OCO Annual Review 11 What is “DMAC integration”? Case study 3: Argo program Internationally planned A single agreed upon format Data openly shared QC and metadata carefully managed Distribution infrastructure (DACs & GDACs)

June '07 OCO Annual Review 12 Surely this is “integration” … but it is only applicable to one platform –Unique formats & distribution infrastructure Call this “platform integration” “Project integration” and “platform integration” are not the concept we are after in DMAC.

June '07 OCO Annual Review 13 An analogy: the electric power grid Energy goes in. Energy comes out. Providers do not target specific consumers. They just adhere to standards (60Hz). Consumers are not aware of specific providers. Analogy appears simplistic until you refine your concept of data. Data must always be tightly bound to its metadata.  DMAC integration is a “data grid” The concept of “integration” in DMAC Analogy is simplistic?

June '07 OCO Annual Review 14 The DMAC Plan (2004) is built around a “data grid” concept (a.k.a. “data commons”) Uniform services (standards) –to interconnect existing systems “Do no Harm” Existing standards are inadequate  An implementation plan, not a specification 240 pages How far have we progressed?

June '07 OCO Annual Review 15 Honest answer: barely at all. Why? 1.Formulation choices in the DMAC Plan 2.Political chaos 3.Community social structure How do we overcome each of these obstacles? How far has DMAC progressed since 2004?

June '07 OCO Annual Review 16 DMAC Plan has detailed milestones But they are not sufficiently tangible – e.g. “publish a community standard for [xxx]”. Solution: Reformulate the Plan as a sequence of tasks that each provide tangible benefits. Obstacle 1: Formulation choices in the plan

June '07 OCO Annual Review 17 Dumb, bad luck timing (post 9/11) & Interagency coordination failures lead to Negligible direct funding (just enough for “volunteer” meetings) (Note: millions have been made available that generated additional demand for DMAC guidance) (Note: millions have been made available that generated additional demand for DMAC guidance) Solution: Better marketing. Map out a Plan that can be marketed to Gov’t managers Obstacle 2: Political chaos

June '07 OCO Annual Review 18 Obstacle 3: Community social structure The diminutive nation of Science Data Management lies nestled among three neighbors: 1. IT Infrastructure 2. Computer Science 3. Science Research Each is larger and more powerful and imposes its viewpoint on our small nation. Science Research Computer Science IT Infrastructure Data Mgmt

June '07 OCO Annual Review 19 Obstacle 3: Community social structure 1. IT Infrastructure (CIO) viewpoint: “Solutions can be purchased if systems engineering discipline is followed.” But integration is not a system you can purchase. It is a change in how we work together. It must be built in partnership w/ data providers and users. Note: The DMAC Plan lays out a strong support role for systems engineering. (Useful reading: “The Innovator’s Dilemma”, by Clayton Christensen)

June '07 OCO Annual Review 20 Obstacle 3: Community social structure 2. Computer Science viewpoint: “The latest developing technology will solve the problems.” You can only standardize stable technology. Setting too-high requirements for technological innovation limits access to funding for IT projects that could yield great practical benefits to science. (The root of the “cyberinfrastructure” problem.) (The root of the “cyberinfrastructure” problem.)

June '07 OCO Annual Review 21 Obstacle 3: Community social structure 3. Science/Research viewpoint: “Reduce complexity by limiting the number of variables to be considered initially.” But data management challenges are largely independent of data content. Analogy: would it reduce complexity in designing an ocean glider if it only had to measure temperature? Data management simplifies by reducing the number of data structures (a.k.a. “data models”).

June '07 OCO Annual Review 22 Recap: Reformulate the DMAC Plan  tasks w/ tangible benefits … so we can  tasks w/ tangible benefits … so we can Market the Plan to Gov’t managers Independence of action from neighbors: 1.Partner with the IOOS community 2.Use available technology (wisely) 3.Reduce the initial problems by addressing data structures one by one.

June '07 OCO Annual Review 23 Proposal: Build the DMAC integration framework as a collection of Virtual Data Assembly Centers (“V-DACs”) by data structure. To be developed one-by-one: 1.Grids (models, satellites, climatologies) 2.Time series 3.Surface Tracks 4.Vertical Profiles and Sections 5.…, Scatters, Swaths, Radials, Polygons, …

June '07 OCO Annual Review 24 time series protocol Time series V-DAC Meta- data TAO BATS OceanSites U. Hawaii Sea Level Center NDBC NODC Imagine the V-DAC for time series data …

June '07 OCO Annual Review 25 time series protocol Time series V-DAC Meta- data TAOBATS OceanSites U. Hawaii Sea Level Center NDBC NODC bricks-and-mortar time series “curator” (funded) standard protocol(s) (“web services”) one access point multiple variables Imagine the V-DAC for time series data

June '07 OCO Annual Review 26 also fund a metadata development activity: –Data discovery –Controlled vocabularies –Data lineage –Geo-referencing –Instrument characterizations –Quality control

How do we build an ocean temperature V-DAC? Time series V-DAC Meta- data Profiles V-DAC Meta- data Grids V-DAC Meta- data Temperature V-DAC Meta- data A single place to access all ocean temperature data

June '07 OCO Annual Review 28 The virtues of this approach: Reductionism: One protocol at a time A concrete deliverable at every step Unites communities of interest (integration) But can we market the idea to management? (Who has the ability to carry the message to management?) The science community has a strong voice. (Much stronger than DM.)

June '07 OCO Annual Review 29 Discussion (Thank you)