Data Management in the U.S. GLOBEC Program SCOR/IGBP Meeting on Data Management for Marine Research Projects Robert C. Groman Woods Hole Oceanographic Institution 8 – 10 December 2003 Click here for PowerPoint versionClick here for PowerPoint version.

U.S. GLOBECU.S. GLOBEC: Goal To understand the population dynamics. Ultimately want to be able to predict changes in distribution and abundance of key species as a result of changes in the physical and biotic environment, such as from climate change.

Three U.S. Programs Georges Bank – field program started in 1995 with some cruises earlier; field program ended in 1999.Georges Bank Northeast Pacific – field program started in Gulf of Alaska too.Northeast Pacific Southern Ocean – field program started in 2001 and ended in 2003.Southern Ocean

Georges Bank Study Area

Northeast Pacific Study Area

Southern Ocean Study Area

Project Components Field program: Georges Bank project completed 120 cruises with 360 days at sea. Laboratory experiments Retrospective studies Analysis and synthesis

Georges Bank Data 120 Cruises in inventory 118 Cruise reports printed 56 Cruise reports on-line Data objects on-line: See web siteSee web site Missing data: Zooplankton counts VPR data Acoustics data Mooring data

Northeast Pacific/CGOA Coastal Gulf of Alaska (CGOA) 35 Cruise reports on-line; 16 will be on-line soon; 36 to come later 35 Event logs on-line 11 from LTOP cruises [29 missing] 9 from Haldorson Trawling cruises [8 missing] 10 from Process/Survey cruises [5 missing] 5 from SECM cruises (via on-line report) [10 missing]

Northeast Pacific/CCS California Current System (CCS) 46 Cruise reports on-line; 2 more in the works 47 Event logs on-line 36 from CCS - LTOP cruises (via on-line report) 11 from CCS – Process/Other cruises

Northeast Pacific Summary Available on-line data include CTD, SST, alongtrack, bottle, SeaSoar, nutrients, pigments, and event logs.on-line data Data soon to be added include CTD, nutrients, and zooplankton.

Southern Ocean 11 Cruises in inventory 11 Cruise reports on-line 11 Event logs on-line 110 data objects on-line, locally or linked remotely, including:data objects Ice core and water column bacteria studies Bird studies BIOMAPERII Chlorophyll, irradiance and productivity studies MOCNESS CTD data Nutrient data Sea ice data Whale sonobuoy data 120 kHz acoustic backscattering data ADCP data Alongtrack data Seal tracking and biology Bathymetry

Southern Ocean Data in the Works IWC Whale data CTD rosette data MOC1/MOC10/net collection data XCTD/XBT data Penguin studies (exists on SO GLOBEC website) ROV data Mooring data

Data Policy Dissemination of data to scientific investigators and others on a timely basis Make available when useful (not necessarily only when finalized) Serve data and information, such as reports, papers, and other program documentation

Data Characteristics and Distribution Approach Data from many, distributed, researchers (greater than 100 contributors) Open access – read only by everyone Restricted access supported, but rarely used Quality control is contributors’ responsibility and on-going Emphasis on access to data and information as early as possible Data sets most useful when used with other data

Data Acknowledgement Policy Any person making substantial use of a data set must communicate with the investigator(s) who acquired the data prior to publication and anticipate that the data collector(s) will be co-author(s) of published results. This extends to model results and to data organized for retrospective studies. See on-line policy statement

Data are accessed from the U.S. GLOBEC Data Server,

Data can be viewed

Or plotted

Track plot of NBP0202

Or downloaded

Data Sources Broad-scale cruises Process cruises Moorings Drifters Satellites Modeling

Instruments CTDs Rosette MOCNESS (3 flavors) Bongo tows Acoustic biomass measurements Video Plankton Recorder Drifters, MET packages,...

SensorsSensors and Computed Parameters Conductivity, temperature, pressure, fluorescence, transmittance, acoustics, light (PAR), video, wind speed/direction, AVHRR,... Biomass, taxonomic composition/size distribution, species (counts, size, stage, status, rates, behavior), density, currents, stratification, heat flux, nutrients, turbulence, chlorophyll,...

Data Access Using the JGOFS Data Management System developed by G. Flierl, J. Bishop, D. Glover, and S. Paranjpe Distributed access via standard web browsersDistributed access via standard web browsers

Web Access Hierarchical list of data objectslistobjects On-line list of dataOn-line list Downloads as ASCII, Matlab files, or reorganized into single or multiple filesDownloads Simple X-Y plotsX-Y plots Created EasyKrig (kriging) and 3-D visualization applicationsEasyKrig3-D visualization

Distributed Data Ten distributed data servers use the US JGOFS software Uses the Web httpd protocol - integrates very well with standard web pages Handles tabular data in ASCII, Matlab format, and user-supplied formats using methods. It is object oriented and data driven.object oriented

Nostalgia In the Olden Days …. Reformatting and processing data was a common activity Merging navigation with measured and computed results also took time First data management system used 9 track tapes for data storage, run in batch Second system used data on disk with techniques to located data within degree squares to improve performance

Meta-data Data about data.data Document information about data elements or attributes (name, size, data type, etc), about records or data structures (length, fields, columns, etc), and about data (where it is located, ownership, etc.). Meta-data may include descriptive information about the context, quality and condition, or characteristics of the data.data elements attributes recordsdata structures

Detailed Meta-data Pros – required for full understanding of data within a database management system. Required if others want to use the data Cons – pain in the neck to prepare, maintain, and enter (Best to take advantage of tools) Currently completing Global Change Master Directory’s DIF records

What’s Happening Organizations creating systems to access their own meta-data and/or data. Umbrella databases linking to other peoples meta-data and/or data. (OBIS, GMBIS, …)OBISGMBIS Linking to meta-data is more manageable than is linking to other people’s data.

Other Efforts LabNet – consortium of marine organizations to make their data available (uses 4D Geobrowser “index cards”)LabNet Ocean Data View - access WOCE, NGDC, and other data sets. CTD, bottle, XBT …Ocean Data View OBIS – “portal” (aggregation server) for biological data (using Darwin Core 2 – OBIS)OBIS

Other Efforts, continued ZOPE – object oriented application serverZOPE LAS – web-based, active-image based data interface for registered data. Used by US JGOFS ProgramUS JGOFS Program uBio – (Universal Biological Indexer and Organizer) a networked information service for biological information resources based on the Taxonomic Name Server (TNS), a thesaurus; an index.

Other Efforts, continued Hexacoral – biggest user in OBIS; uses DiGIR (D.G. Fautin, et al.)Hexacoral DiGIR – Distributed Generic Information Retrieval. Uses XML protocol to get the data. Extends XML to do queries. Uses php software package to execute the code. Supports 14 or 15 databases, e.g SQL based. Three options for JGOFS: export to flat file, export to MySQL, or write own perl script to interface directly to DiGIR (ZooGene -> OBIS)DiGIR

Other Efforts, continued Oregon State University, Randy Keller and Paul Johnson, mapping specialist at HMRG Steve Hankin, “An Implementation Plan for the Data and Communication Subsystem of the U.S. Integrated Ocean Observing System” Margo Edwards at HIG and Dawn Wright at OSU

Other Efforts, continued RIDGE, petrological data. Endeavor Observatory website, Lamont’s PetDBPetDB SIO Ocean Exploration data portal, University of Washington’s Endeavor GIS and Portal to Endeavor Data (PED)GISPED

Educational “Tools” Virtual Research Vessel, University of Oregon and Oregon State UniversityVirtual Research Vessel REVEL, University of WashingtonREVEL Dive and Discover, WHOIDive and Discover

Protocols OpenDAP (was DODS)  http DiGIR  uses XML; but too verbose for physical data. OBIS may use OpenDAP for physical data. JGOFS  http

Other Projects and Protocols Apologies for references I’ve missed. There are many other efforts underway in all these areas.

In the Trenches What temperature: Sea surface, air, at depth? Units? How collected? How calibrated? Data quality control still labor intensive even though we can collect and store gigabytes of data daily

Future Data Management and Display Efforts Enhance data search capabilities Add additional graphical display (visualization) options Improve interface between data system and visualization/analysis tool Consider other protocols, such as OpenDAP

End