Monterey Bay 2006 Data System Overview October 6, 2005 Michael Godin Monterey Bay Aquarium Research Institute (MBARI)

Agenda The Challenge User & System Requirements Usage of the System System Architecture Development Milestones What do we need before the VE? M. Godin, MBARI, 10/6/2005

Data Management Challenge Large number of data sources Large variety of data sources Dynamic systems –Data sources may appear during the campaign No standard data format –Streams or files, automated or manual Large number of data sources Large variety of data sources Dynamic systems –Data sources may appear during the campaign No standard data format –Streams or files, automated or manual M. Godin, MBARI, 10/6/2005

User & System Requirements M. Godin, MBARI, 10/6/2005

User Requirements Participant s Transfer Data To Server Add new Data Streams to Server Add new Data to Server Recognize any Data Format Quality/Ver sion Control Secure Archive Transfer Data From Server Common Format Data Original Format Data Decision Making Tools State of system information Observatio n state Adaptive sampling tools Model Outputs Model-data comparison Other Users Transfer Data From Server Image Access to Embargoed DataImage Access to Embargoed Data Full Access to non-embargoed dataFull Access to non-embargoed data M. Godin, MBARI, 10/6/2005

System Requirements Meeting User Needs System Requirement s Transfer Data To Server Add new Data Streams to Server Add new Data to Server Recognize any Data Format Quality/Vers ion Control Secure Archive Transfer Data From Server Common Format Data Original Format Data Decision Making Tools State of system information Observation state Adaptive sampling tools Model Outputs Model-data comparison Data Access Security M. Godin, MBARI, 10/6/2005

Describing Incoming Data File Streams Interfaces for Adding Data Interfaces for Getting Data & Images Supported Interfaces for Using Data Interfaces for System State Usage of the System M. Godin, MBARI, 10/6/2005

Describing Incoming Data Streams M. Godin, MBARI, 10/6/2005

Interfaces for Adding Data FTP M. Godin, MBARI, 10/6/2005

Interfaces for Adding Data FTPWeb M. Godin, MBARI, 10/6/2005

Interfaces for Adding Data FTPWeb M. Godin, MBARI, 10/6/2005

Dynamic Data Ingest Your Server FTP or HTTP Interfaces for Adding Data FTPWeb Server Pull Your Database Central Server Storage Request Data Response M. Godin, MBARI, 10/6/2005

Live Access Server (LAS)Live Access Server (LAS) Interfaces for Getting Data & Images M. Godin, MBARI, 10/6/2005

Interfaces for Getting Data & Images Live Access Server (LAS)Live Access Server (LAS) Metadata Oriented Query Assistant (MOQuA)Metadata Oriented Query Assistant (MOQuA) M. Godin, MBARI, 10/6/2005

Ocean Data ViewOcean Data View Supported Interfaces for Using Data M. Godin, MBARI, 10/6/2005

Ocean Data ViewOcean Data View Space/Time Toolkit?Space/Time Toolkit? AMES CIP?AMES CIP? Supported Interfaces for Using Data M. Godin, MBARI, 10/6/2005

Interfaces for System State Network Status (Nagios) Network Status (Nagios) M. Godin, MBARI, 10/6/2005

Interfaces for System State Network Status (Nagios) Network Status (Nagios) Last Known Position Of Assets Last Known Position Of Assets M. Godin, MBARI, 10/6/2005

Interfaces for System State Network Status (Nagios) Network Status (Nagios) Last Known Position Of Assets Last Known Position Of Assets Processing Stoplight Chart Processing Stoplight Chart M. Godin, MBARI, 10/6/2005

Where We Were: 2003 Data System Where We Got to: Post-2003 Data System Where We Want to be: MB2006 Data System Architecture Where We Will be Soon: Virtual Experiment Data System Data System Architecture M. Godin, MBARI, 10/6/2005

Original Archive Where We Were: 2003 Data System Architecture Devices Models & Derived Outputs Principal Investigators’ Processing, QC, & Archives Common Archive Principal Investigators’ Downloads Derived Data Primary Data Polarbear MBARI Firewall Custom Processing Public Image Requests mbari.org (static images) M. Godin, MBARI, 10/6/2005

Original Archive Where We Got To: Post-2003 Data System Architecture Devices Models & Derived Outputs Principal Investigators’ Processing, QC, & Archives Common Archive Principal Investigators’ Downloads Derived Data Primary Data Polarbear MBARI Firewall Custom Processing aosn.mbari.org Data Query Data line 1 more data last data Public Image Requests Principal Investigators’ Data Requests Reanalyses M. Godin, MBARI, 10/6/2005

Original Archive Data Stream Descriptions MB2006 Data System Architecture Devices Models & Derived Outputs Dynamic Data Ingest Principal Investigators’ Processing, QC, & Archives Common Archive Data Query Data line 1 more data last data Static Metadata Dynamic Metadata System State Central Server Principal Investigators’ Data Requests Public Image Requests Decision Maker Queries Derived Data Primary Data M. Godin, MBARI, 10/6/2005

Original Archive Data Stream Descriptions Virtual Experiment Data System Architecture Devices Models & Derived Outputs Dynamic Data Ingest Principal Investigators’ Processing, QC, & Archives Common Archive Data Query Data line 1 more data last data Static Metadata Dynamic Metadata System State Central Server Principal Investigators’ Data Requests Public Image Requests Decision Maker Queries Derived Data M. Godin, MBARI, 10/6/2005

Development Milestones Basic Data Ingest, Processing, Storage, and Serving: October 28, 2005 –Mods to MBARI’s Shore Side Data System –Mods to NOAA/PMEL’s Live Access Server Data Version Control, Asset Map: 11/14/2005 Waypoint–Path comparison: before VE? Post-Virtual Experiment Milestones: –User Access Controls & Public Images –Stop-Light Chart, Network Map –Custom Output Data Formats & Graphics –Saved Queries and Routine Outputs M. Godin, MBARI, 10/6/2005

What Do We Need for the VE? Example Data Sets, by 10/21/2005 Desired Data Outputs, by 10/28/2005 Metadata by 11/14/2005 (entry form will be forthcoming…) –Data Import Specifications (how to get data) –Data Reference Information Nice to have by 11/14/2005: –Waypoint files M. Godin, MBARI, 10/6/2005

The End M. Godin, MBARI, 10/6/2005