1. MBARI’s Shore Side Data System From Ships, ROVs, Moorings, AUVs, & ? To Bytes, Plots, Pictures, Samples, & Video

2. 21 Nov 2002 1 What Are Our Goals? Build data systems that can grow over time Build data systems that can grow over time Make adding instruments and data routine Make adding instruments and data routine Easily add new and unimagined components Easily add new and unimagined components Scale to meet growth needs of observing systems Scale to meet growth needs of observing systems Create an extensible IT umbrella Create an extensible IT umbrella Encompass the real world of data sources Encompass the real world of data sources shipboard and shore-side systems shipboard and shore-side systems isolated buoys and networked observatories isolated buoys and networked observatories one-off data files and high-speed (Gb net) streaming data one-off data files and high-speed (Gb net) streaming data Embrace image, video and document archive formats Embrace image, video and document archive formats Still provide users with “do what I want” Still provide users with “do what I want”

3. 21 Nov 2002 2 Cruise (Expedition) Interface

4. 21 Nov 2002 3 Samples Interface

5. 21 Nov 2002 4 Video Annotations Interface

6. 21 Nov 2002 5 3D Visualization Interface

7. 21 Nov 2002 6 What Has MBARI Learned? Metadata: It must accompany the data Metadata: It must accompany the data Data w/out metadata is like a directory with no Readme Data w/out metadata is like a directory with no Readme A system’s power relies on good knowledge of its data A system’s power relies on good knowledge of its data Metadata: It must accompany the instrument Metadata: It must accompany the instrument Every connector between the two increases error rates Every connector between the two increases error rates Once data and metadata detached, reattaching is painful Once data and metadata detached, reattaching is painful Metadata: It must be flexible and yet structured Metadata: It must be flexible and yet structured Flexible: you’ll need to define new kinds of data sources Flexible: you’ll need to define new kinds of data sources Structured: consistency => automation => value Structured: consistency => automation => value Distributed storage, great interfaces, transparency Distributed storage, great interfaces, transparency

8. 21 Nov 2002 7 About MOOS: MBARI’s Ocean Observing System A major observing platform development initiative A major observing platform development initiative Multi-platform, cabled & uncabled, benthic to surface Multi-platform, cabled & uncabled, benthic to surface “What Would It Take?” — develop and test ideas “What Would It Take?” — develop and test ideas Need a way to store metadata with instrument Need a way to store metadata with instrument Need a way to submit metadata to data system Need a way to submit metadata to data system Result: Answers that can work anywhere Result: Answers that can work anywhere Local ‘intelligent’ storage: PUCK concept Local ‘intelligent’ storage: PUCK concept Consistent services: Instrument SW Infrastructure Consistent services: Instrument SW Infrastructure We can iterate to good, tested solutions We can iterate to good, tested solutions

9. 21 Nov 2002 8 MOOS (Showing Data Flow) Devices Deployed Platform Shore Side Data System User Applications (User Tools) Data Tracking Communications Applications/ Interfaces Archiving 101110 110011 110 234 999 Data Presentation Data line 1 more data last data OceanSideShoreSide Portal

10. 21 Nov 2002 9 About SSDS: The Shore Side Data System A MOOS Development Project A MOOS Development Project Goals: low cost, flexible, expandable, reliable Goals: low cost, flexible, expandable, reliable Future systems beyond MOOS (e.g., MARS) Future systems beyond MOOS (e.g., MARS) Now in 3rd year, deploying initial elements Now in 3rd year, deploying initial elements Key Tenets of SSDS Development Key Tenets of SSDS Development Iterative development—improve it as we go Iterative development—improve it as we go Test with real data—new and archival Test with real data—new and archival Build for change—use modular interfaces Build for change—use modular interfaces

11. 21 Nov 2002 10 Shore Side Data System Requirements Ingest data in any described format and save it Ingest data in any described format and save it Capture, publish data descriptions (metadata) Capture, publish data descriptions (metadata) Provide standards-based access to data Provide standards-based access to data Raw data, and other common digital formats Raw data, and other common digital formats APIs for common visualization and analysis tools APIs for common visualization and analysis tools User-oriented web interfaces, quick-look plots User-oriented web interfaces, quick-look plots Merge data (different sources & time intervals) Merge data (different sources & time intervals) Support data visualization & quality control Support data visualization & quality control Provide data access security as needed Provide data access security as needed

12. 21 Nov 2002 11 SSDS Elements Applications Data Presentation Data line 1 more data last data Ingest 110 234 999 223 207 191 Archiving 101110 110011 Arriving Data Data Tracker Data Catalog External Data Stores Data For Analysis Web I/F < Requests Data > Shared Descriptions < Requests Data > Metadata (Re)Processed and New Data Sets Automated Data Flow Internal Interfaces On-Demand Interactions

13. 21 Nov 2002 12 How Does It Work? 1. First, the developer describes what’s in an instrument’s data records (the metadata). the metadatathe metadata 2. That description is stored with (or near) an instrument, and sent to SSDS before any data from the instrument. SSDS tracks this info…. 3. Data records generated include the data type. data typedata type 4. SSDS automatically routes data of each type to the correct ‘data bucket’. 5. SSDS automagically knows about the data, because they’ve been described. Now it can: Plot  Print  Search  Merge Plot  Print  Search  Merge Format (on request)  Describe (in files & headers) Format (on request)  Describe (in files & headers) Send to Applications  Point to by variable name Send to Applications  Point to by variable name 1234

14. 21 Nov 2002 13 Standard Interfaces Applications Data Presentation Data line 1 more data last data Ingest 110 234 999 223 207 191 Archiving 101110 110011 Arriving Data Data Tracker Data Catalog External Data Stores Data For Analysis Web I/F < Requests Data > Shared Descriptions < Requests Data > Metadata (Re)Processed and New Data Sets DODS LAS XML/ DTDs HTTP Z39.50 SQL netCDF FGDC inGrid Browsers and Tools ncBrowse DODS Ferret

15. 21 Nov 2002 14 Strategies (Hard-Earned) Low threshold for user entry (minimal XML) Low threshold for user entry (minimal XML) Stay away from domain-specific solutions Stay away from domain-specific solutions Example: ‘deployment’ is useful concept Example: ‘deployment’ is useful concept Minimize internal structure & assumptions Minimize internal structure & assumptions Our biggest challenge: flexible architecture Our biggest challenge: flexible architecture Be agnostic about input data & file formats Be agnostic about input data & file formats Maximize access/presentation features Maximize access/presentation features Provide many views into data (common first) Provide many views into data (common first) Take advantage of OO methods and reuse Take advantage of OO methods and reuse

16. 21 Nov 2002 15 SSDS Multi-View Interface

17. 21 Nov 2002 16 SSDS Multi-View Interface

18. 21 Nov 2002 17 Data Integration Strategy Data can be remote (managed by links) Data can be remote (managed by links) Domain-specific tasks done externally Domain-specific tasks done externally Domain-specific calibration and QC Domain-specific calibration and QC Data reprocessing and conversions Data reprocessing and conversions Non-automatable data sets (time series) Non-automatable data sets (time series) Custom views unique to domain or media Custom views unique to domain or media SSDS is access point / service provider SSDS is access point / service provider Maintain focus on core services, interfaces Maintain focus on core services, interfaces

19. 21 Nov 2002 18 Development Status First deployed for MOOS Test Mooring First deployed for MOOS Test Mooring 10/1: Training with development tools 10/1: Training with development tools 10/7: First actual code written 10/7: First actual code written 11/8: First end to end test (in use ever since) 11/8: First end to end test (in use ever since) 11/19: Demonstrated Java GUI for data access 11/19: Demonstrated Java GUI for data access 12/3: Data is live from deployed mooring 12/3: Data is live from deployed mooring These are prototype solutions (first round) These are prototype solutions (first round) Planning for AUV CTD data management Planning for AUV CTD data management

20. 21 Nov 2002 19Summary We designed and built a flexible, dynamic data system with an open architecture. Metadata critical to observatory and instrument operation. Metadata critical to observatory and instrument operation. Standard interfaces enforce modularity. Standard interfaces enforce modularity. A layered metadata model with generic concepts provides multiple data access paths. A layered metadata model with generic concepts provides multiple data access paths. Iterative development processes support fast product deployment and improvements. Iterative development processes support fast product deployment and improvements.

21. 21 Nov 2002 20Acknowledgements Monterey Bay Aquarium Research Institute Monterey Bay Aquarium Research Institute David and Lucile Packard Foundation David and Lucile Packard Foundation The SSDS Team: The SSDS Team: Kevin Gomes, John Graybeal, Mike McCann, Brian Schlining, Rich Schramm, Dan Wilkin Kevin Gomes, John Graybeal, Mike McCann, Brian Schlining, Rich Schramm, Dan Wilkin The ISI Team: The ISI Team: Led by Duane Edgington and Tom O’Reilly Led by Duane Edgington and Tom O’Reilly All our committed and helpful users All our committed and helpful users

22. 21 Nov 2002 21Contacts Shore Side Data System: Shore Side Data System: John Graybeal, IAG Lead 831-775-1956graybeal@mbari.org John Graybeal, IAG Lead 831-775-1956graybeal@mbari.orggraybeal@mbari.org Video Annotation and Reference System (VARS): Video Annotation and Reference System (VARS): Dan Wilkin, Lead Developer 831-775-1865wilkin@mbari.org Dan Wilkin, Lead Developer 831-775-1865wilkin@mbari.orgwilkin@mbari.org Samples Database Interface Samples Database Interface Susan Von Thun, Samples Coordinator 831-775-2006svonthun@mbari.org Susan Von Thun, Samples Coordinator 831-775-2006svonthun@mbari.orgsvonthun@mbari.org Cruise (Expedition), SSDS, and ROV 3D Interfaces Cruise (Expedition), SSDS, and ROV 3D Interfaces Mike McCann, Lead Developer 831-775-1769mccann@mbari.org Mike McCann, Lead Developer 831-775-1769mccann@mbari.orgmccann@mbari.org

23. 21 Nov 2002 22 Recap: MOOS Data Flow Devices Deployed Platform Shore Side Data System User Applications (User Tools) Data Tracking Communications Applications/ Interfaces Archiving 101110 110011 110 234 999 Data Presentation Data line 1 more data last data OceanSideShoreSide Portal 4 2b 2c 3 5a 1 5b 2a

24. 21 Nov 2002 23 Metadata Description Example <recordDescription><recordType>1</recordType><bufferStyle>ascii</bufferStyle><bufferLengthType>variable</bufferLengthType><bufferParseType>separated</bufferParseType><bufferItemSeparator>,</bufferItemSeparator> <recordTerminator>\r\n</recordTerminator><variable><columnIndex>1</columnIndex><format>%3s</format><variableName>messageType</variableName><units></units> Message Type Message Type WB indicates house keeping message WB indicates house keeping message </variable><variable><columnIndex>2</columnIndex><format>%i</format><variableName>messageSize</variableName><units>characters</units> Message Size Message Size The number of characters in the message not including the The number of characters in the message not including the message type, size, or checksum characters message type, size, or checksum characters </variable>…

25. 21 Nov 2002 24 Metadata Description Example … 3 %6s date YYMMDD Date in YYMMDD format 4 %4s time HHMM Time in HHMM format 5 %f voltage Volts Voltage …

26. 21 Nov 2002 25 Data Record Identification Each data record sent via ISI is wrapped with ‘record header’ metadata to identify it Each data record sent via ISI is wrapped with ‘record header’ metadata to identify it Contents of record header metadata include: Contents of record header metadata include: Data source ID (all MOOS data sources have ID) Data source ID (all MOOS data sources have ID) Metadata ID (what file has a description to this record) Metadata ID (what file has a description to this record) Record Type ID (which kind of record is this) Record Type ID (which kind of record is this) Timestamp (provided by ISI when record is submitted) Timestamp (provided by ISI when record is submitted) System now knows what to do with each data record (otherwise they’d all be mushed together in a long stream of records) System now knows what to do with each data record (otherwise they’d all be mushed together in a long stream of records) Right now we also ask for and use ‘source platform’ Right now we also ask for and use ‘source platform’ Exact mechanism used to supply this is TBD Exact mechanism used to supply this is TBD

27. 21 Nov 2002 26 Science Data Users: What Do You Need (First)? Raw data via device ID pages? (sort of limited) Raw data via device ID pages? (sort of limited) Standard plots the same as OASIS quality control? Standard plots the same as OASIS quality control? Standard plots with multiple plots per page? Standard plots with multiple plots per page? Access data from applications via a DODS URLs? Access data from applications via a DODS URLs? Matlab, Ingrid, Live Access Server, Excel, IDV, Ferret Matlab, Ingrid, Live Access Server, Excel, IDV, Ferret And hopefully, Ocean Data View And hopefully, Ocean Data View Access data via shipped data files (e.g., ASCII CSV w/headers) opened within desktop applications? Access data via shipped data files (e.g., ASCII CSV w/headers) opened within desktop applications? Excel, ArcView, Ocean Data View Excel, ArcView, Ocean Data View Delivery of data directly into an application? Delivery of data directly into an application? Ability to subset data, for example by time window? Ability to subset data, for example by time window? Ability to merge data from different data sets? Ability to merge data from different data sets?