1. Kevin Gomes, MBARI MBARI Data Architecture OOI Cyberinfrastructure: Data Product Generation Workshop San Diego May 20-21, 2008

2. OOI-CYBERINFRASTRUCTURE MBARI’s Data Architecture (Observatory-related) Sensing and Acquisition –Instruments, Moorings, AUV, FOCE, ALOHA, Benthic Rover Infrastructure –OASIS, SIAM, SSDS, Data Processing Data –Formats, Transformations, Standards Interfaces –Infrastructure, User, API, SOA Other Information from Questionnaire

3. OOI-CYBERINFRASTRUCTURE Inst PUCK NetCDF Ocean Sites JMS HTTP OGC Services Web Services API THREDDS Strategy Raw Data OPeNDAP Inst Rover OASIS Telemetry ALOHA (MARS/Seahurst) Client OASIS 2 SSDS Driver OASIS Service SIAM MOOS Proxy SIAM Portal Inst PUCK Service SIAM Strategy Proxy Inst PUCK Service SIAM FOCE AUV Rover Portal AUV Portal Strategy Proxy SIAM Portal SSDS MetadataNetApp Adapter ESB HTTP Web App Adapter Connector ESB Inst Proxy Adapter Strategy

4. OOI-CYBERINFRASTRUCTURE Inst PUCK NetCDF Ocean Sites JMS HTTP OGC Services Web Services API THREDDS Strategy Raw Data OPeNDAP Inst Rover OASIS Telemetry ALOHA (MARS/Seahurst) Client OASIS 2 SSDS Driver OASIS Service SIAM MOOS Proxy SIAM Portal Inst PUCK Service SIAM Strategy Proxy Inst PUCK Service SIAM FOCE AUV Rover Portal AUV Portal Strategy Proxy SIAM Portal SSDS MetadataNetApp Adapter ESB HTTP Web App Adapter Connector ESB Inst Proxy Adapter Strategy 1001001 001110 12.2, 10.1

5. OOI-CYBERINFRASTRUCTURE Inst PUCK NetCDF Ocean Sites JMS HTTP OGC Services Web Services API THREDDS Strategy Raw Data OPeNDAP Inst Rover OASIS Telemetry ALOHA (MARS/Seahurst) Client OASIS 2 SSDS Driver OASIS Service SIAM MOOS Proxy SIAM Portal Inst PUCK Service SIAM Strategy Proxy Inst PUCK Service SIAM FOCE AUV Rover Portal AUV Portal Strategy Proxy SIAM Portal SSDS MetadataNetApp Adapter ESB HTTP Web App Adapter Connector ESB Inst Proxy Adapter Strategy OASIS Notes: 1.Drivers are part of controller software (I.e. a priori knowledge) 2.Instrument lifecycles are controlled by operator in comms session 3.Very little metadata in system, added later

6. OOI-CYBERINFRASTRUCTURE Inst PUCK NetCDF Ocean Sites JMS HTTP OGC Services Web Services API THREDDS Strategy Raw Data OPeNDAP Inst Rover OASIS Telemetry ALOHA (MARS/Seahurst) Client OASIS 2 SSDS Driver OASIS Service SIAM MOOS Proxy SIAM Portal Inst PUCK Service SIAM Strategy Proxy Inst PUCK Service SIAM FOCE AUV Rover Portal AUV Portal Strategy Proxy SIAM Portal SSDS MetadataNetApp Adapter ESB HTTP Web App Adapter Connector ESB Inst Proxy Adapter Strategy ID: X Parent: Y Data: [0110..] ID: X Parent: Y Data: [0110..]

7. OOI-CYBERINFRASTRUCTURE Inst PUCK NetCDF Ocean Sites JMS HTTP OGC Services Web Services API THREDDS Strategy Raw Data OPeNDAP Inst Rover OASIS Telemetry ALOHA (MARS/Seahurst) Client OASIS 2 SSDS Driver OASIS Service SIAM MOOS Proxy SIAM Portal Inst PUCK Service SIAM Strategy Proxy Inst PUCK Service SIAM FOCE AUV Rover Portal AUV Portal Strategy Proxy SIAM Portal SSDS MetadataNetApp Adapter ESB HTTP Web App Adapter Connector ESB Inst Proxy Adapter Strategy SIAM Notes: 1.Drivers travel with instrument in PUCK (Plug and work) 2.Some aspects of instrument lifecycles can now be done automatically by node (better management) 3.Higher level commands still done by human intervention over comms session. 4.*Metadata attached to instrument. Sent when instrument service is started.

8. OOI-CYBERINFRASTRUCTURE Inst PUCK NetCDF Ocean Sites JMS HTTP OGC Services Web Services API THREDDS Strategy Raw Data OPeNDAP Inst Rover OASIS Telemetry ALOHA (MARS/Seahurst) Client OASIS 2 SSDS Driver OASIS Service SIAM MOOS Proxy SIAM Portal Inst PUCK Service SIAM Strategy Proxy Inst PUCK Service SIAM FOCE AUV Rover Portal AUV Portal Strategy Proxy SIAM Portal SSDS MetadataNetApp Adapter ESB HTTP Web App Adapter Connector ESB Inst Proxy Adapter Strategy ID: X Parent: Y Data: [0110..] ID: X Parent: Y Data: [0110..]

9. OOI-CYBERINFRASTRUCTURE Inst PUCK NetCDF Ocean Sites JMS HTTP OGC Services Web Services API THREDDS Strategy Raw Data OPeNDAP Inst Rover OASIS Telemetry ALOHA (MARS/Seahurst) Client OASIS 2 SSDS Driver OASIS Service SIAM MOOS Proxy SIAM Portal Inst PUCK Service SIAM Strategy Proxy Inst PUCK Service SIAM FOCE AUV Rover Portal AUV Portal Strategy Proxy SIAM Portal SSDS MetadataNetApp Adapter ESB HTTP Web App Adapter Connector ESB Inst Proxy Adapter Strategy Cabled Notes: 1.Middleware doesn’t change, only execution location

10. OOI-CYBERINFRASTRUCTURE Inst PUCK NetCDF Ocean Sites JMS HTTP OGC Services Web Services API THREDDS Strategy Raw Data OPeNDAP Inst Rover OASIS Telemetry ALOHA (MARS/Seahurst) Client OASIS 2 SSDS Driver OASIS Service SIAM MOOS Proxy SIAM Portal Inst PUCK Service SIAM Strategy Proxy Inst PUCK Service SIAM FOCE AUV Rover Portal AUV Portal Strategy Proxy SIAM Portal SSDS MetadataNetApp Adapter ESB HTTP Web App Adapter Connector ESB Inst Proxy Adapter Strategy ID: X Parent: Y Data: [0110..] ID: X Parent: Y Data: [0110..]

11. OOI-CYBERINFRASTRUCTURE Inst PUCK NetCDF Ocean Sites JMS HTTP OGC Services Web Services API THREDDS Strategy Raw Data OPeNDAP Inst Rover OASIS Telemetry ALOHA (MARS/Seahurst) Client OASIS 2 SSDS Driver OASIS Service SIAM MOOS Proxy SIAM Portal Inst PUCK Service SIAM Strategy Proxy Inst PUCK Service SIAM FOCE AUV Rover Portal AUV Portal Strategy Proxy SIAM Portal SSDS MetadataNetApp Adapter ESB HTTP Web App Adapter Connector ESB Inst Proxy Adapter Strategy ID: X Parent: Y Data: [0110..] [1010011101] Or 12.3, 432.0

12. OOI-CYBERINFRASTRUCTURE Inst PUCK NetCDF Ocean Sites JMS HTTP OGC Services Web Services API THREDDS Strategy Raw Data OPeNDAP Inst Rover OASIS Telemetry ALOHA (MARS/Seahurst) Client OASIS 2 SSDS Driver OASIS Service SIAM MOOS Proxy SIAM Portal Inst PUCK Service SIAM Strategy Proxy Inst PUCK Service SIAM FOCE AUV Rover Portal AUV Portal Strategy Proxy SIAM Portal SSDS MetadataNetApp Adapter ESB HTTP Web App Adapter Connector ESB Inst Proxy Adapter Strategy Files: 010101010111 Files: 010101010111

13. OOI-CYBERINFRASTRUCTURE Inst PUCK NetCDF Ocean Sites JMS HTTP OGC Services Web Services API THREDDS Strategy Raw Data OPeNDAP Inst Rover OASIS Telemetry ALOHA (MARS/Seahurst) Client OASIS 2 SSDS Driver OASIS Service SIAM MOOS Proxy SIAM Portal Inst PUCK Service SIAM Strategy Proxy Inst PUCK Service SIAM FOCE AUV Rover Portal AUV Portal Strategy Proxy SIAM Portal SSDS MetadataNetApp Adapter ESB HTTP Web App Adapter Connector ESB Inst Proxy Adapter Strategy AUV Notes: 1.Data itself not located in SSDS, distributed

14. OOI-CYBERINFRASTRUCTURE Inst PUCK NetCDF Ocean Sites JMS HTTP OGC Services Web Services API THREDDS Strategy Raw Data OPeNDAP Inst Rover OASIS Telemetry ALOHA (MARS/Seahurst) Client OASIS 2 SSDS Driver OASIS Service SIAM MOOS Proxy SIAM Portal Inst PUCK Service SIAM Strategy Proxy Inst PUCK Service SIAM FOCE AUV Rover Portal AUV Portal Strategy Proxy SIAM Portal SSDS MetadataNetApp Adapter ESB HTTP Web App Adapter Connector ESB Inst Proxy Adapter Strategy

15. OOI-CYBERINFRASTRUCTURE Inst PUCK NetCDF Ocean Sites JMS HTTP OGC Services Web Services API THREDDS Strategy Raw Data OPeNDAP Inst Rover OASIS Telemetry ALOHA (MARS/Seahurst) Client OASIS 2 SSDS Driver OASIS Service SIAM MOOS Proxy SIAM Portal Inst PUCK Service SIAM Strategy Proxy Inst PUCK Service SIAM FOCE AUV Rover Portal AUV Portal Strategy Proxy SIAM Portal SSDS MetadataNetApp Adapter ESB HTTP Web App Adapter Connector ESB Inst Proxy Adapter Strategy

16. OOI-CYBERINFRASTRUCTURE Inst PUCK NetCDF Ocean Sites JMS HTTP OGC Services Web Services API THREDDS Strategy Raw Data OPeNDAP Inst Rover OASIS Telemetry ALOHA (MARS/Seahurst) Client OASIS 2 SSDS Driver OASIS Service SIAM MOOS Proxy SIAM Portal Inst PUCK Service SIAM Strategy Proxy Inst PUCK Service SIAM FOCE AUV Rover Portal AUV Portal Strategy Proxy SIAM Portal SSDS MetadataNetApp Adapter ESB HTTP Web App Adapter Connector ESB Inst Proxy Adapter Strategy

17. OOI-CYBERINFRASTRUCTURE Inst PUCK NetCDF Ocean Sites JMS HTTP OGC Services Web Services API THREDDS Strategy Raw Data OPeNDAP Inst Rover OASIS Telemetry ALOHA (MARS/Seahurst) Client OASIS 2 SSDS Driver OASIS Service SIAM MOOS Proxy SIAM Portal Inst PUCK Service SIAM Strategy Proxy Inst PUCK Service SIAM FOCE AUV Rover Portal AUV Portal Strategy Proxy SIAM Portal SSDS MetadataNetApp Adapter ESB HTTP Web App Adapter Connector ESB Inst Proxy Adapter Strategy

18. OOI-CYBERINFRASTRUCTURE Inst PUCK NetCDF Ocean Sites JMS HTTP OGC Services Web Services API THREDDS Strategy Raw Data OPeNDAP Inst Rover OASIS Telemetry ALOHA (MARS/Seahurst) Client OASIS 2 SSDS Driver OASIS Service SIAM MOOS Proxy SIAM Portal Inst PUCK Service SIAM Strategy Proxy Inst PUCK Service SIAM FOCE AUV Rover Portal AUV Portal Strategy Proxy SIAM Portal SSDS MetadataNetApp Adapter ESB HTTP Web App Adapter Connector ESB Inst Proxy Adapter Strategy Why NetCDF? 1.Well suited for large data sets 2.Modelers use it = easier data integration 3.Large community of support over many years 4.Developed in support of earth sciences 5.Leveraging atmospheric community (actively supported by UNIDATA) 6.Already works with many tools science users are familiar with (Matlab, Ocean Data View, Ferret, etc.) 7.Community built server side tools around it (LAS, OPeNDAP)

19. OOI-CYBERINFRASTRUCTURE Inst PUCK NetCDF Ocean Sites JMS HTTP OGC Services Web Services API THREDDS Strategy Raw Data OPeNDAP Inst Rover OASIS Telemetry ALOHA (MARS/Seahurst) Client OASIS 2 SSDS Driver OASIS Service SIAM MOOS Proxy SIAM Portal Inst PUCK Service SIAM Strategy Proxy Inst PUCK Service SIAM FOCE AUV Rover Portal AUV Portal Strategy Proxy SIAM Portal SSDS MetadataNetApp Adapter ESB HTTP Web App Adapter Connector ESB Inst Proxy Adapter Strategy

20. OOI-CYBERINFRASTRUCTURE Inst PUCK NetCDF Ocean Sites JMS HTTP OGC Services Web Services API THREDDS Strategy Raw Data OPeNDAP Inst Rover OASIS Telemetry ALOHA (MARS/Seahurst) Client OASIS 2 SSDS Driver OASIS Service SIAM MOOS Proxy SIAM Portal Inst PUCK Service SIAM Strategy Proxy Inst PUCK Service SIAM FOCE AUV Rover Portal AUV Portal Strategy Proxy SIAM Portal SSDS MetadataNetApp Adapter ESB HTTP Web App Adapter Connector ESB Inst Proxy Adapter Strategy

21. OOI-CYBERINFRASTRUCTURE Inst PUCK NetCDF Ocean Sites JMS HTTP OGC Services Web Services API THREDDS Strategy Raw Data OPeNDAP Inst Rover OASIS Telemetry ALOHA (MARS/Seahurst) Client OASIS 2 SSDS Driver OASIS Service SIAM MOOS Proxy SIAM Portal Inst PUCK Service SIAM Strategy Proxy Inst PUCK Service SIAM FOCE AUV Rover Portal AUV Portal Strategy Proxy SIAM Portal SSDS MetadataNetApp Adapter ESB HTTP Web App Adapter Connector ESB Inst Proxy Adapter Strategy

22. OOI-CYBERINFRASTRUCTURE Inst PUCK NetCDF Ocean Sites JMS HTTP OGC Services Web Services API THREDDS Strategy Raw Data OPeNDAP Inst Rover OASIS Telemetry ALOHA (MARS/Seahurst) Client OASIS 2 SSDS Driver OASIS Service SIAM MOOS Proxy SIAM Portal Inst PUCK Service SIAM Strategy Proxy Inst PUCK Service SIAM FOCE AUV Rover Portal AUV Portal Strategy Proxy SIAM Portal SSDS Metadata NetApp Adapter ESB HTTP Web App Adapter Connector ESB Inst Proxy Adapter Strategy COARDS, Climate and Forecast (CF), Ocean Sites, THREDDS: 1.COARDS & CF community (atmospheric) driven = experience 2.Ocean Sites developed by oceanographic science community 1.Ease of use through standards (Joe Q. Public/Student) 3.Aggregation becomes easier 4.* Integration of Earth Sciences

23. OOI-CYBERINFRASTRUCTURE Inst PUCK NetCDF Ocean Sites JMS HTTP OGC Services Web Services API THREDDS Strategy Raw Data OPeNDAP Inst Rover OASIS Telemetry ALOHA (MARS/Seahurst) Client OASIS 2 SSDS Driver OASIS Service SIAM MOOS Proxy SIAM Portal Inst PUCK Service SIAM Strategy Proxy Inst PUCK Service SIAM FOCE AUV Rover Portal AUV Portal Strategy Proxy SIAM Portal SSDS MetadataNetApp Adapter ESB HTTP Web App Adapter Connector ESB Inst Proxy Adapter Strategy Interfaces UserCyberinfrastructure

24. OOI-CYBERINFRASTRUCTURE Inst PUCK NetCDF Ocean Sites JMS HTTP OGC Services Web Services API THREDDS Strategy Raw Data OPeNDAP Inst Rover OASIS Telemetry ALOHA (MARS/Seahurst) Client OASIS 2 SSDS Driver OASIS Service SIAM MOOS Proxy SIAM Portal Inst PUCK Service SIAM Strategy Proxy Inst PUCK Service SIAM FOCE AUV Rover Portal AUV Portal Strategy Proxy SIAM Portal SSDS MetadataNetApp Adapter ESB HTTP Web App Adapter Connector ESB Inst Proxy Adapter Strategy “Give me your tired, your poor, Your huddled masses yearning to breathe free, The wretched refuse of your teeming shore. Send these, the homeless, tempest-tost to me, I lift my lamp beside the golden door!” Huddled Masses = Data, User or Developer?

25. OOI-CYBERINFRASTRUCTURE Inst PUCK NetCDF Ocean Sites JMS HTTP OGC Services Web Services API THREDDS Raw Data OPeNDAP Inst Rover OASIS Telemetry ALOHA (MARS/Seahurst) Client OASIS 2 SSDS Driver OASIS Service SIAM MOOS SIAM Portal Inst PUCK Service SIAM Inst PUCK Service SIAM FOCE AUV Rover Portal AUV Portal SIAM Portal SSDS MetadataNetApp HTTP Web App Adapter ESB Adapter Connector ESB Inst Proxy Adapter Strategy Proxy Strategy Proxy Strategy Proxy

26. OOI-CYBERINFRASTRUCTURE Other Questionnaire Information Security and Policy –Data embargo is about all we could agree on –Data in SSDS is open –Security requirements were difficult to gather Hooks there in SSDS (J2EE, container-managed, LDAP, etc.) Event Detection –What is an Event –Several rounds, MASE is latest QC –We leave mostly to experts due to domain knowledge –SSDS has metadata for basic QC (outliers, etc.) –Users have access to external QC through SSDS Data gaps, long latencies –Handled by OASIS and SIAM, not by SSDS Education and Outreach –Remember interfaces? We haven’t done enough

27. OOI-CYBERINFRASTRUCTURE Conclusion (= Soapbox?) Interfaces! (User mostly) User requirements Do the “easy” things well to pull in users Integrate quickly, then innovate http://www.perceptivepixel.com Questions?