1. Ocean Observatories Initiative OOI Cyberinfrastructure Data Management Michael Meisinger & David Stuebe OOI Cyberinfrastructure Life Cycle Objectives Milestone Review, Release 1 San Diego, CA February 23-25, 2010

2. OOI CI LCO Review, Feb 2010 2 Overview Michael Meisinger Architectural Context David Stuebe Functional Scope in R1 DM Architecture Design and Feasibility OOICI Data Model

3. Ocean Observatories Initiative Data Management Architecture Overview

4. OOI CI LCO Review, Feb 2010 4 Overview The Role of DM in the System Scope in Release 1 Common Data and Metadata Model Core Services Example Scenario

5. OOI CI LCO Review, Feb 2010 5 The Role of DM in the System Wherever information is required in the system (e.g., science data, metadata, associations, user identities, virtual machine images) –Provide access in the form needed –Distribute –Transform –such that no data is ever lost and is available where need and when needed DM is core infrastructure and science supporting application

6. OOI CI LCO Review, Feb 2010 6 Subsystem Dependencies 2650-00008 OV2 CI Sensing & Acquisition Data Management Common Operating Infrastructure Common Execution Infrastructure Invoke Services, Dataset Ingest/Transform/ Present, Dataset Pub/Sub, Repository Service Process Execution, Capability Container Data Management

7. OOI CI LCO Review, Feb 2010 7 Scope of Release 1 Define the initial, extensible common data and metadata model for OOI Dynamic data distribution services Provide a data catalog and repository Basic Persistent Storage

8. OOI CI LCO Review, Feb 2010 8 Outlook: DM Releases 2 & 3 Advanced persistent archive services Search and navigation, i.e., browsing and discovering datasets by metadata and content External data access services: interoperability with external communities and applications Aggregation service, i.e. classification, categorization, and general grouping of data into collections Attribution and association services, i.e., adding metadata attributes for resources Out of scope for release 1

9. OOI CI LCO Review, Feb 2010 9 Information Resources “CI governed” Resource Standard and user metadata attributes (in OOI convention) References to other resources Categories –Information resource –Physical (stateful) resource –Taskable resource 2650-00039 OV7 CI

10. OOI CI LCO Review, Feb 2010 10 Data Management Services (Dataset View) 2910-00001 OV2 DM IngestionTransformationPresentation Distribution Inventory Preservation

11. OOI CI LCO Review, Feb 2010 11 Data Management Services (Distribution View) 2910-00002 OV2 DM IngestionTransformationPresentation Distribution InventoryPreservation

12. OOI CI LCO Review, Feb 2010 12 Use Cases Directly in support of the data scenario –Define a dataset –Fetch data and metadata from an external data source –Ingest data and metadata from an external data source –Ingest data from a real-time data source –Distribute increments of a dataset on a data stream –Access data via a catalog. Query and discover and retrieve

13. Ocean Observatories Initiative Data Management Development Overview

14. OOI CI LCO Review, Feb 2010 14 Overview Michael Meisinger Architecture David Stuebe Functional Scope in R1 DM Architecture Design and Feasibility OOICI Data Model

15. Functional Scope in R1 OOI CI LCO Review, Feb 2010 15 Matt Howard OceanService UMASSD CoolRoom OOI CI free-extras

16. DM Architecture OOI CI LCO Review, Feb 2010 16 free-extras OOI CI

17. Functional Scope in R1 User (IOOS) Find data in the catalog Subscribe to a CI data source Notification of new data availability Access data using DAP, WMS Access data w/streaming protocol OOI CI LCO Review, Feb 2010 17

18. Functional Scope in R1 Scientist (IOOS modeler) Add metadata to define new data source Publish data from the source Set policy for collaboration using that data OOI CI LCO Review, Feb 2010 18

19. Functional Scope in R1 Operator Configure user data management policies Configure dataset caching Configure physical storage strategy –Replication –Backup –Holding period OOI CI LCO Review, Feb 2010 19

20. Functional Scope in R1 Service Data store can persist and retrieve data Block transfer of data Streaming transfer of data Present data & data catalog Ingest external data Transform data OOI CI LCO Review, Feb 2010 20

21. OOI CI LCO Review, Feb 2010 21 Overview Michael Meisinger Architecture David Stuebe Functional Scope in R1 DM Architecture Design and Feasibility OOICI Data Model

22. DM Architecture OOI CI LCO Review, Feb 2010 22 Matt Howard OceanService UMASSD CoolRoom OOI CI free-extras

23. DM Architecture OOI CI LCO Review, Feb 2010 23 free-extras OOI CI

24. DM Architecture OOI CI LCO Review, Feb 2010 24 S&AP&PA&S DM COICEI free-extras

25. DM Architecture OOI CI LCO Review, Feb 2010 25 S&AP&PA&S COICEI IngestionTransformationPresentation DistributionInventory Preservation free-extras

26. DM Architecture OOI CI LCO Review, Feb 2010 26 IngestionTransformationPresentation Distribution Inventory Preservation 2910-00001 OV2 DM Services

27. DM Architecture OOI CI LCO Review, Feb 2010 27 IngestionTransformationPresentation Distribution InventoryPreservation 2910-00002 OV2 DM Services

28. DM Architecture OOI CI LCO Review, Feb 2010 28 Distribution UI Publish Registration Subscription Registration Notification Registration Data Stream Registry COI: Resource Registry Data Stream Routing COI: Exchange 2910-00014 OV2 DM Distribution DM: Inventory

29. DM Architecture OOI CI LCO Review, Feb 2010 29 2910-00020 OV5 DM Register …

30. DM Architecture OOI CI LCO Review, Feb 2010 30 2910-00022 OV6 DM Registration

31. DM Architecture OOI CI LCO Review, Feb 2010 31 S&AP&PA&S DM COICEI free-extras

32. OOI CI LCO Review, Feb 2010 32 Overview Michael Meisinger Architecture David Stuebe Functional Scope in R1 DM Architecture Design and Feasibility OOICI Data Model

33. Design and Feasibility OOI CI LCO Review, Feb 2010 33 LDM Application Level Technologies to Leverage pydap Transformation & Presentation Ingestion Java CF Metadata

34. Design and Feasibility OOI CI LCO Review, Feb 2010 34 Infrastructure Level Technologies to Leverage Proto Buffers Inventory & Resource Registry Message Encodings Development Language Transport

35. Design and Feasibility OOI CI LCO Review, Feb 2010 35 Preservation Technologies to Leverage Storage Resource Management Specification Compliant Berkeley Storage Manager

36. Design and Feasibility OOI CI LCO Review, Feb 2010 36 Presentation & Discovery Technologies to Leverage CUAHSI

37. Design and Feasibility OOI CI LCO Review, Feb 2010 37 Presentation & Discovery Technologies to Leverage OGCSEARCH

38. OOI CI LCO Review, Feb 2010 38 Overview Michael Meisinger Architecture David Stuebe Functional Scope in R1 DM Architecture Design and Feasibility OOICI Data Model

39. OOI CI LCO Review, Feb 2010 39 We have to get this right.

40. OOICI Data Model OOI CI LCO Review, Feb 2010 40 Common Data Model DAP 2HDF 5

41. OOICI Data Model OOI CI LCO Review, Feb 2010 41 ISO 19123 Coverage Types

42. OOICI Data Model OOI CI LCO Review, Feb 2010 42 The existing data models have a physical data dependence, an artificial coupling between algorithm and representation. The GridFields model, separates the scientific question from how to solve it in order to improve interoperability and performance.

43. Data Management OOI CI LCO Review, Feb 2010 43 S&AP&PA&S DM COICEI free-extras

44. OOI CI LCO Review, Feb 2010 44 Thanks !