Ocean Observatories Initiative OOI Cyberinfrastructure Life Cycle Objectives Review January 8-9, 2013 Data Products and Attributes 1 Karen Stocks Data Curator

OOI Cyberinfrastructure Life Cycle Objectives Review January 8-9, 2013 “Effective management and storage of data are fundamental requirements” - OOI Data Management Plan. DCN

OOI Cyberinfrastructure Life Cycle Objectives Review January 8-9, 2013 OOI Data Products 3

OOI Cyberinfrastructure Life Cycle Objectives Review January 8-9, 2013 Core Data Products Produced by OOI owned and operated instruments OOI is responsible for the long-term provision of these data products e.g salinity, vent fluid temperature, PCO2, seafloor pressure (~90) Undergo quality control procedures Defined by Data Product Specifications… 4

OOI Cyberinfrastructure Life Cycle Objectives Review January 8-9, 2013 Data Product Specifications Produced by Project Scientists, implemented by CI. Reviewed, controlled documents 5

OOI Cyberinfrastructure Life Cycle Objectives Review January 8-9, 2013 Data Processing Levels Raw: The full “conversation” with instrument. incl. raw data Parsed Canonical: Data streams converted to internal form (e.g. hex to float); still bundles multiple parameters (interleaved). Often separate streams defined for data vs. instrument status, calibration DPS L0: Unprocessed data. Data that are in instrument/sensor units and at instrument/sensor resolution. They are sensor by sensor (unpacked and/or de- interleaved) and are available in OOI supported formats (e.g., NetCDF) L1: Basic Data: L0 data that have been calibrated, are in well-defined scientific units and may have some QC applied L2: Derived Data Products: Data that have been manipulated and/or combined and include new values not present in original data 6

OOI Cyberinfrastructure Life Cycle Objectives Review January 8-9, 2013 Quality Control Levels a: no QC b: programmatic/automated QC e.g. spike test, range test, drift test c: human-in-the-loop 7

OOI Cyberinfrastructure Life Cycle Objectives Review January 8-9, 2013 Categories of data in OOI Core data products from OOI owned and operated instruments – both raw and derived. Additional data from OOI owned and operated instruments (ancillary data, engineering data, instrument status data) Data from External Observatories (e.g. IOOS, NEPTUNE, WMO) Data from PI-owned instruments deployed on OOI platforms Additional user-provided or user-derived data 8

OOI Cyberinfrastructure Life Cycle Objectives Review January 8-9, 2013 OOI Metadata 9

OOI Cyberinfrastructure Life Cycle Objectives Review January 8-9, 2013 Metadata goals for Data Products Assure the capture of appropriate information for the data to be found, and used, including assessing fitness for use, on the decadal timeframe. Enable interoperability Borrow from existing work vs. reinventing a local solution (ISO, OGC, IOOS, MBARI, NEON, QARTOD, R2R, SDN, etc. etc. ) 10

OOI Cyberinfrastructure Life Cycle Objectives Review January 8-9, 2013 OOI Metadata Approach IO operates on “Resources” Resources can be digital items held in the system (data products, documents), or representations of physical items (instruments, observatories) 11

OOI Cyberinfrastructure Life Cycle Objectives Review January 8-9, 2013 Associations Resources have associations to other resources. Examples: An instrument device is associated to the platform it is deployed on A Data Product is associated with the process that created it (for processed products) 12

OOI Cyberinfrastructure Life Cycle Objectives Review January 8-9, 2013 Resource Model 13

OOI Cyberinfrastructure Life Cycle Objectives Review January 8-9, 2013 Attributes Resources have attributes, i.e. “traditional” metadata. Examples: Instrument Models have a manufacturer Deployments have start and end dates Observatories have contact(s) Parameters have units 14

OOI Cyberinfrastructure Life Cycle Objectives Review January 8-9, 2013 Tracking Attributes Resources and Attributes managed in a relational database that is kept in alignment with the CIUX database & the YML code. Name, definition, type, controlled vocabulary, reference, source In R3, attributes will become a Resource ~149 Resources; 95 excluding abstract base; 35 primarily user-facing 410 attributes; 95 primarily user-facing 15

OOI Cyberinfrastructure Life Cycle Objectives Review January 8-9, 2013 Attachments Resources can have one or many attachments. Examples: Instrument Models can have user manuals Instrument Devices can have scanned factory calibration sheets Core Data Products have Data Processing Specifications 16

OOI Cyberinfrastructure Life Cycle Objectives Review January 8-9, 2013 R2.0 17

OOI Cyberinfrastructure Life Cycle Objectives Review January 8-9, 2013 R2.0 metadata approach Collect metadata required to support NetCDF/CF + ACDD, ISO 19115, and OGC CSW Plus additional attributes needed to meet the 2.0 Use Cases & Requirements (i.e. manage the system, and operate instruments) 18

OOI Cyberinfrastructure Life Cycle Objectives Review January 8-9, 2013 R2.0 Data Product Attributes Identity: ID, various names, description, keywords, processing code, QC level, reference URL(s) geospatial and temporal descriptors available formats provenance events acknowledgement & use license housekeeping: lifecycle state, contact(s), proprietary hold info, external archiving info Parameters… 19

OOI Cyberinfrastructure Life Cycle Objectives Review January 8-9, 2013 Parameter Attributes Identity IDs: primary ID, Human-readable identifier (automatically composed from other attributes), Alternate IDs. Names: Display name, CF Standard name, long name, OOI Data product identifier Description Reference URLs Housekeeping Date created Date modified Recent events User notification requests (list) Life cycle state 20 Use Type: e.g. quantity, category Value encoding: e.g. int32, float64 Code Set: for category parameters, it defines the categories and their meanings Unit of Measure Fill value

OOI Cyberinfrastructure Life Cycle Objectives Review January 8-9, 2013 R2.1 21

OOI Cyberinfrastructure Life Cycle Objectives Review January 8-9, 2013 R2.1 Improvements Overall, largely stable with respect to user-facing resources Additions/refinements stemming from: high priority user-identified gaps (next slide) platform-specific deployment information (e.g. cruise metadata) a small number of calculated attributes deferred out of R2.0 (e.g. last data ingestion time, attachment size) minor refinements for consistency (e.g. using the same set of contact information in different contexts) (Populating via preload pages) 22

OOI Cyberinfrastructure Life Cycle Objectives Review January 8-9, 2013 R2.1 UX Proposed Attributes Summary status indicators counts of critical alerts, warning alerts, help tickets aggregated status: Red-Yellow-Green status indicator summarizing all other status assessment Improved Device history representation: location history, power history, data history 23

OOI Cyberinfrastructure Life Cycle Objectives Review January 8-9, 2013 R3 24

OOI Cyberinfrastructure Life Cycle Objectives Review January 8-9, 2013 R3 Expansion: metadata “externalizations” In R2, the information needed to support key standards (ISO 19115, Net CDF/CF +ACDD, and OGC CSW) is captured. In R3 the processes will be created to map the internal attributes to external labels and formats (e.g. XML) to create metadata reports meeting different specifications 25

OOI Cyberinfrastructure Life Cycle Objectives Review January 8-9, 2013 R3 Expansion: Semantic support Controlled vocabularies/ontologies and mapping between controlled vocabularies/ontologies are required to: improve data discovery (e.g. temperature vs SST vs sea water temperature) facilitate data entry & ensure consistency (e.g. drop-down lists of instrument manufacturer names) verify metadata quality (e.g. IOOS parameter name entries must be from the valid IOOS parameter list) Provide referencing (e.g. URLs to external community definitions) Facilitate Interoperability – linking OOI concepts to externally- defined concepts 26

OOI Cyberinfrastructure Life Cycle Objectives Review January 8-9, 2013 R3 Expansions: Semantic support, cont. Vocabularies/Ontologies will become first class resources within ION (versioning, URIs, etc.) 27

OOI Cyberinfrastructure Life Cycle Objectives Review January 8-9, 2013 R3 Expansions: New resource types sampling plans/mission plans physical samples Instrument calibration and service history annotations workflows numerical models self-reported instrument metadata (e.g. PUCK) additional provenance representation 28

OOI Cyberinfrastructure Life Cycle Objectives Review January 8-9, 2013 R3 Expansions: other Universal Resource Identifiers (e.g. DOIs) for both closed an aggregating data products. Setting and enforcing minimal metadata standards for different data categories (external observatories, PI-owned instruments, etc.) Evaluating the importance of supporting additional standards (ongoing) 29

OOI Cyberinfrastructure Life Cycle Objectives Review January 8-9, 2013 References Data Management Plan (Alfresco) Data Management Plan Metadata Implementation Plan (Alfresco) Metadata Implementation Plan Resource Model ay/syseng/CIAD+APP+Resource+Model Data Product Specifications: ay/science/Data+Product+Specifications ay/science/Data+Product+Specifications 30