Ocean Observatories Initiative EOI User Review June OOI Cyberinfrastructure Release 1 Scope OOI CI - EOI User Review 16 June 2011

EOI User Review June OOI engineering strategy based on multiple modes of communication

EOI User Review June OOI Science Mission

EOI User Review June OOI Science Mission Ocean Observatories!

EOI User Review June CI: Linking the marine infrastructure to science and users Why a Cyberinfrastructure?

EOI User Review June Why a Cyberinfrastructure?

EOI User Review June OOI Integrated Observatory Observatory Requirements: Provide one integrated observatory interface to all users inside and outside the OOI. Enable the users to investigate observations, manage the observatory and its assets and collaborate with each other in teams. Engineering Drivers: A geographically distributed system of systems with observatories at multiple scales and operational authority. OOI- wide need for data distribution, storage, processing and command and control.

EOI User Review June Observatory Requirements: Enable communities of users to work with the instruments and observatory assets they need from across all OOI sites in a uniform observatory environment. Support multiple “virtual” observatories in parallel. Engineering Drivers: Multiple heterogeneous communities of users accessing shared observatory assets across the OOI. Enable access to observatory assets based on user authorization level and observatory policy. Virtual Observatories Observatory Interface

EOI User Review June OOI Deployment Topology Observatory Requirements Provide observatory assets to support measurements of scientific processes. Provide computational and network assets to enable real- time, secure access and processing for the observatory users Engineering Drivers: P roximity to observatory assets, geographic redundancy. High- bandwidth, low latency access to national and international network peering points. Access to commercial and academic compute clouds

EOI User Review June National & International Observatory Integration NEPTUNE Canada NSF OOI Network NOAA N-Wave Network Global Lambda Integrated Facility Engineering Driver: The topology used by the OOI program is easily integrated into national and international cyber-networks. This will facilitate collaboration for developing the international ocean observing network

EOI User Review June Network Messaging Infrastructure Based on the Layer 2 National Network Infrastructure Provides direct application-level messaging connectivity between components (Capability Containers) within the Integrated Observatory Network, and at users’ sites.

EOI User Review June OOI Network Deployment Engineering Drivers: The physical network links the marine operators, external users, and educators to the ocean assets deployed throughout the world’s oceans.

EOI User Review June What is the CI? (view 2) Observatory Topology

EOI User Review June What's Different about OOI? Integrated Observatory Network

EOI User Review June Event Detection and Response Observatory Requirements: Support closed loop scientific activities. Enable interactive and event-driven control of the observatory assets based on interactive and real-time observations and analyses. Engineering Drivers: Workflows accessing real-time measurements, external data sources, real-time processing, QC and event detection, complex numerical models leading to control of mobile and stationary observatory assets.

EOI User Review June Integrated Observatory Network Integrated Observatory Network: 4 Releases R2 R1 R3 R4 R4: Interactive Ocean Observatory R2: Managed Instrument Network R1: Data Distribution Network R3: On-Demand Measurement Processing

EOI User Review June Integrated Observatory Network Integrated Observatory Network: 4 Releases R2 R1 R3 R4 R4: Interactive Ocean Observatory R2: Managed Instrument Network R1: Data Distribution Network R3: On-Demand Measurement Processing

EOI User Review June Scope Definition Process Requirements defined and allocated to releases Basic descriptions of required capabilities Use cases for the release provide storyboard of capabilities Mapped to requirements to ensure coverage More detail than requirements, but still user-oriented Detailed capabilities are described via task definition process Release Tasks give organization; Iteration Tasks have work details Detailed procedures defined for this process Goals adjusted during Release using scope management process Use cases accomplished according to scope assignment

EOI User Review June Release 1 of ION: Data Distribution Network publish data to ION register data sources find and browse data resources examine data resources initiate data notifications and subscriptions download data from system get live data streams create user accounts scale computing resources perform diagnostics and monitoring receive events without delay 'User-visible' Highlights

EOI User Review June Release 1 of ION: Data Distribution Network “Ingest” data products from external data sources and experimentally from sensors in canonical and raw formats Characterize data sources with their metadata attributes (format, structure) Distribute data via streaming and DAP servers to consumers such as data analysts, numerical modelers Provide a platform for instrument integration development, with instrument control and sensor data acquisition Provide a distributed service integration and execution platform in more than one programming language Technical Perspective

EOI User Review June Release 1 of ION: Data Distribution Network Data distribution via publish/subscribe messages received are immediately distributed processes act on messages as soon as received (workflow!) Scalability: manage growing amounts of work in a graceful way new process host (virtual machines) can be started as needed sets up for automated compute resource management Fault tolerance: continue operation when part of system fails can restart a processing unit, pick up where previous left off Security: protect information from misuse, theft, or corruption basic user and policy mechanisms to enable core model Structural Highlights

EOI User Review June Release 1 Components to+Operations

EOI User Review June CI Components Developed to+Operations Python Capability Container Java Capability Container Access Library Web UI Platform Exchange Messaging System Distributed State Infrastructure Resource Registry Framework Data Publish-Subscribe Framework Event Notification Framework Science Data Persistence and Transport Format Instrument Agent Framework Elastic Processing Unit Virtualized Cloud Management Tools

EOI User Review June Release 1 Product Description Contains use cases that describe intended CI functionality Provides user-oriented descriptions of how things work Organizes the work using operational stories Each use case can serve as the basis of a validation test Shows the goals in enough detail for initial technical direction Organize work among the teams Negotiate understanding of detailed processes Provide context for specific tasks Assess progress ease+1

EOI User Review June Use Cases for Release 1 UC.R1.05Synchronize State Data UC.R1.06Distribute Data Product UC.R1.07Subscribe To Data UC.R1.08Persist Streamed Data UC.R1.01Hello User UC.R1.02Hello Instrument UC.R1.03Hello Data Source UC.R1.18Command An Instrument UC.R1.20Command A Resource UC.R1.04Ingest and Describe Data UC.R1.21 Derive Data Product Externally (Merge Data) UC.R1.09Discover Resource UC.R1.14Use Service Anywhere UC.R1.15Put Services Anywhere UC.R1.16Scale the Processing UC.R1.17Replicate Service UC.R1.25Assure Reliability UC.R1.28Operate System UC.R1.29Monitor System UC.R1.30Troubleshoot System Data Services & Operations Basic Connections

Ocean Observatories Initiative EOI User Review June R1 Product Description Use Cases Moving and Saving Data UC.R1.05Synchronize State DataSynchronize state in distributed data store UC.R1.06Distribute Data ProductData made available to many consumers UC.R1.07Subscribe To DataUser finds data, asks for update notifications UC.R1.08Persist Streamed DataStore streamed data for given period of time Basic Connection and Commands UC.R1.01Hello UserUser gets an ID and logs in UC.R1.02Hello InstrumentInstrument gets ID and is hooked up UC.R1.03Hello Data SourceData source registered and connected UC.R1.18Command An InstrumentSend typical commands to specific instrument UC.R1.20Command A ResourceSend typical commands to specific resource Data Manipulation and Presentation UC.R1.04Ingest and Describe DataExternally provided data read and distributed UC.R1.21 Derive Data Product Externally (Merge Data) Provide data unified from many sources UC.R1.09Discover ResourceUser searches for resources meeting criteria

EOI User Review June R1 Product Description Use Cases Services and Scaling: Elastic Computing UC.R1.14Use Service AnywhereMessages go to services wherever they are UC.R1.15Put Services AnywhereAllocate services where need is greatest UC.R1.16Scale the ProcessingIncrease processing quickly to meet demand UC.R1.17Replicate ServiceConfigure service once, deploy many times Interact, Operate, and Maintain UC.R1.25Assure ReliabilityComputer fails, messages resent, work resumes UC.R1.28Operate SystemConfigure system and respond to requests UC.R1.29Monitor SystemAnticipate issues using monitoring tools UC.R1.30Troubleshoot SystemDiagnose issues using logs, feeds, tools

EOI User Review June

"Hello User" Use Case 1. User registers with the system by providing an (external) identity, creating an internal identity for user. Internally, user has an internal identity and a user "profile"; which contains 'life cycle state' of the user and other user attributes such as contact information and preferences. User authenticates to external identity provider, in order to assert external identity Background: The InCommon federation facilitates the exchange of knowledge that enables the assertion of identities across organizations within this federation. The NSF CI-Logon project is developing cyberinfrastructure software to apply InCommon processes within CI systems, such as OOI. A user, for instance from UCSD, will be able to use her organization credentials to register with OOI, and subsequently use OOI credentials. Each user identity is tracked in the user identity repository. 2. User logs in to the system using identity credential (e.g., login ID and password). 3. User accesses the system's web user interface, which calls system services. Features: Status, Register Resources, Search (User, Instrument, Data, Process), Instrument Access 4. User is assigned their default role in system

EOI User Review June Implications of OOI Approach We have user identities in the system, enabling security We keep user attributes of interest to us (e.g., contact info) We assign attributes as needed (e.g., user roles) ION doesn't perform authentication itself, so … Users don't have to remember a custom OOI identity ION doesn't have to maintain user credentials internally passwords and associated security issues support for all the associated human issues software we don't have to write, or maintain This solves a big scaling issue

EOI User Review June Select Identity Provider and Log On Cornell University Duke University Georgetown University Google

EOI User Review June Identity Provider Logs You In We get a certificate saying who (Google says) you are.

EOI User Review June

Data Distribution Use Case 1. Data is accepted at a specific Acquisition Point CyberPoP or any specific capability container. External data will be accepted via an Instrument Agent or other adapter process, and go through an acquisition process before ingested as complete data set or increment for a data set. Information does not have to be ingested (i.e. made available in the resource registry, normalized) to be distributed via the Data Distribution Network 2. Once minimally described, data is sent to the Exchange by the Data Producer. Ideally data should be fully described before routing, not just minimally described. Further description of the data may already be present in the resource registry of data sets. 3. The Exchange routes data messages to all subscribers to a data stream or filter that match certain criteria in the description of the data message Matching certain criteria is also fundamental to several other use cases, including explicitly in Version a Resource. 4. Subscribers equally see data, whether internal (local or remote) or external users. A data store (or several different stores) is a subscriber to most forms of ingested science data, in no way different to any other subscriber. In case there is no data store subscriber, data will be distributed across the Data Distribution Network without any form of persistence; subscribers are solely responsible for maintaining the information they need.

EOI User Review June Subscribe to Data Use Case 1. Data Consumer identifies material of interest. 2. Variant A: Data Consumer selects existing data stream that provides the material of interest. 3. Variant B: Data Consumer selects historic data set resource … 4. Data Consumer registers (through API or web form) with Subscription service to be delivered material of interest, per a defined set of criteria. 5. Data Consumer specifies subscription delivery modality with Subscription service. 6. Registration is acknowledged via message returned to Data Consumer. 7. Registration is verified through inspection of the Inventory and Data Stream Registry. 8. Data messages meeting registered criteria arrive within Exchange, where they are routed to subscribed Data Consumer. 9. Event notification messages generated from event producing processes (e.g. Instrument Agents, Data Source adapter, event aggregator), arrive within the Exchange, where they are routed to registered Data Consumer.

EOI User Review June Implications of "Data Distribution" User gets data when we get data (no waiting!) Users or modules can subscribe to selected data Processing routines kick off when data arrives … workflows! Fundamental artifacts are data streams (not static sets) ION internalizes the data in its constituent elements Allows fully using the data's natural structure Can present the data according to users' needs All kinds of data sources: models, sensors, observatories, people, … ION can keep a 'reference copy' for fast, reliable access The data, and products, can be well curated (good metadata)

EOI User Review June CI System Design Engineering Milestones Passed Life Cycle Objectives (LCO) Review for Release-1 (Mar 2010) Approval of Release-1 system architecture baseline and use cases Demonstration of critical technologies for risk mitigation Passed Life Cycle Architecture (LCA) Review for Release-1 (Aug 2010) Approval of refined Release-1 use cases and system architecture for Construction Demonstration of an end-to-end running system and a scalable architecture Passed 2 subsystem LCA reviews: Instrument Platform Agent Architecture, External Observatory Integration (December 2010) Passed Initial Operational Capability (IOC) Review (May 2011) Up-to-date approved architecture and design Added CyberPoP and network design; user experience and user interface designs

EOI User Review June CI System Design Artifacts: System Architecture Main System Architecture Artifacts : Architecture Specification (DoDAF structured) baselined and approved; most recently April 2011 for Release-1 as version pages high-level architecture and component design (Confluence, PDF) Tracing to R1 Use Cases and Requirements (in DOORS) Service components for R1-4 identified in schedule : Technology List Specification Drawings (Enterprise Architect, Visio) 2650/2660-nnnnn: CI Level Operational Drawings (53) 28nn/29nn-nnnnn: Subsystem Level Drawings (178) 2990-nnnnn: Terrestrial CyberPoP and Network Drawings (19) Product Description Artifacts Release-1 Product Description Release scope definition based on use cases

EOI User Review June CI System Design Artifacts: User Experience Main User Experience Artifacts User Experience Process Description Release-1 Functional Design Specification Release-1 Functional Detail Specification baselined export from online asset database User Engagement Strategic Plan (Candidate Draft) Design Specifications for workflows (10), screens (26) and panels (72) UX Online Asset Tracking Database: links screens, workflows, services Other User Experience Artifacts and Process User Research Documentation interviewed ~50 users and project scientists from Tempe workshop, Rutgers, SIO, UH, etc. GBs of protected video/audio

EOI User Review June CI Design Progress by Subsystem IDSubsystem # R1 L4 Req’s # R1 Use Cases # Design Pages # Spec Drawings # Design Drawings SASensing & Acquisition ASAnalysis & Synthesis PPPlanning & Prosecution DMData Management COICommon Operating Infrastructure CEICommon Execution Infrastructure IPAInstrument and Platform Agents EOIIOOS Integration Package UXUser ExperienceN/A1* ITVIntegration, Test and VerificationN/A 05 TCNTerrestrial CyberPoPs and Network20N/A81615

EOI User Review June CI Development Progress by Subsystem IDSubsystem # R1 L4 Req’s Satisfied # R1 Use Cases Satisfied % Design complete SLOC # Tasks complete SASensing & Acquisition3 (7%)3 (75%)39%1,19830 ASAnalysis & SynthesisN/A 00 PPPlanning & ProsecutionN/A 00 DMData Management41 (59%)5 (56%)95%23, COICommon Operating Infrastructure31 (26%)4 (36%)95%34, CEICommon Execution Infrastructure14 (56%)8 (100%)77%35, IPAInstrument and Platform Agents14 (41%)1 (100%)100%11,58752 EOIIOOS Integration Package 115 (100%)1 (100%)100%7,79324 UXUser ExperienceN/A0 (0%)80%24,70167 ITVIntegration, Test and VerificationN/A 100%6,984N/A TCNTerrestrial CyberPoPs and NetworkN/A 100%N/A 41

EOI User Review June EOI Requirements in R1 Each External Observatory Agent shall be the logical representation of a specified external observatory The External Observatory Agent shall acquire data from an external observatory The External Observatory Agent shall implement poll mode data transfer The External Observatory Agent shall implement push mode data transfer The External Observatory Agent shall acquire associated metadata for all data transferred from an external observatory The External Observatory Agent shall transform the external observatory data model to the OOI Common Data Model The External Observatory Agent shall transform the external observatory metadata model to the OOI Common Metadata Model The External Observatory Agent shall publish data to the Exchange The External Observatory Agent shall publish invariant metadata to the Exchange The External Observatory Agent shall publish variant metadata to the Exchange when change occurs The External Observatory Agent shall acquire commands from the Exchange The External Observatory Agent shall mediate commands from the CI to the external observatory The External Observatory Agent shall manage the synoptic notion of time The IOOS External Observatory Agent shall process data that conform with the Unidata Common Data Model The IOOS External Observatory Agent shall process data that conform to a described ASCII text model.

EOI User Review June EOI Presentation

EOI User Review June What is the OOI CI Vision?

EOI User Review June OOI CI Development Plan Organization Chart User Experience Team

EOI User Review June OOI CI Development Plan