OOI CI LCA REVIEW August 2010 Ocean Observatories Initiative OOI Cyberinfrastructure Common Operating Infrastructure Subsystem Michael Meisinger, Brian Fox, Stephen Pasco, Dorian Raymer, Munindar Singh, Kartik Tadanki Life Cycle Architecture Review La Jolla, CA

OOI CI LCA REVIEW August Agenda Subsystem Purpose Release 1 Product Description Use Case Overview Architectural Overview Status of Progress by Service Key Achievements, Technology Challenges Emergent Risks

OOI CI LCA REVIEW August Common Operating Infrastructure: Purpose Provide the integration & communication environment for all the other subsystem services and interface processes

OOI CI LCA REVIEW August User Interfaces Core Infrastructure Services COI Purpose: Infrastructure Common Operating Infrastructure Data Distribution and Preservation Infrastructure Common Execution Infrastructure Data Management Application Level Services Externalization and Integration Instrument Agents External Observatory Integration Sensing & Acquisition

OOI CI LCA REVIEW August COI R1 Product Description Use Case Overview IDTitleDescription UC.R1.01Hello UserUser gets an ID and logs in UC.R1.05Synchronize State DataSynchronize state in distributed data store UC.R1.10Define InteractionDescribe pattern of interaction between actors UC.R1.11Define New ServiceAdd new service to system capabilities UC.R1.14Use Service AnywhereMessages go to services wherever they are UC.R1.31Assert Access PolicyDefine access policy for given resource UC.R1.32Conduct NegotiationNegotiate agreement (or not) between agents UC.R1.33Enroll in an OrgEnter as a member into an Organization (Org) UC.R1.34Share an Org ResourceCollaborate with Org member offering a resource UC.R1.35 Share Affiliated Orgs' Resources Orgs form a contract to allow resource sharing UC.R1.36Create an OrgCreate an Org with defined characteristics Responsible For

OOI CI LCA REVIEW August COI R1 Product Description Use Case Overview IDTitleDescription UC.R1.06Distribute Data ProductData made available to many consumers UC.R1.07Subscribe to DataUser finds data, asks for update notifications UC.R1.09Discover ResourceUser searches for resources meeting criteria UC.R1.12Annotate DataInformation or description is added to resource UC.R1.15Put Services AnywhereAllocate services where need is greatest UC.R1.17Replicate ServiceConfigure service once, deploy many times UC.R1.18Command An InstrumentSend typical commands to specific instrument UC.R1.20Command A ResourceSend typical commands to specific resource UC.R1.24Version A ResourceResource is supplanted by changed version UC.R1.25Assure ReliabilityComputer fails, messages resent, work resumes UC.R1.26Virtualize EverythingVirtual processes embody all system services UC.R1.27Configure Access PageConfigure user’s web page that accesses ION UC.R1.28Operate SystemConfigure system and respond to requests UC.R1.29Monitor SystemAnticipate issues using monitoring tools UC.R1.30Troubleshoot SystemDiagnose issues using logs, feeds, tools Supports

OOI CI LCA REVIEW August COI Development Scope of Release 1 Provide a “capability container” for service and agent integration in Python and Java Secure reliable asynchronous messaging Distributed state management: “persistence layer” Resource registration Governed resource sharing and access based on flexible identity and policy management Basic user interface platform

OOI CI LCA REVIEW August Architectural Overview

OOI CI LCA REVIEW August Capability Container and Subsystems

OOI CI LCA REVIEW August COI Services and Components Service Framework Resource Management Presentation Framework Capability Container Exchange Identity Management Governance Framework Distributed State Managemt OV2 CI

OOI CI LCA REVIEW August COI Technology List Capability Container Python, Twisted, txAMQP Java, Spring Open Telecom Platform (OTP) architecture FIPA ACL Message Format (standard headers) in Common Format Messaging RabbitMQ AMQP broker (with federation extensions) Google Protocol Buffers Policy and Governance Jess Rules engine Identity Management CILogon Internet2 Security infrastructure Distributed State Management Cassandra Key Value Store GIT compatible Content Addressable Storage (CAS) architecture Presentation Framework Grails platform (Java/Groovy)

OOI CI LCA REVIEW August COI R1 Use Case Status IDTitleDescription UC.R1.11Define New ServiceAdd new service to system capabilities UC.R1.14Use Service AnywhereMessages go to services wherever they are UC.R1.01Hello UserUser gets an ID and logs in UC.R1.05Synchronize State DataSynchronize state in distributed data store UC.R1.10Define InteractionDescribe pattern of interaction between actors UC.R1.31Assert Access PolicyDefine access policy for given resource UC.R1.32Conduct NegotiationNegotiate agreement (or not) between agents UC.R1.33Enroll in an OrgEnter as a member into an Organization (Org) UC.R1.34Share an Org ResourceCollaborate with Org member offering a resource UC.R1.35 Share Affiliated Orgs' Resources Orgs form a contract to allow resource sharing UC.R1.36Create an OrgCreate an Org with defined characteristics

OOI CI LCA REVIEW August COI Status by Service / Component Capability Container Messaging, Common Message Format Service Framework Distributed State Management, Common Object Model Resource Management Security and Identity Management Governance Framework Presentation Framework

OOI CI LCA REVIEW August Capability Container, Exchange, Service Framework Relevant Use Cases: UC.R1.11 Define New Service UC.R1.14 Use Service Anywhere Support: All other use cases

OOI CI LCA REVIEW August Capability Container and Messaging

OOI CI LCA REVIEW August Python Capability Container Technologies Python 2.5 Twisted Infrastructure Start the container with arguments “Spawn” a set of processes Define and start a “process” (from a Python module) Attach a process to messaging to send and receive Process life cycle management (parent, init, shutdown) CC Agent responsible for the container management Heartbeat monitoring, integration with Execution infrastructure

OOI CI LCA REVIEW August Java Capability Container Technologies Java 1.6 OSGi service container AMQP messaging library Infrastructure Core library with OOI messaging format and data object serialization encoding/decoding Spring MVC container for defining services as bundles with full OSGi management capability Full interoperability Java-Python

OOI CI LCA REVIEW August Exchange: Secure Reliable Messaging Capability (Service) Capability (Service) Agent (Process) Agent (Process) UI Client (Process)

OOI CI LCA REVIEW August AMQP Messaging

OOI CI LCA REVIEW August Capability Container: Processes and Messaging

OOI CI LCA REVIEW August Common Message Format All Exchange messages have common headers Based on ACL FIPA message structure spec Identifies Interaction participants Message operation Message encoding, structure Interaction and conversation type Identity management and security headers Encoding Google Protocol Buffers Message handling in container goes through interceptors After a service sends a message, before it receives a message Message signing and validation Policy enforcement Governance tracking

OOI CI LCA REVIEW August Example Message AMQP Headers: Method(name=deliver, id=60): False, 'magnet.topic', 'mm.instrument_management'), children=[], properties={'content type': 'application/msgpack', 'content encoding': 'binary'}, delivery_info={}, delivery_tag=6, headers=None, reply_to=None, OOI Headers: accept-encoding='application/ion-jsond', conv- id='#22', conv-seq=1, encoding='application/ion-jsond', op='start_instrument_agent', receiver='mm.instrument_management', reply-to='10a79b88-d2f0- 4d3d bb2e7fb3bb.1', sender='10a79b88-d2f0-4d3d bb2e7fb3bb.1', Content: instrumentID='602d286b-dc f92-b37c1b02ba95', model='SBE49'

OOI CI LCA REVIEW August Exchange Implementation Status Technologies RabbitMQ message broker (supporting AMQP 0.9.1) Federated RabbitMQ broker prototype for Exchange Spaces Integration Python and Java “ioncore” messaging libraries, part of the capability containers Naming abstraction for processes doing messaging Unique location-independent identifiers for processes, additional “known” messaging names for services Direct, worker queue and fanout messaging pattern

OOI CI LCA REVIEW August Distributed State Management Relevant Use Cases: UC.R1.05 Synchronize State Data Supports: UC.R1.14 Use Service Anywhere All other use cases

OOI CI LCA REVIEW August Distributed State Management Technologies Backend: Cassandra Key-Value Store Object repository: GIT architecture Integration (see DM presentation) Storage Persistence Model: Content Addressable Store Structured object abstraction Structure definition and identification Transport with flexible encoding Persistence

OOI CI LCA REVIEW August Distributed State: Content Addressable Store

OOI CI LCA REVIEW August Resource Management Relevant Use Cases: No direct use case Support: All use cases

OOI CI LCA REVIEW August CI Resources “CI governed” Resource: Resource Structured Data Object Standard OOI metadata attributes Resource specific attributes References to other resources Categories Information resource Stateful resource Taskable resource

OOI CI LCA REVIEW August Resource Management Services Resource Agent Resource Registry Resource (external) OV2 COI

OOI CI LCA REVIEW August Resource Registry Service Operations Operations register_resource (Resource description) get_resource (Resource Reference) get_resource_by_id (ID string) find_resource (Resource query description) Data Objects Resource Instance – filled out typed attributes or default Resource Reference – to a specific version or the most recent Resource Query – filter on resource attribute values (or regex) Resource Description – resource type

OOI CI LCA REVIEW August Security and Identity Management Relevant Use Cases: UC.R1.01 Hello User

OOI CI LCA REVIEW August IdM Architecture Overview Technologies InCommon, IdM federation of choice for U.S. higher ed. today. CILogon, builds on InCommon to support thick clients. Security messaging leverages XML Security Messaging

OOI CI LCA REVIEW August IdM Secure Messaging Overview

OOI CI LCA REVIEW August Federated Facility and Governance Relevant Use Cases UC.R1.31Assert Access Policy UC.R1.32Conduct Negotiation UC.R1.33Enroll in an Org UC.R1.34Share an Org Resource UC.R1.35Share Affiliated Orgs' Resources UC.R1.36Create an Org

OOI CI LCA REVIEW August Governance: Resource Agents Resource Agent Resource Agent Proxy Agent Capability Container Capability Container Capability Container Capability Container

OOI CI LCA REVIEW August Governance: Interaction Management Technologies Graffle: Language for specifying interactions Example: import ExternalIdentityCredential; import IdentityCredential as Credential; protocol RegisterUser { role UnregisteredEndUser:OOIUser as User; role IntegratedObservatoryOperator:OOIOperator as Operator; pre { User.registered == false } ecred:ExternalIdentityCredental from User to Operator; choice from Operator to User { { externalSaysValid(ecred) } } ok(): { cred:Credential { cred==generateInternalCredential(ecred) } from Operator to User; post { User.registered == true } /// assertion [ Operator.UIR.add(cred, generateProfile(cred)) ] // action } ko() { not externalSaysValid(ecred) } : { ; } }

OOI CI LCA REVIEW August Presentation Framework Relevant Use Cases UC.R1.01 Hello User

OOI CI LCA REVIEW August Presentation Framework Implementation Technologies Engineering UIs and Provisional R1 UIs Grails (Groovy on Rails) Java platform 1.6 Integration Automatic build and deployment Core messaging library “ioncore” started as service Establishes a connection to the message broker Calls services, interacts with Python side Model-View-Controller (MVC) UI design

OOI CI LCA REVIEW August 2010 Presentation Framework Screenshot 39

OOI CI LCA REVIEW August Key Achievements, Technical Challenges Key Achievements Robust messaging: AMQP / RabbitMQ Definition of a process abstraction Distributed data store and structured object transport Resource Registries Presentation UI Framework Python-Java interoperability Technical Challenges Cassandra Key-Value Store Stability Identity Management technology complexity Rules engines (Jess, Pyke) for Governance rules Scribble: Maturity of the language for interaction specification of all core interactions; application for interaction monitoring

OOI CI LCA REVIEW August 2010 Risk Assessment - COI Use Cases COI IDNameDescirption Risk of Availability Level of Maturity Target Use UC.R1.1Hello UserUser gets an ID and logs inLowExpectedUser UC.R1.14Use Service AnywhereMessages go to services wherever they areLowExpectedDeveloper UC.R1.11Define New ServiceAdd new service to system capabilitiesMediumExpectedDeveloper UC.R1.5Synchronize State DataSynchronize state in distributed data storeMediumNecessaryTest UC.R1.29Monitor SystemAnticipate issues using monitoring toolsMediumNecessaryOperator UC.R1.31Assess Access PolicyDefine access policy for a given resourceMediumNecessaryOperator UC.R1.33Enroll in an OrgEnter as a member into an Organization (Org)MediumNecessaryTest UC.R1.36Create an Org Create an Organization (Org) with defined characteristicsMediumNecessaryDeveloper UC.R1.10Define InteractionDescribe pattern of interaction between actorsHighDiminishedTest UC.R1.32Conduct NegotiationNegotiate agreement (or not) between agentsHighDiminishedTest UC.R1.34Share an Org ResourceColloborate with Org member offering a resourceHighDiminishedTest UC.R1.35 Share Affiliated Orgs' ResourcesOrgs form a contract to allow resource sharingHighDiminishedTest

OOI CI LCA REVIEW August 2010 Risk Assessment - COI Services COI Name Risk of Availability Level of MaturityTarget Use CC: Presentation FrameworkLowExpectedOperator Resource Registry and CatalogLowNecessaryUser CC: Service FrameworkMediumExpectedDeveloper CC: Capability ContainerMediumExpectedDeveloper CC: Exchange, MessagingMediumExpectedDeveloper CC: Distributed State ManagementMediumNecessaryDeveloper Identity & Policy ManagementHighNecessaryDeveloper Federated Facility (Virtual Organization)HighDiminishedTest

OOI CI LCA REVIEW August Thanks ! Questions ?

OOI CI LCA REVIEW August Capability Container Components (1)

OOI CI LCA REVIEW August Capability Container Components (2)

OOI CI LCA REVIEW August Resource Agent Services

OOI CI LCA REVIEW August Policy and Governance Services

OOI CI LCA REVIEW August Exchange Space Exchange Space is comprised of –Distributed Application Facility (DAF) –Distributed IPC Facility (DIF)

OOI CI LCA REVIEW August Back-End Infrastructure

OOI CI LCA REVIEW August The “Exchange” Applications communicate through Exchange Spaces Exchange Spaces are namespaces of “communicators” Applications need to enroll in Exchange Spaces Governance applies within Exchange Spaces

OOI CI LCA REVIEW August Exchange Space and Points Exchange Points are the message routing and queuing resources of Exchange Spaces Communicators play the role of Producer, Consumer, and Distributer (Broker)

OOI CI LCA REVIEW August Messaging Service Interfaces

OOI CI LCA REVIEW August Risk Mitigation Development Out of the box: RabbitMQ AMQP message broker Python: flexible and powerful applications Twisted: distributed application framework txAMQP: messaging library Integration Message abstraction for services Intercepting message handler (in and out) Policy and governance integration (via agents) Development console

OOI CI LCA REVIEW August Enrolling in an Exchange Space OV6 COI

OOI CI LCA REVIEW August Send a message OV6 COI

OOI CI LCA REVIEW August Receive a message OV6 COI

OOI CI LCA REVIEW August Plan for Construction End-to end integration with all subsystems Separation the COI "ION-Core" from other COI services and release as module Stabilize core, container and service interfaces Apply IdM and policy enforcement throughout the system pervasively; integrate in capability container and leverage resource agents Define an initial facility and apply facility governance Provide a UI development platform for other subsystems Provide a unit test, integration test and operational execution platform with scalability test harness Build out the Java capability container R1C1. Full software integration and refactoring for production release R1C2. Functional completeness and end-to-end integration with subsystems R1C3. Robustness and governance

OOI CI LCA REVIEW August Messaging Abstraction Capability Container Capability Container Capability Container Send Message From: “name2” To: “name4” In: Exchg-Space1 Action: “invoke-service”