1. Ocean Observatories Initiative OOI Cyberinfrastructure Overview Matthew Arrott VMware Presentation March 5, 2010

2. OOI CI LCO Review, Feb 2010 2 Goals for this Presentation

3. OOI CI LCO Review, Feb 2010 3 Overview The OOI Integrated Observatory OOI Network Architecture OOI Logical Architecture

4. OOI CI LCO Review, Feb 2010 4 Distributed System of Systems Building Blocks –Messaging Fabric –Exchange Spaces –Capability Container –Execution Environments –Federation and Governance

5. OOI CI LCO Review, Feb 2010 5 OOI Program Overview

6. OOI CI LCO Review, Feb 2010 6 OOI Integrated Observatory

7. OOI CI LCO Review, Feb 2010 7 Integrated Observatory

8. OOI CI LCO Review, Feb 2010 8 Science & Education Operations Interactive Ocean Observing Interactive Ocean Modeling & Data Assimilation Automated Data Product Generation Discipline-Driven Semantic Organization of Data Interactive Instrument Network Integrated Observatory Management User-Driven Integration of Resource Observatory Activity Model

9. 9 Goal after Release 5

10. OOI CI LCO Review, Feb 2010 10 Network Architecture

11. OOI CI LCO Review, Feb 2010 11 Network Architecture

12. OOI CI LCO Review, Feb 2010 12 International Research Networks

13. OOI CI LCO Review, Feb 2010 13 National Research Networks

14. OOI National Network OOI CI LCO Review, Feb 2010 14

15. OOI CI LCO Review, Feb 2010 15 Network Deployment

16. OOI CI LCO Review, Feb 2010 16 OOI Logical Architecture Architecture documentation available at: www.oceanobservatories.org/spaces/display/syseng/Archit ecture+and+Design

17. OOI CI LCO Review, Feb 2010 17 Release-1 OOI CI Functions Instrument and platform interfaces Instrument and platform control from the observatory Direct access to instruments and platforms Data acquisition from sensors and external observatories Management of science data and other observatory information Data inventory and metadata annotation Data ingestion, i.e. canonicalization and metadata annotation Data discovery, cataloging and presentation Data distribution, replication, streaming Data persistence, assurance, retrieval Process and service execution Virtualization of computing Elastic scaling to demand High availability services Service-orientation Reliable message based asynchronous communication Security: identity and policy management Multi facility system integration Sensing & Acquisition Data Management Common Operating Infrastructure Common Execution Infrastructure Interface Adapters Application Infrastructure

18. OOI CI LCO Review, Feb 2010 18 Resource – the Base Object “CI governed” Resource Standard and user metadata attributes (in OOI convention) References to other resources Categories –Information resource –Physical (stateful) resource –Taskable resource

19. OOI CI LCO Review, Feb 2010 19 The Capability Container Ingredients: Capability: function, service, component Plumbing: Messaging, security Container: drop it all in as needed Packaging: label it, tape it shut… done!

20. OOI CI LCO Review, Feb 2010 20 Secure Reliable Messaging Capability Container Capability Container Capability Container Capability Container Capability Container Capability (Service) Capability (Service) Capability (Service) Capability (Service) Capability (Service)

21. OOI CI LCO Review, Feb 2010 21 The “Exchange” Applications communicate through Exchange Spaces Exchange Spaces are namespaces of “communicators” Applications need to enroll in Exchange Spaces Governance applies within Exchange Spaces

22. OOI CI LCO Review, Feb 2010 22 Exchange Space and Points Exchange Points are the message routing and queuing resources of Exchange Spaces Communicators play the role of Producer, Consumer, and Distributer (Broker)

23. OOI CI LCO Review, Feb 2010 23 Exchange Space Exchange Space is comprised of –Distributed Application Facility (DAF) –Distributed IPC Facility (DIF)

24. DIF Member Process OOI CI LCO Review, Feb 2010 24

25. OOI CI LCO Review, Feb 2010 25 Messaging Abstraction Capability Container Capability Container Capability Container Send Message From: “name2” To: “name4” In: Exchg-Space1 Action: “invoke-service”

26. OOI CI LCO Review, Feb 2010 26 Common Message Format Capability Container –Provides message handling through interceptors After a service sends a message, before it receives a message Message signing and validation Policy enforcement Governance tracking –Provides a common message format for all CI messages Based in ACL FIPA message structure Content, encoding, ontology

27. OOI CI LCO Review, Feb 2010 27 Services and Resources as Agents Resource Agent Resource Agent Proxy Agent Capability Container Capability Container Capability Container Capability Container

28. OOI CI LCO Review, Feb 2010 28 Scenario An instrument, a physical resource, is represented by an agent to the system and its users Users request control of the instrument Capabilities are projected into another domain of authority by a proxy agent

29. OOI CI LCO Review, Feb 2010 29 COI Infrastructure Services The Common Operating Infrastructure (COI) is the integration & communication environment for all the other subsystem services

30. OOI CI LCO Review, Feb 2010 30 Integration Strategy Capability Container Secure reliable messaging “Exchange” Service-oriented architecture Non-central governance COI Infrastructure Services

31. OOI CI LCO Review, Feb 2010 31 CI Interfaces and Subsystems

32. OOI CI LCO Review, Feb 2010 32 Preparing Services for Deployment Instrument Point Acquisition Point Ingest Point Application Point Storage Point Access Portal

33. OOI CI LCO Review, Feb 2010 33 Deployment Scenario

34. OOI CI LCO Review, Feb 2010 34 Deployment Plan and Strategy Deployment of capability containers in multiple execution environments by leveraging virtualization –The CEI will make it work CyberPoP (Cyberinfrastructure Point of Presence): physical deployment of one or multiple capability containers –Different types

35. OOI CI LCO Review, Feb 2010 35 Available, Elastic Execution

36. OOI CI LCO Review, Feb 2010 36 CEI Service Interactions Planner Controller (“Executive”) Provisioner Fault Monitoring and Compensation Registry Resource (Taskable) 2970-00001 OV2 CEI

37. OOI CI LCO Review, Feb 2010 37 Scenario: Planner and Controller The pattern Planner- Controller-Resource can repeat itself at multiple (nested levels) 2970-00018 OV2 CEI

38. OOI CI LCO Review, Feb 2010 38 CEI Architecture Overview OOI CI LCO Review, Feb 2010 38 EPU EPU Worker (Operational Unit) EPU Worker (Operational Unit) EPU Worker (Operational Unit) EPU Worker (Operational Unit) EPU Worker (Operational Unit) EPU Worker (Operational Unit) HA Service (OOI Application) VM (Deployable Unit) VM (Deployable Unit) VM (Deployable Unit) Application Software (Deployable Type)

39. OOI CI LCO Review, Feb 2010 39 CEI Architecture Overview OOI CI LCO Review, Feb 2010 39

40. OOI CI LCO Review, Feb 2010 40 CEI Service Interaction OOI CI LCO Review, Feb 2010 40 Planner ControllerProvisioner Monitoring Information Deployable Type Registry Context Broker IaaS AdapterIaaS Exchange VM info Proc Info Other Info

41. OOI CI LCO Review, Feb 2010 41 Elastic Processing Unit Elastic Processing Unit (EPU): High- Availability Services with policy based scaling to demand OOI CI LCO Review, Feb 2010 41 EPU Controller Register HA service EPU Worker N1 EPU Worker N1 Msg to N EPU Worker N2 EPU Worker N2 EPU Worker NX EPU Worker NX …

42. OOI CI LCO Review, Feb 2010 42 EPU: Prototype 1 EPU design and prototype: –Policy-configurable planner/scheduler –Scaling policy: size of work queue –All components integrated with Magnet/AMQP OOI CI LCO Review, Feb 2010 42

43. OOI CI LCO Review, Feb 2010 43 Federation & Governance Other areas of computing facilitated greatly by a movement to a communication oriented architecture.

44. OOI CI LCO Review, Feb 2010 44 Federation

45. OOI CI LCO Review, Feb 2010 45 Governance Architecting with RM-ODP – J. Putman B

46. OOI CI LCO Review, Feb 2010 46 Distribute Application Framework

47. OOI CI LCO Review, Feb 2010 47 Thank you

48. OOI CI LCO Review, Feb 2010 48 Motivating Governance Administering collaborations –Based on framing normative relationships among peers –Abstracting away from low-level details OOI, broadly: many stakeholders; many resources; longevity of decades Exchange spaces, narrowly: abstractions for communicating; assembly of multiple topologies for messaging; analogous to traditional enterprise integration patterns

49. OOI CI LCO Review, Feb 2010 49 Elements of a Service Engagement Enactment: doing the domain work – what the end user cares most about Administration: captured via contracts –Partnerships –Rules of encounter Identity Enforcement

50. OOI CI LCO Review, Feb 2010 50 What is Governance? Broadly, administering service engagements IT Governance: How IT resources are administered SOA Governance: How services are created, deployed, removed, … Currently, governance is manual –Low productivity –Poor scalability for fine-grained, real time governance decisions –Hidden, implicit considerations yield low confidence and poor maintainability

51. OOI CI LCO Review, Feb 2010 51 Why Governance? Stakeholders using resources to best serve individual and collective needs –Share resources in a controlled manner –Configure and reconfigure dynamically –Enable unanticipated uses for resources –Respect human organizational needs In a nutshell, stakeholders administer themselves

52. OOI CI LCO Review, Feb 2010 52 Separation of Concerns Protocol: specifying the interactions among autonomous parties Policy: specifying the decision making of each autonomous party as it participates in various protocols Behavior: specifying the implementation that realizes the interactions

53. OOI CI LCO Review, Feb 2010 53 Exchange Space Use Case OOI CI LCO Review, Feb 2010 53

54. OOI CI LCO Review, Feb 2010 54 Messaging View of Enrollment OOI CI LCO Review, Feb 2010 54

55. OOI CI LCO Review, Feb 2010 55 Community Affiliation Use Case OOI CI LCO Review, Feb 2010 55

56. OOI CI LCO Review, Feb 2010 56 Combined Scenario, Schematically OOI CI LCO Review, Feb 2010 56

57. OOI CI LCO Review, Feb 2010 57 IdM Architecture Overview

58. OOI CI LCO Review, Feb 2010 58 Conclusions

59. OOI CI LCO Review, Feb 2010 59 Summary

60. OOI CI LCO Review, Feb 2010 60 Next Steps

61. OOI CI LCO Review, Feb 2010 61 Thanks !

62. OOI CI LCO Review, Feb 2010 62 Scope Release 1: Data Distribution Network (2011) –“Ingest” data (products) from external data sources; both streaming and cataloged –Characterize data sources with their metadata attributes (format, structure, meaning) –Transform external data into internal OOI data formats, for stream-based distribution and persistence –Distribute data via streaming and DAP servers to data analysts and numerical modelers –Provide initial instrument integration, control and sensor data acquisition –Provide instrument and platform direct access –Provide service integration and execution platform

63. OOI CI LCO Review, Feb 2010 63 Scope of Future Releases Release 2: Managed Instrument Network (2012) –Fully managed instrument activation and control; ready for deployment on OOI moorings and cable infrastructure –Qualified data products based on automated data processing Release 3: On-Demand Measurement Processing (2013) –Observatory management and resource scheduling –On demand and event driven measurements –Data processing workflows Release 4: Integrated Modeling Network (2014) –Interactive analysis and synthesis –Numerical model integration Release 5: Interactive Ocean Observatory (2015) –Full control of data, processes, models, instruments –Closed loop, adaptive sensing and platform control

64. OOI CI LCO Review, Feb 2010 64 Release Schedule

65. OOI CI LCO Review, Feb 2010 65 Back-End Infrastructure

66. OOI CI LCO Review, Feb 2010 66 Exchange Points and the DIF

67. OOI CI LCO Review, Feb 2010 67 Message Brokers over DIF