1. Ocean Observatories Initiative Common Execution Environment Kate Keahey OOI Cyberinfrastructure Life Cycle Objectives Milestone Review, Release 1 San Diego, CA February 23-25, 2010

2. OOI CI LCO Review, Feb 2010 2 Overview Use Cases Architecture Overview CEI Components in R1 CEI Elements –Elastic Provisioning Unit –Infrastructure as a Service –Deployable Type management

3. Use Case Overview OOI CI LCO Review, Feb 2010 3 - Highly Available Services - Rapidly provision resources - Scale to demand

4. Architecture Overview OOI CI LCO Review, Feb 2010 4 EPU EPU Worker (Operational Unit) EPU Worker (Operational Unit) EPU Worker (Operational Unit) EPU Worker (Operational Unit) EPU Worker (Operational Unit) EPU Worker (Operational Unit) HA Service (OOI Application) VM (Deployable Unit) VM (Deployable Unit) VM (Deployable Unit) Application Software (Deployable Type)

5. Architecture Overview OOI CI LCO Review, Feb 2010 5

6. Taskable Resources OOI CI LCO Review, Feb 2010 6 Process Operational Unit HA Service Simple or complex deployed appliance, e.g., a deployed VM instance, a virtual cluster, platforms for workflow execution e.g., an executing program or a long- running service e.g., a core or application service Deployable Type Deployable Unit e.g., a description of an environment in rBuilder e.g., a VM image Taskable Resources Start/Stop Monitorable Controllable

7. Architecture Overview OOI CI LCO Review, Feb 2010 7

8. Architecture Overview OOI CI LCO Review, Feb 2010 8

9. 9 CEI Service Interactions Planner Controller (“Executive”) Provisioner Fault Monitoring and Compensation Registry Resource (Taskable) 2970-00001 OV2 CEI

10. CEI Service Interaction OOI CI LCO Review, Feb 2010 10 Planner ControllerProvisioner Monitoring Information Deployable Type Registry Context Broker IaaS AdapterIaaS Exchange VM info Proc Info Other Info

11. Elastic Processing Unit Elastic Processing Unit (EPU): High- Availability Services with policy based scaling to demand OOI CI LCO Review, Feb 2010 11 EPU Controller Register HA service EPU Worker N1 EPU Worker N1 Msg to N EPU Worker N2 EPU Worker N2 EPU Worker NX EPU Worker NX …

12. Elastic Processing Unit OOI CI LCO Review, Feb 2010 12 Planner ControllerProvisioner Monitoring Information Deployable Type Registry Context Broker IaaS AdapterIaaS Exchange VM info Proc Info Other Info Will this whole thing work (even in a small way)?

13. EPU: Prototype 1 EPU design and prototype: –Policy-configurable planner/scheduler –Scaling policy: size of work queue –All components integrated with Magnet/AMQP OOI CI LCO Review, Feb 2010 13

14. EPU: Prototype 2 Updates technology choices –Uses Redis to distribute work to VM instances and as Information Aggregator New policy –load Demonstration later OOI CI LCO Review, Feb 2010 14

15. EPU: Technology Choices IaaS Adapter: still evaluating –Deltacloud, jcloud, libcloud, dasein, fog –Standards: OCCI-WG Nimbus Context Broker Redis for Information storage OOI CI LCO Review, Feb 2010 15

16. EPU: Elaboration Iterations Atomic HA for EPU controller Detailed design on interaction with COI Scalability, scalability, scalability Finalize technology choices Prototype end-to-end EPU Stress testing and harness Investigate storage/compute/network Refine implementation OOI CI LCO Review, Feb 2010 16

17. Infrastructure-as-a-Service OOI CI LCO Review, Feb 2010 17 Planner ControllerProvisioner Monitoring Information Deployable Type Registry Context Broker IaaS AdapterIaaS Exchange VM info Proc Info Other Info Will it work with COI? Where are we going to get IaaS resources? Can we adapt existing resources?

18. Infrastructure-as-a-Service Targeted deployment platforms –Commercial (AWS, rackspace) –Scientific (FutureGrid, Magellan, …) IaaS for science –The Nimbus Workspace Service –Evaluate Ganeti/SGE/Condor Nimbus –Make implementation technology-independent –Make implementation WSRF-independent –Provide user management functions OOI CI LCO Review, Feb 2010 18

19. Deployable Types OOI CI LCO Review, Feb 2010 19 Planner ControllerProvisioner Monitoring Information Deployable Type Registry Context Broker IaaS AdapterIaaS Exchange VM info Proc Info Other Info complexity? Will it work with COI? scalability?

20. Deployable Types Complexity –Document requirements and use cases Web App, Hadoop, Job Queue, NFS, Torque –Design overall approach –Preliminary roundup and evaluation of relevant technologies Puppet, Chef, CohesiveFT, rBuilder Working choices: rBuilder and Chef –Prototype integration of Chef with the Nimbus Context Broker OOI CI LCO Review, Feb 2010 20

21. Deployable Types Protocol-independence –Make the Context Broker WSRF-independent –REST interfaces Scalability and Sustainability –Prototype scalability harness –Scalability tests Contextualizing up to 500 nodes Challenges in harness design and efficiency OOI CI LCO Review, Feb 2010 21

22. DT: Elaboration Iterations Finalize technology choices for management of Deployable Types/Units Deployable Type Registry More comprehensive harness and more contextualization scalability tests Understanding performance OOI CI LCO Review, Feb 2010 22

23. OOI CI LCO Review, Feb 2010 23 Thanks !