Ocean Observatories Initiative Common Execution Environment Kate Keahey OOI Cyberinfrastructure Life Cycle Objectives Milestone Review, Release 1 San Diego, CA February 23-25, 2010

OOI CI LCO Review, Feb Overview Use Cases Architecture Overview CEI Components in R1 CEI Elements –Elastic Provisioning Unit –Infrastructure as a Service –Deployable Type management

Use Case Overview OOI CI LCO Review, Feb Highly Available Services - Rapidly provision resources - Scale to demand

Architecture Overview OOI CI LCO Review, Feb EPU EPU Worker (Operational Unit) EPU Worker (Operational Unit) EPU Worker (Operational Unit) EPU Worker (Operational Unit) EPU Worker (Operational Unit) EPU Worker (Operational Unit) HA Service (OOI Application) VM (Deployable Unit) VM (Deployable Unit) VM (Deployable Unit) Application Software (Deployable Type)

Architecture Overview OOI CI LCO Review, Feb

Taskable Resources OOI CI LCO Review, Feb Process Operational Unit HA Service Simple or complex deployed appliance, e.g., a deployed VM instance, a virtual cluster, platforms for workflow execution e.g., an executing program or a long- running service e.g., a core or application service Deployable Type Deployable Unit e.g., a description of an environment in rBuilder e.g., a VM image Taskable Resources Start/Stop Monitorable Controllable

Architecture Overview OOI CI LCO Review, Feb

Architecture Overview OOI CI LCO Review, Feb

9 CEI Service Interactions Planner Controller (“Executive”) Provisioner Fault Monitoring and Compensation Registry Resource (Taskable) OV2 CEI

CEI Service Interaction OOI CI LCO Review, Feb Planner ControllerProvisioner Monitoring Information Deployable Type Registry Context Broker IaaS AdapterIaaS Exchange VM info Proc Info Other Info

Elastic Processing Unit Elastic Processing Unit (EPU): High- Availability Services with policy based scaling to demand OOI CI LCO Review, Feb EPU Controller Register HA service EPU Worker N1 EPU Worker N1 Msg to N EPU Worker N2 EPU Worker N2 EPU Worker NX EPU Worker NX …

Elastic Processing Unit OOI CI LCO Review, Feb Planner ControllerProvisioner Monitoring Information Deployable Type Registry Context Broker IaaS AdapterIaaS Exchange VM info Proc Info Other Info Will this whole thing work (even in a small way)?

EPU: Prototype 1 EPU design and prototype: –Policy-configurable planner/scheduler –Scaling policy: size of work queue –All components integrated with Magnet/AMQP OOI CI LCO Review, Feb

EPU: Prototype 2 Updates technology choices –Uses Redis to distribute work to VM instances and as Information Aggregator New policy –load Demonstration later OOI CI LCO Review, Feb

EPU: Technology Choices IaaS Adapter: still evaluating –Deltacloud, jcloud, libcloud, dasein, fog –Standards: OCCI-WG Nimbus Context Broker Redis for Information storage OOI CI LCO Review, Feb

EPU: Elaboration Iterations Atomic HA for EPU controller Detailed design on interaction with COI Scalability, scalability, scalability Finalize technology choices Prototype end-to-end EPU Stress testing and harness Investigate storage/compute/network Refine implementation OOI CI LCO Review, Feb

Infrastructure-as-a-Service OOI CI LCO Review, Feb Planner ControllerProvisioner Monitoring Information Deployable Type Registry Context Broker IaaS AdapterIaaS Exchange VM info Proc Info Other Info Will it work with COI? Where are we going to get IaaS resources? Can we adapt existing resources?

Infrastructure-as-a-Service Targeted deployment platforms –Commercial (AWS, rackspace) –Scientific (FutureGrid, Magellan, …) IaaS for science –The Nimbus Workspace Service –Evaluate Ganeti/SGE/Condor Nimbus –Make implementation technology-independent –Make implementation WSRF-independent –Provide user management functions OOI CI LCO Review, Feb

Deployable Types OOI CI LCO Review, Feb Planner ControllerProvisioner Monitoring Information Deployable Type Registry Context Broker IaaS AdapterIaaS Exchange VM info Proc Info Other Info complexity? Will it work with COI? scalability?

Deployable Types Complexity –Document requirements and use cases Web App, Hadoop, Job Queue, NFS, Torque –Design overall approach –Preliminary roundup and evaluation of relevant technologies Puppet, Chef, CohesiveFT, rBuilder Working choices: rBuilder and Chef –Prototype integration of Chef with the Nimbus Context Broker OOI CI LCO Review, Feb

Deployable Types Protocol-independence –Make the Context Broker WSRF-independent –REST interfaces Scalability and Sustainability –Prototype scalability harness –Scalability tests Contextualizing up to 500 nodes Challenges in harness design and efficiency OOI CI LCO Review, Feb

DT: Elaboration Iterations Finalize technology choices for management of Deployable Types/Units Deployable Type Registry More comprehensive harness and more contextualization scalability tests Understanding performance OOI CI LCO Review, Feb

OOI CI LCO Review, Feb Thanks !