1. OOI CI LCA REVIEW August 2010 Ocean Observatories Initiative Common Execution Environment Kate Keahey, Tim Freeman, Alex Clemesha, John Bresnahan, David LaBissoniere, Life Cycle Architecture Review La Jolla, CA

2. OOI CI LCA REVIEW August 2010 Subsystem Purpose Provide basic capabilities in resource provisioning in clous on commercial and national infrastructure Provide highly available services Rapidly provision resources to compensate for failure Scale to demand

3. OOI CI LCA REVIEW August 2010 R1 Use Cases IDTitleDescription UC.R1.14Use Service AnywhereMessages go to services wherever they are UC.R1.15Put Services AnywhereAllocate services where need is greatest UC.R1.16Scale the ProcessingIncrease processing quickly to meet demand UC.R1.17Replicate ServiceConfigure service once, deploy many times UC.R1.20Command A ResourceSend typical commands to specific resource UC.R1.25Assure ReliabilityComputer fails, messages resent, work resumes UC.R1.26Virtualize EverythingVirtual processes embody all system services UC.R1.28Operate SystemConfigure system and respond to requests UC.R1.30Troubleshoot SystemDiagnose issues using logs, feeds, tools

4. OOI CI LCA REVIEW August 2010 User’s View of the Architecture EPU EPU Worker (Operational Unit) EPU Worker (Operational Unit) HA Service (OOI Application) VM (Deployable Unit) Application Software (Deployable Type) EPU Worker (Operational Unit) EPU Worker (Operational Unit) EPU Worker (Operational Unit) EPU Worker (Operational Unit) VM (Deployable Unit) VM (Deployable Unit)

5. OOI CI LCA REVIEW August 2010 Architecture Overview

6. OOI CI LCA REVIEW August 2010 HA Provisioning

7. OOI CI LCA REVIEW August 2010 Bootstrapping

8. OOI CI LCA REVIEW August 2010 Implementation/Achievement Status All major components implemented: Provisioner, Controller, Planner, Sensor Aggregator Integrated with ION Needing refinement: exchange points, EPU adapters, cc-agent, use of Registry, Planner, Image building and management Tested on infrastructure ranging from Magellan to EC2 Need to put together HA-Provisioner and bootstrapping Draft user and administrator process

9. OOI CI LCA REVIEW August 2010 Technology Choices Services: Python, Twisted Messaging: ION container (AMQP) Infrastructure: Nimbus IaaS, EC2 IaaS, libcloud, Nimboss, boto Contextualiztion: Nimbus Context Broker/agent, Nimboss, Chef Solo, Fabric Special: txrabbitmq, Twotp (queue length sensor)

10. OOI CI LCA REVIEW August 2010 R1 Use Cases Demonstrated UC.R1.16: Scale the processing A load is put on the system Additional demand is recognized via different sensors Message queue length, CPU loads, disk usage System scales up to meet increased demand System scales down when demand goes away UC.R1.25: Assure reliability Failures happen Remedial actions happen No significant impact on observatory operation

11. OOI CI LCA REVIEW August 2010 The Testfest Scenarios run on EC2: CEI infrastructure deployed on two small instances: Provisioner, DTRS, EPU_Controller, Sensor Aggregator, client RabbitMQ server on a large instance with elastic IP UC Context Broker instance Fully experimental (except for queue length) Target: scale the scenarios to up to 1000 VMs (237 achieved)

12. OOI CI LCA REVIEW August 2010 Scale the Processing Test1 70 jobs Submitted over 28 minutes, 5 jobs every 2 minutes Long job duration, one job per VM Test 2 70 jobs Submitted at the same time Long job duration, one job per VM

13. OOI CI LCA REVIEW August 2010 Assure Reliability How does the system react to failure? Killing 5 VMs in 5 minute intervals Saturating the system with 10 second jobs Bounded policy: 40 VMs

14. OOI CI LCA REVIEW August 2010 Technology Challenges Most significant resolved issues: Semantic differences between IaaS providers Issues in contextualization recepies Multiple simple bugs Most significant unresolved issues: Messaging system connections close unexpectedly Currently prevents us from running at scale Inspecting queue remotely issue Non-concurrent container issue Pulling work issue

15. OOI CI LCA REVIEW August 2010 Plan for Construction Complete feature set Continue evaluation and improvement Make code ready for users Refine documentation Continue integration with other subsystems Acceptance tests and harness

16. OOI CI LCA REVIEW August 2010 Questions?