Cloud Computing and Virtualization with Globus Oakland, May 2008 Kate Keahey Tim Freeman University of Chicago Argonne National Laboratory

05/14/08 Virtual Workspaces: http//workspace.globus.org Cloud Computing Tutorial Hands-on l To participate in the hands-on part of the tutorial, send your PKI X509 subject line to l The first 10 requests will be given access to the nimbus cloud l Hurry!

05/14/08 Virtual Workspaces: http//workspace.globus.org Overview l Motivation l The Workspace Ecosystem: Abstractions and Background l The Workspace Deployment Tools l Managing Resources with Virtual Workspaces l Appliance management and contextualization l Virtual Cluster Management with Workspace Tools l Application Example: the STAR experiment l Cloud Computing l Run on the cloud: hands-on tutorial

Motivation

05/14/08 Virtual Workspaces: http//workspace.globus.org 1)Configuration: finding environment tailored to my application 2) Leasing: negotiating a resource allocation tailored to my needs ? A Good Workspace is Hard to Find

05/14/08 Virtual Workspaces: http//workspace.globus.org Consumers Perspective: Quality of Life l Real life applications are complex u STAR example: Developed over more than 10 years, by more than 100 scientists, comprises ~2 M lines of C++ and Fortran code l … and require complex, customized environments u Rely heavily on the right combination of compiler versions and available libraries l Environment validation u To ensure reproducibility and result uniformity across environments

05/14/08 Virtual Workspaces: http//workspace.globus.org Consumers Perspective: Quality of Service l There is life beyond submitting batch jobs u Resource leases rather than job submission l Control of resources u Explicit SLA: different sites offer different quality of service l Satisfying peak demand u Experiment season, paper deadlines, etc.

05/14/08 Virtual Workspaces: http//workspace.globus.org Providers Perspective l Providing resources is easy, providing environments is hard u User comment: I have 512 nodes I cannot use ;-) l Fine-tuning environments for different communities is expensive u Evaluating, installing and maintaining software packages etc. u Reconciling conflicts u Coordinating update schedules for different communities is a nightmare l It may be hard to justify configuring/dedicating resources if they are only needed 1% of the time - - even if the 1% is very important for one of your users

The Workspace Ecosystem: Abstractions and Background

05/14/08 Virtual Workspaces: http//workspace.globus.org Virtual Workspaces l A dynamically provisioned environment u A complete environment: a complete (software) environment as required by our community or applications provisioned on demand. u Resource allocation: provision the resources the workspace needs (CPUs, memory, disk, bandwidth, availability), allowing for dynamic renegotiation to reflect changing requirements and conditions. u Deployment point of view l Appliances/virtual appliances u A complete environment that can be packaged in various formats u Packaging point of view

05/14/08 Virtual Workspaces: http//workspace.globus.org Workspace Implementations l Traditional tools u Base environment (discovery) u Automated configuration u Typically long deployment time u Isolation l Performance isolation u Runtime environment l Virtual machines u Complete environment l Contextualization u Short deployment time u Very good isolation u Runtime performance impact Paper: Virtual Workspaces: Achieving Quality of Service and Quality of Life in the Grid

05/14/08 Virtual Workspaces: http//workspace.globus.org The Virtues of Virtualization Hardware Virtual Machine Monitor (VMM) / Hypervisor Guest OS (Linux) Guest OS (NetBSD) Guest OS (Windows) VM App Xen VMWare UML KVM etc. Parallels l Bring your environment with you l Excellent enforcement and isolation l Fast to deploy, enables short-term leasing l Have a performance impact but it is acceptable for most modern hypervisors l Suspend/resume, migration

05/14/08 Virtual Workspaces: http//workspace.globus.org Creating a Virtual Cluster that Works Resource VM Obtain a lease on a raw resource Deploy VMs onto the resource Put the VMs in context VM Deploy virtual machines Contextualization layer Create a functioning virtual ensemble

05/14/08 Virtual Workspaces: http//workspace.globus.org The Workspace Ecosystem Resource Providers: Grid providers: TeraGrid, OSG, etc. Commercial providers: EC2, Sun, etc. Appliance Providers: off-the-shelf environment bundles certified/endorsed for safety leverage appliance software commercial and open marketplaces Appliance Deployment: Mapping environments onto leased computing resources Coordinating creation of virtual resources A mix of open source software and proprietary tools communicating via common protocols

05/14/08 Virtual Workspaces: http//workspace.globus.org Roles and Responsibilities l Division of labor u Resource providers provide resources u Virtual organizations provide appliances u Middleware that maps appliances onto resources l Appliance management software u Appliance creation, maintenance, validation, etc. u Not an appliance provider l Shifting the work around u Into the hands of the parties most motivated and qualified to do it

Workspace Deployment Tools

05/14/08 Virtual Workspaces: http//workspace.globus.org Virtual Workspaces: Vital Stats l Virtual Workspace software allows an authorized client to dynamically deploy and manage workspaces u Virtual Workspace Service (VWS), workspace control, Context Broker l Currently implements workspaces as Xen VMs u KVM coming this summer l Also, contextualization layer l Globus incubator project l Started ~2003, first release in September 2005 l Current release (March 08) l Download it from: u

05/14/08 Virtual Workspaces: http//workspace.globus.org Using Workspaces (Deployment) Pool node Pool node Pool node Pool node Pool node Pool node Pool node Pool node Pool node Pool node Pool node Pool node Workspace -Workspace metadata -Pointer to the image -Logistics information -Deployment request -CPU, memory, node count, etc. VWS Service

05/14/08 Virtual Workspaces: http//workspace.globus.org Using Workspaces (Interaction) Pool node Trusted Computing Base (TCB) Pool node Pool node Pool node Pool node Pool node Pool node Pool node Pool node Pool node Pool node Pool node The workspace service publishes information on each workspace as standard WSRF Resource Properties. Users can query those properties to find out information about their workspace (e.g. what IP the workspace was bound to) Users can interact directly with their workspaces the same way the would with a physical machine. VWS Service

05/14/08 Virtual Workspaces: http//workspace.globus.org Workspace Service (what sits inside) Pool node Trusted Computing Base (TCB) Pool node Pool node Pool node Pool node Pool node Pool node Pool node Pool node Pool node Pool node Pool node VWS Service Workspace WSRF front-end that allows clients to deploy and manage virtual workspaces Resource manager for a pool of physical nodes Deploys and manages Workspaces on the nodes Contextualization creates a common context for a virtual cluster Each node must have a VMM (Xen) installed, as well as the workspace control program that manages individual nodes Workspace back-end:

05/14/08 Virtual Workspaces: http//workspace.globus.org Workspace Service Components l GT4 WSRF front-end u Leverages GT core and services, notifications, security, etc. u Roughly follows the OGF WS-Agreement provisioning model l Lease descriptions l Publishes available lease terms l Workspace Service back-end u Works with multiple Resource Managers u Workspace Control for on the node functions l Contextualization u Put the virtual appliance in its deployment context

Managing Resources with Virtual Workspaces

05/14/08 Virtual Workspaces: http//workspace.globus.org Workspace Back-Ends l Default resource manager (basic slot fitting) u datacenter technology equivalent u Used for OSG Edge Services project l Challenge: finding Xen-enabled resources u Amazon Elastic Compute Cloud (EC2) l Software similar to Workspace Service (no virtual clusters, contextualization, fine-grain allocations, etc.) u Solution: develop a back-end to EC2 u Grid credential admission -> EC2 charging model u Address contextualization needs l Challenge: integrating VMs into current provisioning models u Solution: gliding in VMs with the Workspace Pilot

05/14/08 Virtual Workspaces: http//workspace.globus.org The Workspace Pilot l Challenge: how can I provide a cloud using virtualization without disrupting the current operation of my cluster? l Flying Low: the Workspace Pilot u Integrates with popular LRMs (such as PBS) u Implements best effort leases u Glidein approach: submits a pilot program that claims a resource slot u Includes administrator tools l Deployment u U of Victoria (Atlas), Ian Gable and collaborators u Adapting for the use of the Atlas CERN, Omer Khalid u TeraPort (small partition)

05/14/08 Virtual Workspaces: http//workspace.globus.org Workspace Pilot in Action VWS LRM/PBS Xen dom0 VM Level 1: provision raw resources Level 2: provision VMs

05/14/08 Virtual Workspaces: http//workspace.globus.org The Pilot Program l Uses Xen balloon driver to reduce/restore domain0 memory so that guest domains (VMs) can be deployed l Secure VM deployment u The pilot requires sudo privilege and thus can be used only with site administrators approval u The workspace service provides fine-grained authorization for all requests l Signal handling u SIGTERM: pilot exceeded its allotted time l Notifies VWS, allows it to clean up l After a configurable time period takes things into its hands. l Default policy: one VM per physical node l Available for download u Workspace Release 1.3.1: u

05/14/08 Virtual Workspaces: http//workspace.globus.org Workspace Control l VM control u Starting, stopping, pausing, etc. l Integrating a VM into the network u Assigning MAC addresses and IP addresses u DHCP delivery tool u Building up a trusted (non-spoofable) networking layer l VM image propagation l Image management and reconstruction u creating blank partitions, sharing partitions l Contextualization information management l Talks to the workspace service via ssh l Can be used as a standalone component

05/14/08 Virtual Workspaces: http//workspace.globus.org Workspace Back-Ends l Default resource manager (basic slot fitting) u datacenter technology equivalent u Used for OSG Edge Services project l Challenge: finding Xen-enabled resources u Amazon Elastic Compute Cloud (EC2) l Software similar to Workspace Service (no virtual clusters, contextualization, fine- grain allocations, etc.) u Solution: develop a back-end to EC2 u Grid credential admission -> EC2 charging model u Address contextualization needs l Challenge: integrating VMs into current provisioning models u Solution: gliding in VMs with the Workspace Pilot Long-term solutions l Leasing model with explicit terms u Semantically rich leases: advance reservations, urgent leases, renegotiable leases, etc. u Cost-effective lease semantics

Appliance Management and Contextualization

05/14/08 Virtual Workspaces: http//workspace.globus.org Where Do Appliances Come From? Marketplaces (VMWare, EC2, Workspace …) appliance description Appliance Provider (a user, a VO, a Grid…) Good… but: maintenance? ease of use? formats?

05/14/08 Virtual Workspaces: http//workspace.globus.org Where Do Appliances Come From? Marketplaces (VMWare, EC2, Workspace …) appliance description Appliance Provider (a user, a VO, a Grid…) Appliance Management Software (OSFarm, rPath,…)) Xen VMware CDROM Better

05/14/08 Virtual Workspaces: http//workspace.globus.org Deploying Appliances l Appliances need to be portable u So that they can be reused in many contexts l Making the appliance context- aware: u Other appliances u Site-specific information (e.g. a DNS server) u User/group/VO/Grid-specific information (e.g. public keys, host certs, gridmapfiles, etc.) l Security issues u Who do I trust to provide legitimate context information? u How do I make sure that appliances adhere to my site policies? VM site Virtual Organization

05/14/08 Virtual Workspaces: http//workspace.globus.org Where Do Appliances Come From? Marketplaces (VMWare, EC2, Workspace …) appliance description appliance assertions appliance contextualization Appliance Provider (a user, a VO, a Grid…) Appliance Management Software (OSFarm, rPath, CohesiveFT…)) Xen VMware CDROM

Creating Virtual Clusters with Workspace Tools

05/14/08 Virtual Workspaces: http//workspace.globus.org Make Me a Working Cluster l You got some VMs and youve deployed them… Now What? u What network are they connected to? Do they actually represent something useful? (like a ready-to-use OSG cluster?) Do the VMs know about each other? Can they share some disk? How do they integrate into the site storage/account system? Do they have host certificates? And a gridmapfile? And all the other things that will integrate them into my VO? l Challenge: what is a virtual cluster? u A more complex virtual machine l Networking, shared storage, etc. u Available at the same time and sharing a common context u Example: an OSG cluster l Solutions u Ensemble management u Exporting and sharing common context u Sophisticated networking configurations. Paper: Virtual Clusters for Grid Communities, CCGrid 2006

05/14/08 Virtual Workspaces: http//workspace.globus.org Contextualization l Challenge: Putting a VM in the deployment context of the Grid, site, and other VMs u Assigning and sharing IP addresses, name resolution, application-level configuration, etc. l Solution: Management of Common Context Paper: A Scalable Approach To Deploying And Managing Appliances, TeraGrid conference 2007 u Configuration-dependent l provides&requires u Common understanding between the image vendor and deployer u Mechanisms for securely delivering the required information to images across different implementations contextualization agent Common Context IP hostname pk

05/14/08 Virtual Workspaces: http//workspace.globus.org Appliance context agent Contextualizing Appliances Appliance context template application-specific context agents appliance content disk image Appliance Provider Appliance Deployer Context Broker Resource Provider generic context appliance context

Application Example: Virtualization with the STAR experiment

05/14/08 Virtual Workspaces: http//workspace.globus.org Virtual Workspaces for STAR l STAR image configuration u A virtual cluster composed of one OSG headnode and multiple STAR worker nodes l Using the workspace service over EC2 to provision resources u Allocations of up to 100 nodes u Dynamically contextualized for out-of-the-box cluster

05/14/08 Virtual Workspaces: http//workspace.globus.org l Deployment stages: u Create an ensemble defining the virtual cluster u Deploy the virtual machines u Contextualize to provide an out-of the-box cluster l Contextualization: u Cluster applications: NFS & PBS u Grid information: gridmapfile and host certificates l Runs u Using VWS on the nimbus cloud for small node allocations (VWS + default + Context Broker) u Using VWS with EC2 backend for allocations of ~100 nodes (VWS + EC2 backend + Context Broker) Virtual Workspaces for STAR

05/14/08 Virtual Workspaces: http//workspace.globus.org Running jobs : 300 PDSF Fermi VWS/EC2BNL Running jobs : 230 Running jobs : 150 Running jobs : 50 Running jobs : 150 Running jobs : 300Running jobs : 282Running jobs : 243Running jobs : 221Running jobs : 195Running jobs : 140Running jobs : 76Running jobs : 0 Running jobs : 200 Running jobs : 50 Running jobs : 150Running jobs : 142Running jobs : 124Running jobs : 109Running jobs : 94Running jobs : 73Running jobs : 42 Running jobs : 195Running jobs : 183Running jobs : 152Running jobs : 136Running jobs : 96Running jobs : 54Running jobs : 37Running jobs : 0 Running jobs : 42Running jobs : 39Running jobs : 34Running jobs : 27Running jobs : 21Running jobs : 15Running jobs : 9Running jobs : 0 Job Completion : File Recovery : WSU with thanks to Jerome Lauret and Doug Olson of the STAR project

05/14/08 Virtual Workspaces: http//workspace.globus.org Nersc PDSF EC2 (via Workspace Service) WSU Accelerated display of a workflow job state Y = job number, X = job state with thanks to Jerome Lauret and Doug Olson of the STAR project

Cloud Computing

05/14/08 Virtual Workspaces: http//workspace.globus.org The Workspace Cloud Client l We took the workspace client and made it easy to use u Narrowing down the functionality u Wrapper on top of the workspace client l Allows scientists to lease VMs roughly following Amazons EC2 model (simplified) u PKI X509 credentials and quotas instead of payment l The goal is to restore/evolve this functionality as user requests come in u Saving VMs, network configurations u In the future: richer leases, etc. l Cloudkit coming out in next release, due soon

05/14/08 Virtual Workspaces: http//workspace.globus.org University of Chicago l Objectives u Make it easy for scientific community to experiment with this mode of resource provisioning u Learn about the requirements of scientific projects and evolve the infrastructure u Features, SLAs, security and sharing concerns, etc. l Vital Stats u Deployed on 16 nodes of TeraPort UC u Powered by the workspace set of tools u Image management handled via gridFTP u Made available mid-March 08 u l To obtain access mail u Available to scientific, educational projects, open source testing, etc.

05/14/08 Virtual Workspaces: http//workspace.globus.org Science Clouds l A group of clouds making resources available on the nimbus model u Nimbus, (Mauricio Tsugawa), FZK in Germany (almost done, Lizhe Wang), others expressed interest u EC2 l Some differences in setup, policies u UFL requires private networks (using OpenVPN) l Currently youd use the same credential for the cloud and for the virtual private network u EC2 requires payment l Cloud federation u Moving an app from a hardware platform to a cloud is relatively hard l Need image, learn new paradigm, etc. u Moving between clouds is relatively easy u … if you have rough consensus on interfaces, image formats, etc.

05/14/08 Virtual Workspaces: http//workspace.globus.org Who runs on the clouds and what do they do?

05/14/08 Virtual Workspaces: http//workspace.globus.org Related Projects l Portal development (Josh Boverhof, LBNL) l Workspace KVM backend (Michael Fenn, Clemson University) l Integration with the Nebula project (University of Madrid)

Lets get on the cloud!

05/14/08 Virtual Workspaces: http//workspace.globus.org Parting Thoughts

05/14/08 Virtual Workspaces: http//workspace.globus.org Parting Thoughts l Come and run on science clouds l Not just cloud computing u A bunch of technologies have to come together to make cloud computing widespread l The way we do computing is changing u Today we build horseless carriages u Tomorrow we might do things differently

05/14/08 Virtual Workspaces: http//workspace.globus.org Credits l Workspace team: u Kate&Tim l Guest appearances u Ian Foster, Frank Siebenlist l With thanks to many collaborators: u Jerome Lauret (STAR, BNL), Doug Olson (STAR, LBNL), Marty Wesley (rPath), Stu Gott (rPath), Ken Van Dine (rPath), Predrag Buncic (Alice, CERN), Haavard Bjerke (CERN), Rick Bradshaw (Bcfg2, ANL), Narayan Desai (Bcfg2, ANL), Duncan Penfold-Brown (Atlas,uvic), Ian Gable (Atlas, uvic), David Grundy (Atlas, uvic), Ti Legget (University of Chicago), Greg Cross (University of Chicago), Lizhe Wang (FZK), Marcel Kunze (FZK), Mauricio Tsugawa (UFL), Jose Fortes (UFL), Renato Figueiredo (UFL), Omer Khalid (CERN), Artem Harutyunyan (CERN), Mike Fenn (U of Clemson), Sebastien Goasguen (U of Clemson), Josh Boverhof (LBNL), Leve Hajdu (STAR, BNL), Lidia Didenko (STAR, BNL), David Bartle (Atlas, uvic), Lee Liming (ANL), Frank Wuerthwein (OSG, SDSC), Abhishek Rana (OSG, SDSC), Jeff Chase (Duke), and many others.

05/14/08 Virtual Workspaces: http//workspace.globus.org Sponsors l NSF SDCI Missing Links l NSF CSR Virtual Playgrounds l TeraGrid l DOE SciDAC CEDPS