Globus Virtual Workspaces An Update SC 2007, Reno, NV Kate Keahey Argonne National Laboratory University of Chicago

Motivation and Background

SC07, Reno, NVVirtual Workspaces: Why Virtual Workspaces? l Quality of Service u We get: batch-style provisioning l One size fits all l Side-effect of job scheduling u We need: advance reservations, urgent computing, periodic, best-effort, and others l Separation of job scheduling and resource management l E.g. workflow-based apps and batch apps have different needs l Quality of Life u We have: a 100 nodes we cannot use u Complex applications l Hard to install l Require validation u Separation of environment preparation and resources leasing

SC07, Reno, NVVirtual Workspaces: What are Virtual Workspaces? l A dynamically provisioned environment u Environment definition: we get exactly the (software) environment we need on demand. u Resource allocation: Provision the resources the workspace needs (CPUs, memory, disk, bandwidth, availability), allowing for dynamic renegotiation to reflect changing requirements and conditions. l Implementation u Traditional means: publishing, automated configuration, coarse-grained enforcement u Virtual Machines: encapsulated configuration and fine-grained enforcement Paper: Virtual Workspaces: Achieving Quality of Service and Quality of Life in the Grid

SC07, Reno, NVVirtual Workspaces: Virtual Machines Hardware Virtual Machine Monitor (VMM) / Hypervisor Guest OS (Linux) Guest OS (NetBSD) Guest OS (Windows) VM App Xen VMWare UML KVM etc. Parallels l Bring your environment with you l Fast to deploy, enables short-term leasing l Excellent enforcement, performance isolation l Very good isolation

Globus Virtual Workspaces: How Do They Work?

SC07, Reno, NVVirtual Workspaces: Virtual Workspaces: Vital Stats l The GT4 Virtual Workspace Service (VWS) allows an authorized client to deploy and manage workspaces on-demand. u GT4 WSRF front-end (one per site) l Leverages GT core and services, notifications, security, etc. u Follows WS-Agreement provisioning model u Currently implements workspaces as Xen VMs l Other implementations could also be used u Implements multiple deployment modes l Best-effort, leasing, etc. u Current release 1.3 (November 07) u Globus incubator project u More information at: More information at:

SC07, Reno, NVVirtual Workspaces: Deploying Workspaces Remotely Pool node Pool node Pool node Pool node Pool node Pool node Pool node Pool node Pool node Pool node Pool node Pool node Workspace -Workspace metadata -Pointer to the image -Logistics information -Deployment request -CPU, memory, node count, etc. VWS Service

SC07, Reno, NVVirtual Workspaces: Interacting with Workspaces Pool node Trusted Computing Base (TCB) Pool node Pool node Pool node Pool node Pool node Pool node Pool node Pool node Pool node Pool node Pool node The workspace service publishes information on each workspace as standard WSRF Resource Properties. Users can query those properties to find out information about their workspace (e.g. what IP the workspace was bound to) Users can interact directly with their workspaces the same way the would with a physical machine. VWS Service

SC07, Reno, NVVirtual Workspaces: Workspace Service Components Pool node Trusted Computing Base (TCB) Pool node Pool node Pool node Pool node Pool node Pool node Pool node Pool node Pool node Pool node Pool node VWS Service Workspace WSRF front-end that allows clients to deploy and manage virtual workspaces Resource manager for a pool of physical nodes Deploys and manages Workspaces on the nodes Contextualization creates a common context for a virtual cluster Each node must have a VMM (Xen) installed, as well as the workspace control program that manages individual nodes Workspace back-end:

SC07, Reno, NVVirtual Workspaces: Workspace Service Components l GT4 WSRF front-end u Leverages GT core and services, notifications, security, etc. u Follows the OGF WS-Agreement provisioning model l Publishes available lease terms u Provides lease descriptions l Workspace Service back-end u Currently focused on Xen u Works with multiple Resource Managers u Workspace Control l Contextualization u Put the virtual appliance in its deployment context

Managing Resources with Virtual Workspaces

SC07, Reno, NVVirtual Workspaces: Workspace Back-Ends l Default resource manager (basic slot fitting) u Commercial datacenter technology would also fit l Challenge: finding Xen-enabled resources u Amazon Elastic Compute Cloud (EC2) u Selling cycles as Xen VMs u Software similar to Workspace Service l No virtual clusters, contextualization, fine-grain allocations, etc. u Solution: develop a back-end to EC2 u Grid credential admission -> EC2 charging model

SC07, Reno, NVVirtual Workspaces: Virtual Workspaces for STAR l STAR image configuration u A virtual cluster composed of an OSG headnode and STAR worker nodes l Using the workspace service over EC2 to provision resources u Allocations of up to 100 nodes u Dynamically contextualized for out-of-the-box cluster

SC07, Reno, NVVirtual Workspaces: Running jobs : 300 PDSF Fermi VWS/EC2BNL Running jobs : 230 Running jobs : 150 Running jobs : 50 Running jobs : 150 Running jobs : 300Running jobs : 282Running jobs : 243Running jobs : 221Running jobs : 195Running jobs : 140Running jobs : 76Running jobs : 0 Running jobs : 200 Running jobs : 50 Running jobs : 150Running jobs : 142Running jobs : 124Running jobs : 109Running jobs : 94Running jobs : 73Running jobs : 42 Running jobs : 195Running jobs : 183Running jobs : 152Running jobs : 136Running jobs : 96Running jobs : 54Running jobs : 37Running jobs : 0 Running jobs : 42Running jobs : 39Running jobs : 34Running jobs : 27Running jobs : 21Running jobs : 15Running jobs : 9Running jobs : 0 Job Completion : File Recovery : WSU with thanks to Jerome Lauret and Doug Olson of the STAR project

SC07, Reno, NVVirtual Workspaces: Nersc PDSF EC2 (via Workspace Service) WSU Accelerated display of a workflow job state Y = job number, X = job state with thanks to Jerome Lauret and Doug Olson of the STAR project

SC07, Reno, NVVirtual Workspaces: Workspace Back-Ends l Default resource manager (basic slot fitting) u Commercial datacenter technology would also fit l Challenge: finding Xen-enabled resources u Amazon Elastic Compute Cloud (EC2) u Selling cycles as Xen VMs u Software similar to Workspace Service l No virtual clusters, contextualization, fine-grain allocations, etc. u Grid credential admission -> EC2 charging model u Solution: develop a back-end to EC2 l Challenge: integrating VMs into current provisioning models u Solution: gliding in VMs with the Workspace Pilot

SC07, Reno, NVVirtual Workspaces: Providing Resources: The Workspace Pilot l Challenge: find the simplest way to integrate VMs into current provisioning models l Glide-ins (Condor): poor mans resource leasing u Best-effort semantics: submit a job pilot that claims resources but does not run a job l The Workspace Pilot u Resources booted to dom0 u Pilot adjusts memory u VWS leases slots to VMs u Functional closure: kill-all facility, etc.

SC07, Reno, NVVirtual Workspaces: Workspace Control l VM control u Starting, stopping etc. u To be replaced by Xen API l Integrating into the network u Assigning MAC addresses and IP addresses u DHCP Delivery tool u Building up a trusted networking layer l VM image propagation l Image management and reconstruction u creating blank partitions l Talks to the workspace service via ssh u To be replaced

SC07, Reno, NVVirtual Workspaces: Workspace Back-Ends l Default resource manager (basic slot fitting) u Commercial datacenter technology would also fit l Challenge: finding Xen-enabled resources u Amazon Elastic Compute Cloud (EC2) u Selling cycles as Xen VMs u Software similar to Workspace Service l No virtual clusters, contextualization, fine-grain allocations, etc. u Grid credential admission -> EC2 charging model u Solution: develop a back-end to EC2 l Challenge: integrating VMs into current provisioning models u Solution: gliding in VMs with the Workspace Pilot l Long-term solutions u Interleaving soft and hard leases u Providing better articulated leasing models u Developed in the context of existing schedulers

So -- youve deployed* some VMs… Now What? *Do they have public IP addresses? Do they actually represent something useful?(BTW, I need an OSG cluster) Can the VMs find out about each other? Can they share storage? How do they integrate into the site storage/account system? Do they have host certificates? And gridmapfile? And all the other things that will integrate them into my VO?

SC07, Reno, NVVirtual Workspaces: l Challenge: what is a virtual cluster? u A more complex virtual machine l Networking, shared storage, etc. that will be portable across sites and implementations u Available at the same time and sharing a common context u Example: l A set of worker nodes with some edge services in front and NFS-based shared storage l Solution: management of ensembles and sharing u Configurable cluster deployment l A set of worker nodes l A few Edge Services enabling access to those nodes u Exporting and sharing a common context l Configuring and joining context u Networking l Edge Services have public IPs l Worker nodes are on a private network shared with the Edge Services Paper: Virtual Clusters for Grid Communities, CCGrid 2006 Virtual Clusters

SC07, Reno, NVVirtual Workspaces: Contextualization l Challenge: Putting a VM in the deployment context of the Grid, site, and other VMs u Assigning and sharing IP addresses, name resolution, application- level configuration, etc. l Solution: Management of Common Context Paper: A Scalable Approach To Deploying And Managing Appliances, TeraGrid conference 2007 u Configuration-dependent l provides&requires u Common understanding between the image vendor and deployer u Mechanisms for securely delivering the required information to images across different implementations contextualization agent Common Context IP hostname pk

Where do VM images come from?

SC07, Reno, NVVirtual Workspaces: Appliance Management l Short term solution: Marketplaces u The Workspace Marketplace u u Providing described images for scientific community Appliance providers and marketplaces l Long-term solution: Appliance Providers u Automated image production, attestation and signing l Automated management u Collaboration with configuration management communities and projects l rPath company: the rBuilder project (DOE SBiR) l Bcfg2, adopted on many ANL resources l CERN OpenLab, serving the scientific community Appliance providers

SC07, Reno, NVVirtual Workspaces: Workspace Ecosystem Resource Providers: Local clusters, Grid resource providers (TeraGrid, OSG) Commercial providers: EC2, Sun, slicehost, Provisioning a resource, not a platform Appliance Providers: OSFarm, rPath, CohesiveFT, bcfg2, etc. marketplaces of all kinds Virtual Organizations: configuration, attestation, maintenance Middleware: appliances --> resources manage appliance deployment Combining networks and storage VWSEC2In-Vigo

SC07, Reno, NVVirtual Workspaces: Parting Thoughts l VMs are the raw materials from which a working system can be built u But we still have to build it! u Technical challenges: taking one step at a time u Social/procedural challenges l Division of labor u Resource providers u Appliance providers u Can we build trust between these two groups? l If you think we can help you out, give us a call: u

SC07, Reno, NVVirtual Workspaces: Acknowledgements l Workspace team: u Kate Keahey u Tim Freeman u Borja Sotomayor l Funding u NSF SDCI Missing Links u NSF CSR Virtual Playgrounds u DOE CEDPS Project l With thanks to many collaborators: u Jerome Lauret (STAR, BNL), Doug Olson (STAR, LBNL), Marty Wesley (rPath), Stu Gott (rPath), Ken Van Dine (rPath), Predrag Buncic (Alice, CERN), Haavard Bjerke (CERN), Rick Bradshaw (Bcfg2, ANL), Narayan Desai (Bcfg2, ANL), Duncan Penfold-Brown (Atlas,uvic), Ian Gable (Atlas, uvic), David Grundy (Atlas, uvic), Ti Leggit (University of Chicago), Greg Cross (University of Chicago), Mike Papka (University of Chicago/ANL)