A Case for Virtualizing Nodes on Network Experimentation Testbeds Konrad Lorincz Harvard University June 1, 2015June 1, 2015June 1, 2015

June 1, Motivation In the beginning …. 1960s, 1970s u hardware was expensive => commodity u computer systems shared by multiple usersToday u hardware is relatively cheap u connectivity and location are a commodity u Network experimentation testbeds are expensive => shared » ex: PlanetLab, Globus, Centurion PlanetLab

June 1, Sharing Scenario l Current mode of sharing 1. Each user has an account 1. Each user has an account on the testbed 2. To perform experiment – user logs on one node, runs script which starts jobs on a subset of the nodes l Unfortunate Scenarios 1. Inconsiderate user: Node N is performing critical experimental computation for user A » User B runs his/her script that starts computationally intensive job on node N 2. Considerate user: Node N is running A’s experiment » User B checks load on node N, determines it’s very low, starts a moderately computation intensive job on node N

June 1, Current Way to Solve This l In a Small Lab u John posts a message on the bulletin board … l Analysis u May work OK in a small lab => clearly does not scale u Inefficient use of resources Please don’t use machines X, Y, and Z between 5/2/02 – 5/5/02. I am running my experiments. Thanks, -John PS. If you absolutely must log on one of these machines during this time, please don’t run any computationally intensive jobs. Please don’t use machines X, Y, and Z between 5/2/02 – 5/5/02. I am running my experiments. Thanks, -John PS. If you absolutely must log on one of these machines during this time, please don’t run any computationally intensive jobs.

June 1, Desired Sharing Properties Need a good mechanism for sharing so that users don’t get in each other’s way u Ideal Scenario: Researcher A specifies to the scheduler that it needs N number of machines/nodes, for H number of hours, each with W amount of RAM, X amount of disk size, CPU equivalent to Y MHz, and bandwidth of Z Mbps. Desired Properties of Mechanism for Sharing u Efficient multiplexing of resources amongst users u Control over resources » Reservation of resources » Enforcement u Isolation » Security » Performance u Customization

June 1, Solution Claims 1. Sharing problem can be solved by => virtualizing the testbed nodes 2. Good way to virtualize nodes => use a VMM

June 1, Advantages of Virtualization l Efficient Multiplexing of Resources u Takes advantage of statistical multiplexing (demand-based) u Resources requested, if granted, are guaranteed for the duration of the experiment » Only need to request resources needed => may be very few

June 1, Advantages of Virtualization l Control Over Resources u Can be fine-grained and precise u Allows for reservation/allocation of resources u Enforcement of reservations and policies u Commercialization? – if it can be controlled and accounted for => service could be sold

June 1, Advantages of Virtualization l Isolation u Security » Almost perfect security from other VMs (i.e. users) » Namespaces of each VM is private => cannot even name a resource on another VM » Even if a VM is compromised, the rest are not » Downside: n Resources don’t span multiple protection domains => can’t share a common service across VMs u Performance – aim for soft guarantees » Actions of other users on the node don’t affect my availability to resources (guarantees) n If Bob overloads his VM, it won’t affect my VM

June 1, Advantages of Virtualization l Customization u User may ask for exact machine configuration he/she requires (RAM, disk size, CPU, network bandwidth) u May load different OSes or modified/customized ones » User may require a particular OS (FreeBSD) » Denali kernel can be loaded and run on » User has superuser privileges u Easy management » user configures once a customized VM image => transfers it to required nodes or other testbeds

June 1, Arguments Against Virtualization l Overhead of Running a VMM 10% u Relatively small! VMware server, Denali + others report about 10% u Hardware is getting faster => overhead will be even less of a factor

June 1, Arguments Against Virtualization l Use Dedicated Physical Machines u Reservation too-coarse grained » must dedicate an entire physical machine to a user => may require the equivalent of 1/8 of the machine u More expensive » More expensive to buy 6-12 machines then one more powerful one + VMM "We're running 20 virtual machines on one four-way system, and it's handling everything from CRM applications and security to application development and testing, all of which has saved us huge amounts of time and money in hardware costs." -- Alan Thomas Senior Technical Consultant, National Gypsum Company

June 1, Arguments Against Virtualization l Do/Implement the sharing at the OS level u VMMs are much simpler than conventional OSes u Other mechanisms at the OS level impose particular software interfaces or models to the user => with a VMM it’s clear, you get a “raw” machine u Cannot provide strong performance isolation => the need to support high-level abstractions prevents OSes from providing strong performance guarantees » Precise resource accounting is difficult because resources intertwined in the implementation of abstractions. ex: file buffer cache and TCP\IP socket buffers consume memory resources that aren’t charged to any particular app. [Denali paper]

June 1, Can It Be Implemented? l Not only feasible, but practical u Several off the shelf products exist: » VMware ESX server » IBM z/VM » Disco (for NUMA) u For us, VMMs only needs to support half-doze to dozen VMs, can easily handle (VMware ESX supports up to 64 VMs) u If a particular VMM doesn’t provide guarantees for a resource, it can be easily extended with well known scheduling algs. » Fair queuing => network bandwidth » Stride scheduling => CPU » Cello framework => disk bandwidth allocation

June 1, Conclusion l Network experimentation testbeds are a commodity l Users have different needs + require guarantees. Currently no formal way to handle this l Virtualizing testbed nodes by running a VMM is a good solution. 1. Efficient multiplexing of resources 2. Fine-grained control over physical resources 3. Isolation 4. Customization

June 1, Hardware (HW) VMM OS 1OS 2OS 5 App 1App 2App 1 … … HW