1. Jaeyoung Yoon Computer Sciences Department University of Wisconsin-Madison jyoon@cs.wisc.edu http://www.cs.wisc.edu/condor Virtual Machines in Condor

2. www.cs.wisc.edu/condor Virtual Machines › VMware  Pros: Full virtualization, no modification for guest OS, and checkpoint/restart capability  Cons: Commercial product and performance issue compared to Xen › Xen  Pros: Open source, good performance, checkpoint/restart and live migration capability  Cons: Requires OS modification and must divide memory between host and VMs in advance › UML (User Mode Linux) etc.

3. www.cs.wisc.edu/condor Benefit of using Virtual Machine in Condor › Sandbox  Security and Isolation › Independent environment  Customizing environment for Condor › Several OS’s on a single physical machine  Support for a wider variety of jobs › Finer Resource Control  Assign memory size to each VM explicitly › Checkpoint and migration  All memory of VM can be saved(or suspended) and restarted(or resumed) later

4. www.cs.wisc.edu/condor Difficulty of using Virtual Machine in Condor › Hard to manage system memory efficiently › Need to know some information of host machine inside VM › Need to setup some environments in VM › If a VM cannot use the distributed file system, Condor’s file transfer or remote IO mechanism should be used › Need IP address for each VM

5. www.cs.wisc.edu/condor How to use VM in Condor Scenario 1 › Already launched VM is ready to be used as a execution machine for Condor jobs › Condor daemons should be installed and run on both the virtual and the host machine, which are all exposed to the pool › Condor startd on the host machine controls when a launched VM is used for Condor. › Supported by Condor 6.7.18 and all future releases › Pros: Easy to implement › Cons: Inefficient memory management

6. www.cs.wisc.edu/condor Scenario 1 Host Machine Virtual machine Central Manager Collector Startd Negotiator Startd Execution machine Schedd Submit machine Communication pathway

7. www.cs.wisc.edu/condor Current Implementation How can VM get the information for host machine 1. Query ClassAd for host ClassAd for VM 2. ClassAd for host machine Virtual Machine VMP_HOST_MACHINE = host.domain.com ClassAd for host Name = host.domain.com” TotalLoadAvg = 1.860000 KeyboardIdle = 50 …. Host Machine VMP_VM_LIST = vmware1.domain.com Name = “vmware1.domain.com” TotalLoadAvg = 2.670000 KeyboardIdle = 517656 …. Before Query Name = “vmware1.domain.com” TotalLoadAvg = 2.670000 KeyboardIdle = 517656 … HOST_Name=“host.domain.com” HOST_TotalLoadAvg = 1.860000 HOST_KeyboardIdle = 50 … After Query

8. www.cs.wisc.edu/condor Current Implementation How does a VM get permission from host machine 1. Send VM_REGISTER ClassAd for VM Virtual Machine VMP_HOST_MACHINE = host.domain.com ClassAd for host START = ((KeyboardIdle > 150 ) && ( LoadAvg <= 0.3 )) Host Machine VMP_VM_LIST = vmware1.domain.com START = False If host status == (‘owner’ | ‘unclaimed’) START = ((KeyboardIdle > 150 ) && ( LoadAvg <= 0.3 )) else 2. Reply permission START = ((KeyboardIdle > 150 ) && (HOST_KeyboardIdle > 150 ) && ( LoadAvg <= 0.3 ) && (HOST_TotalLoadAvg <= 0.3 )) If permission == yes else START = False

9. www.cs.wisc.edu/condor Issues in current implementation for Scenario 1 › Problem: › Problem: host machine cannot be used for Condor any more after sending a permission to a virtual machine. › Possibility: › Possibility: A user may want to use both virtual and host machine in a SMP machine. › Possible solution: › Possible solution: After sending permission, host machine does not change START expression. Instead, the virtual machine sends its status to host machine periodically and host machine decides the permission for each virtual machine when a Condor job is assigned.

10. www.cs.wisc.edu/condor How to use VM in Condor Scenario 2 › Virtual Machine is launched on demand to serve a Condor job › Checkpoint and migration per virtual machine base can be used › Startd on host machine may have to advertise more than one OS. › A specific daemon in virtual machine needs to communicate with host machine. The daemon receives a command from host machine and executes it when a Condor job is assigned.

11. www.cs.wisc.edu/condor How to use VM in Condor Scenario 2 › Not yet implemented in Condor but we hope to do it soon. › Pros: Efficient memory management › Cons: Complex to implement

12. www.cs.wisc.edu/condor Scenario 2 Host Machine Virtual machine Daemon Startd Execution machine Negotiator Central Manager Collector Schedd Submit machine 1 launching Starter Shadow Communication pathway Creating/forking process daemon Submit machine 2 Virtual machine schedd Host Machine

13. www.cs.wisc.edu/condor Issues in Scenario 2 › Stop VM and save the entire memory of VM instead of suspending an executed Condor job when a user returns to the host machine › During migration if there is no shared file system, files used by a Condor job, including a program file, should be transferred because it is very hard to copy entire disk image. › Xen live migration technique can be effectively used for direct migration without checkpointing.

14. www.cs.wisc.edu/condor Scenario 2 - Migration Host Machine Virtual machine Central Manager Collector Daemon Negotiator Startd Execution machine 1 Schedd Submit machine Host Machine Virtual machine Daemon Startd Execution machine 2 migration launching Communication pathway before migration Communication pathway after migration Shadow Starter Creating/forking process

15. www.cs.wisc.edu/condor › Virtual Machine offers flexible solution in Condor  Sandbox for security  Can provide more than one OS on a single physical machine  Can provide customized environment for Condor › Scenario 1 has been already supported since Condor 6.7.18 › Scenario 2 is not yet implemented in Condor. That is a future work in Condor. Summary