INFSO-RI An On-Demand Dynamic Virtualization Manager Øyvind Valen-Sendstad CERN – IT/GD, ETICS Virtual Node bootstrapper (VNB)
INFSO-RI Contents Brief introduction to ETICS ETICS build infrastructure Improving the build/test process The Virtual Node Bootstrapper Achievements Status, Lessons learned and Further work Summary 2
INFSO-RI What ETICS is It’s a software engineering management system It’s a build and test infrastructure It provides tools and resources to configure, manage and analyse build and test runs It provides a common interface to diverse projects to facilitate knowledge sharing and operations management It has an open repository of configuration metadata, packages, reports. The goal is to share information, but also to reliably store and preserve information It has a plugin-based architecture and APIs to allow integrating ETICS into existing processes and extending it with custom actions It’s multi-platform and independent from any specific build or test tool 3
INFSO-RI Architecture ETICS is not ‘just’ a build system
INFSO-RI ETICS build infrastructure - static platforms X 5 SL5 / 64 X 5 Mac OSX X 5 SL5 / 32 X 5 SL5 / 64 X 5 SLC4 X 5 Debian X 5 Ubuntu 7 X 5 SL5 New SL5 Job Physical node Virtual static nodes Problem 1: The composition of the pool is fixed. If we have 20 SL5 nodes all busy and a new request for SL5 arrives, it is queued even if many nodes are available with other platforms
INFSO-RI ETICS build infrastructure - static platforms X 5 SL5 / 64 X 5 Mac OSX X 5 SL5 / 32 X 5 RH7 X 5 SLC4 X 5 Debian X 5 Ubuntu 7 X 5 SL5 New RH7 Job Physical node Virtual static nodes Problem 2: If a platform is very rarely used, but we want to make it available, one or more nodes will do nothing most of the time
INFSO-RI The proposed solution: dynamic virtual platforms New job Available X 5 Physical machine Virtual machine Available X 5 Available X 5 Available VM repository X Available X 5 Available X 5 Available X 5 Available X 5
INFSO-RI The bootstrap process 8 1.User submits a build/test for a specific platform 2.Request arrives to VNB 3.VNB determines platform and hypervisor support 4.Downloads a suitable VM from repository (or local cache) 5.Deploys the VM 6.Executes job 7.Returns results 8.Cleans up
INFSO-RI Deploying the bootstrapper
INFSO-RI Deploying the bootstrapper
INFSO-RI
Deploying the bootstrapper
INFSO-RI Deploying the virtual machine
INFSO-RI Deploying the virtual machine
INFSO-RI Deploying multiple virtual machines on the same node
INFSO-RI Deploying multiple virtual machines on the same node
INFSO-RI Deploying multiple virtual machines on the same node
INFSO-RI Achievements Perform platform deployment on demand Increase availability Increase capability Provide reproducible environments Reduce maintenance New releases Upgrades Security patches Accommodate for specific software dependencies Offer privileged access (root access to the VM, not the host) Enable post build analysis (VM snapshot can be saved with results) Virtual machine image customization (users can provide their own) 19
INFSO-RI Status Working Bootstrap process, tested on VMware and KVM Remaining Testing on Xen Support for NT-based operating systems Caching VM images locally 20
INFSO-RI Lessons learned Privileges The bootstrapper runs in the user-space and need privileges to use Xen and Vmware Compatibility The kernel in a virtual machine created in VMware is not necessarily able to run in a Xen or KVM environment 21
INFSO-RI Further work Deploy virtual machine image into a folder and chroot in order to run natively on a worker node Update VNB on VM during startup Support for NT-based operating systems
INFSO-RI Open issues Reproducibility Different implementations of the virtual machine hardware can cause different behaviour of a virtual machine Firewall Ephemeral port range can be blocked by strict firewalls 23
INFSO-RI Summary The VNB gives Better service availability Better resource utilization The VNB has Generic design Emulates when no hypervisors are present The VNB allows Running multiple bootstrappers on a single host