INFSO-RI-223782 An On-Demand Dynamic Virtualization Manager Øyvind Valen-Sendstad CERN – IT/GD, ETICS Virtual Node bootstrapper.

Slides:



Advertisements
Similar presentations
Virtual Machine Technology Dr. Gregor von Laszewski Dr. Lizhe Wang.
Advertisements

Profit from the cloud TM Parallels Dynamic Infrastructure AndOpenStack.
An Approach to Secure Cloud Computing Architectures By Y. Serge Joseph FAU security Group February 24th, 2011.
SC7 WG6 Rome Engineering Ingegneria Informatica S.p.A. INFSO-RI Isabel Matranga ETICS Automated Building,Testing and Quality Assurance.
EUROPEAN UNION Polish Infrastructure for Supporting Computational Science in the European Research Space User Oriented Provisioning of Secure Virtualized.
Presented by Sujit Tilak. Evolution of Client/Server Architecture Clients & Server on different computer systems Local Area Network for Server and Client.
Virtualization for Cloud Computing
Installing software on personal computer
LINUX Virtualization Running other code under LINUX.
Tanenbaum 8.3 See references
CERN IT Department CH-1211 Genève 23 Switzerland t Virtualization with Windows at CERN Juraj Sucik, Emmanuel Ormancey Internet Services Group.
Virtual Infrastructure in the Grid Kate Keahey Argonne National Laboratory.
INTRODUCTION TO CLOUD COMPUTING CS 595 LECTURE 7 2/23/2015.
 Cloud computing  Workflow  Workflow lifecycle  Workflow design  Workflow tools : xcp, eucalyptus, open nebula.
INFSO-RI Quality Assurance with ETICS – multi- node automated testing CGW 09 M.Zurek, A. A. Rodriguez, A. Aimar, A. di Meglio, L. Dini CERN Krakow,
1 port BOSS on Wenjing Wu (IHEP-CC)
Introduction to VMware Virtualization
+ CS 325: CS Hardware and Software Organization and Architecture Cloud Architectures.
Presented by: Sanketh Beerabbi University of Central Florida COP Cloud Computing.
The Open Source Virtual Lab: a Case Study Authors: E. Damiani, F. Frati, D. Rebeccani, M. Anisetti, V. Bellandi and U. Raimondi University of Milan Department.
INFSO-RI Module 01 ETICS Overview Alberto Di Meglio.
Grids, Clouds and the Community. Cloud Technology and the NGS Steve Thorn Edinburgh University Matteo Turilli, Oxford University Presented by David Fergusson.
INTRODUCTION TO VIRTUALIZATION KRISTEN WILLIAMS MOSES IKE.
COMS E Cloud Computing and Data Center Networking Sambit Sahu
INFSO-RI Module 01 ETICS Overview Etics Online Tutorial Marian ŻUREK Baltic Grid II Summer School Vilnius, 2-3 July 2009.
Focus on SCVMM features and an introduction on how to implement into your current environment. Overview of System Center Virtual Machine Manager 2012 Jim.
ETICS All Hands meeting Bologna, October 23-25, 2006 NMI and Condor: Status + Future Plans Andy PAVLO Peter COUVARES Becky GIETZEL.
INFSO-RI SA1 Service Management Alberto AIMAR (CERN) ETICS 2 Final Review Brussels - 11 May 2010.
INFSO-RI Support for IPv6 in ETICS EGEE’08 Conference, Istanbul, September 2008 Marian ZUREK CERN - ETICS
Enabling Grids for E-sciencE EGEE and gLite are registered trademarks Usage of virtualization in gLite certification Andreas Unterkircher.
Conference name Company name INFSOM-RI Speaker name The ETICS Job management architecture EGEE ‘08 Istanbul, September 25 th 2008 Valerio Venturi.
Enabling Grids for E-sciencE EGEE and gLite are registered trademarks Tools and techniques for managing virtual machine images Andreas.
INFSO-RI SA1 Status Report Status and Progress of the ETICS Services ETICS2 First Review Alberto AIMAR CERN Brussels 3 April 2009.
Protection of Processes Security and privacy of data is challenging currently. Protecting information – Not limited to hardware. – Depends on innovation.
D4Science and ETICS Building and Testing gCube and gCore Pedro Andrade CERN EGEE’08 Conference 25 September 2008 Istanbul (Turkey)
INFSO-RI ETICS The Software Engineering Infrastructure EGEE 08 Istanbul, September 2008 Alberto Di Meglio CERN – ETICS Project manager.
INFSO-RI SA2 ETICS2 first Review Valerio Venturi INFN Bruxelles, 3 April 2009 Infrastructure Support.
INFSO-RI ETICS Local Setup Experiences A Case Study for Installation at Customers Location 4th. All Hands MeetingUwe Müller-Wilm VEGA Bologna, Nov.
EGEE-III INFSO-RI Enabling Grids for E-sciencE EGEE and gLite are registered trademarks Patch Preparation SA3 All Hands Meeting.
INFSO-RI JRA2 Test Management Tools Eva Takacs (4D SOFT) ETICS 2 Final Review Brussels - 11 May 2010.
EMI is partially funded by the European Commission under Grant Agreement RI Build and Test Services of the EMI project: Lessons Learned and Perspectives.
36 th LHCb Software Week Pere Mato/CERN.  Provide a complete, portable and easy to configure user environment for developing and running LHC data analysis.
Predrag Buncic (CERN/PH-SFT) CernVM Status. CERN, 24/10/ Virtualization R&D (WP9)  The aim of WP9 is to provide a complete, portable and easy.
ETICS An Environment for Distributed Software Development in Aerospace Applications SpaceTransfer09 Hannover Messe, April 2009.
EMI is partially funded by the European Commission under Grant Agreement RI EMI SA2 Report Andres ABAD RODRIGUEZ, CERN SA2.4, Task Leader EMI AHM,
INFSOM-RI Elisabetta Ronchieri INFN CNAF ETICS 2 nd EU Review (CERN) 15 February 2008 WP3 - Software Configuration Tools and Methodologies.
HEPiX Virtualisation working group Andrea Chierici INFN-CNAF Workshop CCR 2010.
Workspace Management Services Kate Keahey Argonne National Laboratory.
Intro To Virtualization Mohammed Morsi
Unit 3 Virtualization.
Virtualization for Cloud Computing
Chapter 6: Securing the Cloud
Containers as a Service with Docker to Extend an Open Platform
Cloud Technology and the NGS Steve Thorn Edinburgh University (Matteo Turilli, Oxford University)‏ Presented by David Fergusson.
Introduction to VMware Virtualization
Fundamentals Sunny Sharma Microsoft
Blueprint of Persistent Infrastructure as a Service
Infrastructure Orchestration to Optimize Testing
Building a Virtual Infrastructure
StratusLab Final Periodic Review
StratusLab Final Periodic Review
The ETICS Build and Test Service
ETICS Services Management
Virtualization in the gLite Grid Middleware software process
Running other code under LINUX
OS Virtualization.
Virtualization Layer Virtual Hardware Virtual Networking
Module 01 ETICS Overview ETICS Online Tutorials
Internet Engineering Course
Containers on Azure Peter Lasne Sr. Software Development Engineer
Presentation transcript:

INFSO-RI An On-Demand Dynamic Virtualization Manager Øyvind Valen-Sendstad CERN – IT/GD, ETICS Virtual Node bootstrapper (VNB)

INFSO-RI Contents Brief introduction to ETICS ETICS build infrastructure Improving the build/test process The Virtual Node Bootstrapper Achievements Status, Lessons learned and Further work Summary 2

INFSO-RI What ETICS is It’s a software engineering management system It’s a build and test infrastructure It provides tools and resources to configure, manage and analyse build and test runs It provides a common interface to diverse projects to facilitate knowledge sharing and operations management It has an open repository of configuration metadata, packages, reports. The goal is to share information, but also to reliably store and preserve information It has a plugin-based architecture and APIs to allow integrating ETICS into existing processes and extending it with custom actions It’s multi-platform and independent from any specific build or test tool 3

INFSO-RI Architecture ETICS is not ‘just’ a build system

INFSO-RI ETICS build infrastructure - static platforms X 5 SL5 / 64 X 5 Mac OSX X 5 SL5 / 32 X 5 SL5 / 64 X 5 SLC4 X 5 Debian X 5 Ubuntu 7 X 5 SL5 New SL5 Job Physical node Virtual static nodes Problem 1: The composition of the pool is fixed. If we have 20 SL5 nodes all busy and a new request for SL5 arrives, it is queued even if many nodes are available with other platforms

INFSO-RI ETICS build infrastructure - static platforms X 5 SL5 / 64 X 5 Mac OSX X 5 SL5 / 32 X 5 RH7 X 5 SLC4 X 5 Debian X 5 Ubuntu 7 X 5 SL5 New RH7 Job Physical node Virtual static nodes Problem 2: If a platform is very rarely used, but we want to make it available, one or more nodes will do nothing most of the time

INFSO-RI The proposed solution: dynamic virtual platforms New job Available X 5 Physical machine Virtual machine Available X 5 Available X 5 Available VM repository X Available X 5 Available X 5 Available X 5 Available X 5

INFSO-RI The bootstrap process 8 1.User submits a build/test for a specific platform 2.Request arrives to VNB 3.VNB determines platform and hypervisor support 4.Downloads a suitable VM from repository (or local cache) 5.Deploys the VM 6.Executes job 7.Returns results 8.Cleans up

INFSO-RI Deploying the bootstrapper

INFSO-RI Deploying the bootstrapper

INFSO-RI

Deploying the bootstrapper

INFSO-RI Deploying the virtual machine

INFSO-RI Deploying the virtual machine

INFSO-RI Deploying multiple virtual machines on the same node

INFSO-RI Deploying multiple virtual machines on the same node

INFSO-RI Deploying multiple virtual machines on the same node

INFSO-RI Achievements Perform platform deployment on demand Increase availability Increase capability Provide reproducible environments Reduce maintenance New releases Upgrades Security patches Accommodate for specific software dependencies Offer privileged access (root access to the VM, not the host) Enable post build analysis (VM snapshot can be saved with results) Virtual machine image customization (users can provide their own) 19

INFSO-RI Status Working Bootstrap process, tested on VMware and KVM Remaining Testing on Xen Support for NT-based operating systems Caching VM images locally 20

INFSO-RI Lessons learned Privileges The bootstrapper runs in the user-space and need privileges to use Xen and Vmware Compatibility The kernel in a virtual machine created in VMware is not necessarily able to run in a Xen or KVM environment 21

INFSO-RI Further work Deploy virtual machine image into a folder and chroot in order to run natively on a worker node Update VNB on VM during startup Support for NT-based operating systems

INFSO-RI Open issues Reproducibility Different implementations of the virtual machine hardware can cause different behaviour of a virtual machine Firewall Ephemeral port range can be blocked by strict firewalls 23

INFSO-RI Summary The VNB gives Better service availability Better resource utilization The VNB has Generic design Emulates when no hypervisors are present The VNB allows Running multiple bootstrappers on a single host