Predrag Buncic (CERN/PH-SFT) Virtualization R&D (WP9) Status Report

CernVM - WP9 Status 11/11/ CernVM Project Portable Analysis Environment using Virtualization Technology (WP9)  Provide a complete, portable and easy to configure user environment for developing and running LHC data analysis locally and on the Grid independent of physical software and hardware platform (Linux, Windows, MacOS) Decouple application lifecycle from evolution of system infrastructure Reduce effort to install, maintain and keep up to date the experiment software People  Carlos Augado Sanchez (fellow, since 01/01/2008)  Leandro Franco (Marie Currie fellow, extended for 2 months to work on CVMFS)  Steffen Klemer (Summer student)  Artem Harutyunyan (visitor for 6 weeks, worked on adapting CernVM for ALICE)  Jacob Bloomer (Doctoral student, starting from 01/01/2009)  Predrag Buncic (PH/SFT 50%), Pere Mato (PH/SFT 5%) Experiment contacts  ATLAS (Paolo Calafura, Yushu Yao)  CMS (Benedikt Hagner, Marko Petek)  LHCb (Marko Clemencic, Hubert Degaudenzi)  ALICE (P.B.)

CernVM - WP9 Status 11/11/ Current Status  Technology survey and initial prototyping carried out during first 3 months  Procured hardware and software licenses and setup a service and support infrastructure (web site, project and bug tracker)  Following the successful Kickoff Workshop in April, work plan redefined based on feedback from experiments Opted for frequent beta releases with constant interaction with experiments  Close interaction with experiments Bi-weekly steering meetings Regular reports to Architects Forum ATLAS is showing the biggest interest with LHCb and ALICE following CMS has recently found and committed manpower  Conferences and Talks White Area Lecture (CERN IT/GD) ACAT 2008 (plenary talk & poster)  First release ahead of the original schedule, delivered on time at the end of the October

How do we want to do that? Run exp. software locally Edit, compile, debug.. Submit jobs to Grid Retrieve and display results

CernVM - WP9 Status 11/11/ Architecture “Thin” Virtual Software Appliance

CernVM - WP9 Status 11/11/ Key Components rPath Linux 1 (  Slim Linux OS binary compatible with RH/SLC4 rAA - rPath Linux Appliance Agent  Web & XMLRPC User Interface rBulder  A tool to build VM images for various virtualization platforms  1 user licence acquired in April Substantial discount and no extra charges for support CVMFS - CernVM file system  Read only file system optimized for software distribution Aggressive caching  Operational in offline mode For as long as you stay within the cache  Derived from Open Source components of Parrot and GrowFS ( Adapted for CernVM by Leandro Franco Build types  Installable CD/DVD  Stub Image  Raw Filesystem Image  Netboot Image  Compressed Tar File  Demo CD/DVD (Live CD/DVD)  Raw Hard Disk Image  Vmware ® Virtual Appliance  Vmware ® ESX Server Virtual Appliance  Microsoft ® VHD Virtual Apliance  Xen Enterprise Virtual Appliance  Virtual Iron Virtual Appliance  Parallels Virtual Appliance  Amazon Machine Image  Update CD/DVD  Appliance Installable ISO

CernVM - WP9 Status 11/11/ rAA Web UI

CernVM - WP9 Status 11/11/ CernVM Components group-cernvm (core packages) group-cernvm (core packages) group-cernvm-devel (development tools) group-cernvm-devel (development tools) group-slc4 (SLC4 compatibility libs) group-slc4 (SLC4 compatibility libs) group- (groups and extra packages required by experiment) group- (groups and extra packages required by experiment) 100 MB compat-db4 compat-openssl compat-linstdc++slc3 compat-libxml2 compat-readline compat-tcl compat-tk group- -desktop (lightweight X environment) group- -desktop (lightweight X environment) group-cernvm-desktop X11 group-cernvm-desktop X11

CernVM - WP9 Status 11/11/ Scalable Infrastructure 5 new servers (bought in August) 1 inherited from ALICE 2 retired from IT

Experiments are in control of their software

CernVM - WP9 Status 11/11/ Application benchmarks

CernVM - WP9 Status 11/11/ CernVM v.s. LXPLUS CernVM for ATLAS (benchmarks by Yushu Yao)  For the first time running, CernVM is 2-3 times slower, depending on the network speed, since it needs to cache files  Once the files are cached, the speed on CernVM is roughly the same as on LXPLUS.  CernVM is faster when reading large files (since they are local) Note: LXPLUS has a higher CPU rate than test computer (3GHz vs. 2.66GHz) Ratio to the execution time in LXPLUS

CernVM - WP9 Status 11/11/ Developing CernVM JANFEBMARAPRMAYJUNJULAUGSEPOCTNOVDEC -Preparation Release Release 0.6 On time! Release 1.0 Available for download now! Kickoff Workshop - Release Release Release 0.91 (RC1) -Release 0.92 (RC2)

CernVM - WP9 Status 11/11/ Release Available now for download from  Can be run on  Linux (KVM, Xen,VMware Player, VirtualBox)  Windows(WMware Player, VirtualBox)  Mac (Fusion, Parallels, VirtualBox) Release Notes HowTo Appliance can be configured and used with ALICE, LHCb, ATLAS (and CMS) software frameworks

Download history 1.0 released 10 days ago

CernVM - WP9 Status 11/11/ Primary (master) software repository Secondary software repository Secondary software repository Reverse Proxy Load Balancing Reverse Proxy Load Balancing Reverse Proxy CernVM (thin client) Site Reverse Proxy Load Balancing Reverse Proxy Regional Reverse Proxy Removing Single Point of Failure Content Delivery Network

CernVM - WP9 Status 11/11/ Bridging Grids & Clouds BOINC  Open-source software for volunteer computing and grid computing   Summer student project in OpenLab based on experience and CernVM image, aims to run ATLAS simulation using BOINC infrastructure  Successfully run ATLAS PanDA job in CernVM  CoPilot - an addition to CernVM to provide enable Pilot Jobs to run in VM AliEn,DIRAC,PanDA.. BOINC E PanDA Pilot CoPilot

CernVM - WP9 Status 11/11/ Mailing lists  (Open list for announcements and discussion) (End-user support for the CernVM project) Savannah Portal  Please submit bugs and feature requests to Savannah at CernVM Home Page:  rBuilder & Download Page:  CernVM Links…

2008 Expenditure

2008 Budget

CernVM - WP9 Status 11/11/ Work Plan Remove single point of failure, develop and test Content Delivery Network  Doctoral student (J. Bloomer)  Compare different approaches and select the one most appropriate to our problem  Requires distributed testbed Can be realized as contributions from different labs participating in LHC experiments Alternative is to use emerging commercial Cloud infrastructure (Amazon) Migrate CernVM to rPath Linux 2 (SLC5 compatible)  Currently we are SLC4 compatible but experiments plan to move to SLC4 Migrate to VMware Infrastructure 3  Got 12 VI3 (free) licenses from VMware  Requires certified hardware SAN Continue collaboration with experiments  Improving UI and usability for ATLAS (2 months)  Work on developing CoPilot framework for ALICE (2 months) Initiate closer collaboration with IT  Replicating our pilot services and running them on IT hosted resources Consult and support other projects trying to use this technology  Geant4 Virtual Appliance for tutorials and testing  May require extra rBuilder license

2009 Budget