Experience integrating a production private cloud in a Tier 1 Grid site Ian Collier Andrew Lahiff, George Ryall STFC RAL Tier 1 ISGC 2015 – March 20 th 2015

RAL Tier 1 Context Scientific Computing Department Cloud Batch farm related virtualisation

Context RAL Tier 1 sits in Science & Technology Facilities Council (STFC) Scientific Computing Department Primary project is WLCG Tier 1 For many reasons we seek to make our resources more adaptable, and more readily used by other projects –Many department projects, for example those funded by Horizon 2020 will benefit from an elastic and scalable resource –Self-service internal development systems –New interfaces for WLCG VOs –Investigating providing compute resources to other parts of STFC (Diamond Light Source, ISIS Neutron Source) for whom we already provide data services

Context March 2014 secured funding for 30 well specified hypervisors & 30 storage servers –~1000 cores –~ 4TB ram –~1PB raw storage In September secured first dedicated effort to turn earlier experiments into a service with a defined service level This all builds on some 2 ½ years of experiments

Virtualisation & RAL Context Scientific Computing Department Cloud Dynamically provisioned worker nodes

History - SCD Cloud Began as small experiment 3 years ago –Initially using StratusLab & old worker nodes –Initially very quick and easy to get working –But fragile, and upgrades and customisations always harder Work until now implemented by graduates on 6 month rotation –Disruptive & variable progress Worked well enough to prove usefulness Something of an exercise in managing expectations Self service VMs proved very popular

SCD Cloud Present Carried out a fresh technology evaluation. –Things have moved on since we began with StratusLab –Chose Opennebula with ceph backend Now a service with defined (if limited) service level for users across STFC –integrated in to the existing Tier 1 configuration & monitoring frameworks IaaS upon which we can offer PaaS –One platform could ultimately be the Tier 1 itself –Integrating cloud resources in to Tier 1 grid work

SCD Cloud Setup –OpenNebula –Ceph for image store and running images –Collaborating with UGent on configuration tools for both OpenNebula and ceph –AD as initial authentication Covers all STFC staff and many partners Will add others as use cases require 2 staff began work in September one full time, one half time

SCD Cloud Ceph –Separate project to identify suitable alternatives to Castor for our grid storage identified ceph as being interesting –Have been building up experience over last 2 years –Have separate test bed (half a generation of standard disk servers (1.7PB raw) being tested for grid data storage Ceph as cloud backend –Image store, running images, and possibly a data service coupled to the cloud –Performance tests show little difference compared to local storage for running VM images & instantiation is much faster

SCD Cloud Summer (high school) student developed web interface –Kerberos authentication –Streamlined selection of images & VM templates –VNC console to running machines

SCD Cloud Being launched on self service basis to entire Scientiifc Computing department later today Still a development platform – capabilities will be changing rapidly Integrated into Tier 1 monitoring & exception handling –NO out of hours support OpenNebula and Ceph managed by Quattor/Aquilon –Result of collaboration with Ugent Standalone VMs and VMs managed by Quattor/Aquilon Early H2020 funded project - INDIGO-DataCloud

RAL Context Scientific Computing Department Cloud Batch farm related virtualisation

Bursting the batch system into the cloud Last year spoke about leveraging HTCondor power management features to dynamically burst batch work in to cloud Aims –Integrate cloud with batch system –First step: allow the batch system to expand into the cloud Avoid running additional third-party and/or complex services Leverage existing functionality in HTCondor as much as possible Proof-of-concept testing carried out with StratusLab –Successfully ran ~11000 jobs from the LHC VOs We can now ensure our private cloud is always used –LHC VOs can be depended upon to provide work

Initial situation: partitioned resources –Worker nodes (batch system) –Hypervisors (cloud) Likely to be a common situation at sites providing both batch & cloud resources Ideal situation: completely dynamic –If batch system busy but cloud not busy Expand batch system into the cloud –If cloud busy but batch system not busy Expand size of cloud, reduce amount of batch system resources 14 Bursting the batch system into the cloud cloudbatch cloudbatch

Based on existing power management features of HTCondor Virtual machine instantiation –ClassAds for offline machines are sent to the collector when there are free resources in the cloud –Negotiator can match idle jobs to the offline machines –HTCondor rooster daemon notices this match & triggers creation of VMs Virtual machine lifetime –Managed by HTCondor on the VM itself. Configured to: Only start jobs when a health-check script is successful Only start new jobs for a specified time period Shuts down the machine after being idle for a specified period –Virtual worker nodes are drained when free resources on the cloud start to fall below a specified threshold 20/05/2014HEPiX Spring RAL Site Report

20/05/2014HEPiX Spring RAL Site Report condor_collectorcondor_negotiator Worker nodes condor_startd condor_rooster Virtual worker nodes condor_startd ARC/CREAM CEs condor_schedd Central manager Offline machine ClassAds Draining

Expansion into cloud 20/05/2014HEPiX Spring RAL Site Report Cores in the cloud Running & idle jobs Idle Used in batch Used (not batch) Idle jobs Running jobs

Bursting the batch system into the cloud Last year this was a short term experiment with StratusLab Our cloud is now entering production status Ability to expand batch farm into our cloud is being integrated into our production batch system The challenge is to have a variable resource so closely bound to our batch service HTCondor makes it much easier – elegant support for dynamic resources But significant changes to monitoring –Moved to the condor health check – no nagios on virtual WNs –This has in turn fed back in to the monitoring of bare metal WNs

The Vacuum Model “Vacuum” model is becoming popular in the UK –Alternative to CE + batch system or clouds –No centrally-submitted pilot job or requests for VMs –VMs appear by “spontaneous production in the vacuum” –VMs run the appropriate pilot framework to pull down jobs –Discussed by Jeremy Coles’ in his talk on Tuesday Can we incorporate the vacuum model into our existing batch system? –HTCondor has a “VM universe” for managing VMs

Vacuum Model & HTCondor Makes use of not-commonly used features of HTCondor, including –Job hooks, custom transfer plugins, condor_chirp Features –Uses same config file as Vac –Images downloaded & cached on worker nodes –Quarantining of disk images after VMs are shutdown –Accounting data sent directly to APEL –Stuck VMs killed by PeriodicRemove expression

VM lifecycle in HTCondor 20/05/2014HEPiX Spring RAL Site Report Download disk image or copy cached image to job sandbox Setup sparse disk for CVMFS cache, create contextualization iso, … VM created Update time of last heartbeat from VM Copy disk image to quarantine area, add ShutdownCode from VM to job ClassAd Transfer plugin Job prepare hook condor_vm-gahp Job update hook condor_chirp Job exit hook condor_chirp

The Vacuum Model & HTCondor Usage –Successfully running regular SAM tests from the GridPP DIRAC instance –Running ATLAS jobs

RAL Context Scientific Computing Department Cloud Batch farm related virtualisation

Summary Private cloud has developed from a small experiment to a service with a defined service level –With constrained effort - Slower than we would have liked –The prototype platforms has been well used –Ready to provide resources to funded projects on schedule. Demonstrated transparent expansion of batch farm into cloud and Vacuum model. Whole Tier 1 service becoming more flexible

Questions ?