Pete Gronbech GridPP Project Manager April 2016 Oxford Experiences Pete Gronbech GridPP Project Manager April 2016

Oxford Grid Cluster GridPP4 status Autumn 2015 Current capacity 16,768HS06 1300TB DPM Storage a mixture of older Supermicro 26, 24 and 36 bay servers, but the major capacity is provided by Dell 510 and 710s. 12 bay with 2 or 4 TB disks. Majority of CPU nodes are ‘twin-squared’ Viglen Supermicro worker nodes have been installed. Intel E5- 8 core (16 Hyper-threaded cores each) provides 1300 job slots with 2GB RAM. The Grid Cluster now runs HT Condor behind an ARC CE. April 2016

Oxford’s Grid Cluster April 2016

April 2016

Primary VOs ATLAS LHCb CMS Tier-3 ALICE Support Others April 2016 Main priority 6th largest UK site (By metrics) ~1300 CPU cores with 1.3PB storage LHCb 6th Largest UK site (By metrics) CMS Tier-3 Supported by RALPPD’s PhEDEx server Useful for CMS, and for us, keeping the site busy in quiet times During 2015 delivered the same percentage as Bristol However can block Atlas jobs and during accounting period not so desirable ALICE Support There is a need to supplement the support given to ALICE by Birmingham. Made sense to keep this in SouthGrid so Oxford have deployed an ALICE VO box Oxford provides roughly 40% of ALICE CPU (No storage) Others 5th largest by Metrics April 2016

Other Oxford Work UK Regional Monitoring IPv6 Testing Oxford runs the nagios based WLCG monitoring for the UK These include the Nagios server itself, and support nodes for it, SE, MyProxy and WMS/LB Multi VO Nagios Monitoring. IPv6 Testing We have taken a leading part in the IPv6 testing, many services enabled and tested by the community. perfSONAR IPv6 enabled. RIPE Atlas probe also on IPv6. Cloud Development Openstack test setup (Has run Atlas jobs) VAC setup (LHCb, Atlas & GridPP Dirac server jobs) Viab now installed April 2016

Approx. FTE Grid Tasks HTCondor Batch 0.2 DPM Storage Monitoring and tuning in the batch system DPM Storage Installation, testing, monitoring and maintenance of SPACETOKENS, and h/w Grid Ops & Management Installation and management system (Cobbler, Puppet). GGUS tickets etc Ovirt VM infrastructure 0.1 SL based ovirt system setup to provide VM infrastructure for all service and test nodes. IPv6 testing IPv6 UI used by others in GridPP. IPv6 se, Perfsonar Early adopter and also ATLAS RIPE probes Security Team membership Meetings and running UK security tests VAC testing Early adopter and tester. Viab testing 0.05 Recent testing National Nagios infrastructure GridPP wide essential service VO Nagios for UK GridPP wide service Backup VOMS Server DPM on SL7 testing Part of storage group, early testing and bug discovery. Openstack Cloud test setup Open stack setup used for testing FTE ≈1.5 April 2016

Juggling Many Tasks Successfully April 2016

Plan for GridPP5 Manpower ramp down from 1.5FTE to 0.5 (average of 1FTE over GridPP5) Need to simplify the setup and reduce the number of development tasks April 2016

Plan for GridPP5 Storage Upgrade DPM to 1.8.10 running on SL6 fully puppetized (Initially tried to install on SL7 but found too many incompatibilities to deal with in the short time available) We were the first site to install the latest DPM with the puppet modules supplied by the DPM developers. This is a good thing but meant we were on the bleeding edge and fed back many missing features and bugs. Ewan worked with Sam to find all the missing parts. We are still finding out now about some of the issues such as publishing. Decommission out of warranty h/w April 2016

Storage directions Reduced emphasis on storage?? Heading in the direction of T2C, and were acting as a test site for this way of working. Atlas FAX is intended to deal with cases where the files are missing. Jobs could be sent to a site knowing the data is not there to force use of FAX, whether this scales is as yet unknown. Actually using xrootd redirection without an SE is as yet untested. Currently short of staff so we will continue in traditional mode. Question needs answering before next h/w round, do we continue in the T2C direction or actually act as a T2D for SouthGrid sites. We do have good networking and a fair sized resource currently. We have rather too much storage to be used just as cache. What do the experiments want from us? Will continue running an SE for the foreseeable future. April 2016

Decommissioning old storage 17 Supermicro servers removed from the DPM SE. (Reduction of 320TB new total 980TB) April 2016

Storage March 2016 Simplified  Old servers removed, switched off or repurposed Software up to date OS up to date Simplified  April 2016

Openstack I’m here, any work? Panda Yes, job WN’s 8 Atlas image OpenStack Infrastructure Head Node 8 P. Love Atlas account Store Atlas VM image Storage Atlas VM image CMS VM image This model is good if you have an existing Open Stack infrastructure. Helps with OS independence but quiet complex to setup and not straight forward to setup. Networking 8 April 2016

VAC I’m here, any work? Yes, job Install SL Install VAC rpms VAC Factory AKA WN’s Yes, job 8 Layers KVM Hypervisor - KVM Libvirt manager - libvirt Virt manager - VAC Vrsh - installation/management image Install SL Install VAC rpms scp config file 8 VAC layer – when should I start a VM? Do I have an empty slot, 30 seconds, yes start 1 VM, what type, (Different types Atlas, LHCb, GridPP…) Prioritisation defined in /etc/vac.conf (Actually now multiple config files) Uses the site squid. 8 April 2016

viab Configured mainly by web pages Only have to install first node manually (eg from USB stick) All the rest can boot from any of the other installed nodes. Each node runs dhcp, tftp and squid cache to act as an installation server. Everything comes from the web including certificates, but having the private part on the web would be bad so they are encrypted with a passphrase that is stored locally and must be copied to each node. Only have to copy contents of /etc/viabkeys and republish the rpm via the web page. All nodes network boot and always reinstall. Overall simpler setup. April 2016

viab April 2016

Vcycle Vcycle is VAC running on OpenStack – not tested at Oxford Could be useful if central Advanced research computing cluster starts offering an Open Stack setup. April 2016

Plan for GridPP5 CPU GridPP4+ h/w money spent on CPU. April 2016

CPU Upgrade March 2016 Lenovo NeXtScale 25 Nodes each with Dual E5-2640 v3 & 64GB RAM 800 new cores (new total ~2200) April 2016

Plan for GridPP5 CPU GridPP4+ h/w money spent on CPU. This rack is identical to the kit used by Oxford Advanced Research Computing. Initial plan is to plug into our Condor Batch as WNs When staff levels and time allows, we will investigate integrating the rack into the ARC cluster. April 2016

Can be part of a much bigger cluster

Plan for GridPP5 CPU GridPP4+ h/w money spent on CPU. This rack is identical to the kit used by Oxford Advanced Research Computing. Initial plan is to plug into our Condor Batch as WNs When staff levels and time allows, we will investigate integrating the rack into the ARC cluster. Initially stick with what we know ie HTCondor & viab Have to work out how to integrate with a shared cluster later. Decommission out of warranty hardware. Should we move away from the standard grid middleware to viab? April 2016

Reduction in Staff count Recent loss of staff shows We April 2016

Approx. FTE Grid Tasks HTCondor Batch 0.2 DPM Storage 0.15 Monitoring and tuning in the batch system DPM Storage 0.15 Installation, testing, monitoring and maintenance of SPACETOKENS, and h/w Grid Ops & Management Installation and management system (Cobbler, Puppet). GGUS tickets etc Ovirt VM infrastructure 0.1 SL based ovirt system setup to provide VM infrastructure for all service and test nodes. IPv6 testing Perfsonar Early adopter and also ATLAS RIPE probes Security Team membership VAC testing Viab testing 0.05 National Nagios infrastructure GridPP wide essential service VO Nagios for UK GridPP wide service Backup VOMS Server DPM on SL7 testing Openstack Cloud test setup FTE ≈1.05 April 2016

Conclusions A time of streamlining and rationalisation. Can continue as a useful site by sticking to core tasks. Need to make a decision on the storage question. Still a lot of work to do, to investigate integrating with university resources. Will this be possible, will it save time, or allow bursting to greater resources? Possibly benefits of cost savings on h/w maintenance and electricity costs. April 2016