Understanding the Universe with help from OpenStack, CERN and Budapest Tim Bell tim.bell@cern.ch 29/05/2013 CERN OpenStack CEE Day

CERN was founded 1954: 12 European States “Science for Peace” Today: 20 Member States ~ 2,300 staff ~ 1,000 other paid personnel > 11,000 users Budget (2013) ~1,000 MCHF Member States: Austria, Belgium, Bulgaria, the Czech Republic, Denmark, Finland, France, Germany, Greece, Hungary, Italy, the Netherlands, Norway, Poland, Portugal, Slovakia, Spain, Sweden, Switzerland and the United Kingdom Candidate for Accession: Romania Associate Members in Pre-Stage to Membership: Israel, Serbia Applicant States for Membership or Associate Membership: Brazil, Cyprus (awaiting ratification), Pakistan, Russia, Slovenia, Turkey, Ukraine Observers to Council: India, Japan, Russia, Turkey, United States of America; European Commission and UNESCO 29/05/2013 CERN OpenStack CEE Day

What are the origins of mass ? 29/05/2013 CERN OpenStack CEE Day

Matter/Anti Matter Symmetric? 29/05/2013 CERN OpenStack CEE Day

Where is 95% of the universe? 29/05/2013 CERN OpenStack CEE Day

29/05/2013 CERN OpenStack CEE Day

Over 1,600 magnets lowered down shafts and cooled to -271 C to become superconducting. Two beam pipes, vacuum 10 times less than the moon 29/05/2013 CERN OpenStack CEE Day

29/05/2013 CERN OpenStack CEE Day

Collisions 29/05/2013 CERN OpenStack CEE Day

July 4, 2012 29/05/2013 CERN OpenStack CEE Day

29/05/2013 CERN OpenStack CEE Day

Initial data reconstruction Data distribution Tier-0 (CERN): Data recording Initial data reconstruction Data distribution Tier-1 (11 centres): Permanent storage Re-processing Analysis Tier-2 (~200 centres): Simulation End-user analysis Data is recorded at CERN and Tier-1s and analysed in the Worldwide LHC Computing Grid In a normal day, the grid provides 100,000 CPU days executing over 2 million jobs The Worldwide LHC Computing grid is used to record and analyse this data. The grid currently runs over 2 million jobs/day, less than 10% of the work is done at CERN. There is an agreed set of protocols for running jobs, data distribution and accounting between all the sites which co-operate in order to support the physicists across the globe. OpenStack London December 2012 Tim Bell, CERN

The CERN Meyrin Data Centre 29/05/2013 CERN OpenStack CEE Day

29/05/2013 CERN OpenStack CEE Day

A Big Data Challenge Nearly 100PB storage today LHC produces up to 35PB/year today Peaks of 25GB/second Data rates double in 2015 Experiments will run for 20 years and data needs to be preserved Exabytes of storage to maintain…. 29/05/2013 CERN OpenStack CEE Day

29/05/2013 CERN OpenStack CEE Day

New Data Centre Approved Data centre in Geneva at the limit of electrical capacity at 3.5MW New centre chosen in Budapest, Hungary Additional 2.7MW of usable power Local on-site support for hardware maintenance and installations 29/05/2013 CERN OpenStack CEE Day

29/05/2013 CERN OpenStack CEE Day

29/05/2013 CERN OpenStack CEE Day

Good News, Bad News Additional data centre in Budapest now online Increasing users of CERN’s facilities and higher computing requirements as data rates increase Staff numbers are fixed, no more people Materials budget decreasing, no more money Legacy tools are high maintenance and brittle How do we maximise our computing resources within these constraints ? 29/05/2013 CERN OpenStack CEE Day

Approach Remodel IT services on Cloud layered models IaaS, PaaS, SaaS Move to commonly used open source tools Focus on strong communities and momentum Implement clouds at scale Aim for 90% infrastructure virtualised Exploit ecosystem solutions rather than writing from scratch 29/05/2013 CERN OpenStack CEE Day

Training for Newcomers Buy the book rather than guru mentoring 29/05/2013 CERN OpenStack CEE Day

Job Opportunities 29/05/2013 CERN OpenStack CEE Day

mcollective, yum Bamboo Puppet AIMS/PXE Foreman JIRA OpenStack Nova git Koji, Mock Yum repo Pulp Active Directory / LDAP Hardware database Lemon / Hadoop / LogStash / Kibana Puppet-DB 29/05/2013

Service Models Pets are given names like pussinboots.cern.ch They are unique, lovingly hand raised and cared for When they get ill, you nurse them back to health Cattle are given numbers like vm0042.cern.ch They are almost identical to other cattle When they get ill, you get another one 29/05/2013 CERN OpenStack CEE Day

CERN Status CERN IT OpenStack Cloud LHC experiment farms Running Folsom around 500 hypervisors on KVM and Hyper-V High availability components using load balancing All Puppet managed to configure OpenStack Over 100 users creating up to 300 new VMs/day Cattle service level only LHC experiment farms CMS currently running 1,300 hypervisors with 50,000 cores ATLAS starting to ramp up to a similar size Other science grid sites moving to private cloud on OpenStack Brookhaven, IN2P3, FutureGrid, NeCTAR, IHEP, … 29/05/2013 CERN OpenStack CEE Day

Block Storage Provider CERN Network Database Block Storage Provider Cinder Network Account mgmt system Compute Scheduler Keystone Nova Microsoft Active Directory 29/05/2013 CERN OpenStack CEE Day Horizon Glance CERN DB on Demand

http://www. eucalyptus http://www.eucalyptus.com/blog/2013/04/02/cy13-q1-community-analysis-%E2%80%94-openstack-vs-opennebula-vs-eucalyptus-vs-cloudstack 29/05/2013 CERN OpenStack CEE Day

29/05/2013 CERN OpenStack CEE Day

Preproduction Service 29/05/2013 CERN OpenStack CEE Day

Linux Security Incident New operational procedures being developed with cloud flexibility 29/05/2013 CERN OpenStack CEE Day

Outlook Track stable Grizzly releases in RedHat EPEL Scaling Up to date but not too close to the leading edge Scaling Expect 15,000 hypervisors, 150,000 VMs by 2015 Manageability Metering, Orchestration with Heat, Bare Metal Functionality Load Balancing, High Availability Storage and Pets 29/05/2013 CERN OpenStack CEE Day

High Energy Physics Clouds Distributed cloud computing using HEP and non-HEP clouds Long-term preservation of software and data of HEP experiments Simplify the management of heterogeneous in-house resources Utilize special computing resources attached to the detectors Use commercial clouds for exceptional computing demands 29/05/2013 CERN OpenStack CEE Day

What have we learnt? Automate everything from the beginning Puppet and Stackforge are a great help Distributions and appliances make getting started much easier Constant rate of change requires a different approach Focus on core technologies and keep up to date Track new projects but don’t adopt too early unless strategic Many of our users are cloud aware Culture changes for legacy application coding and IT services Communities are major motivators But administrators need to engage and adapt rather than re-invent 29/05/2013 CERN OpenStack CEE Day

Conclusions CERN IT is re-engineering to deliver additional capacity to 11,000 physicists within fixed resources Clouds models can simplify current large scale computing infrastructure OpenStack and its ecosystem allows us to meet this challenge and help others through open source 29/05/2013 CERN OpenStack CEE Day

Questions ? 29/05/2013 CERN OpenStack CEE Day