Towards energy efficient HPC HP Apollo 8000 at Cyfronet Part I Patryk Lasoń, Marek Magryś

ACC Cyfronet AGH-UST established in 1973 part of AGH University of Science and Technology in Krakow, PL provides free computing resources for scientific institutions centre of competence in HPC and Grid Computing IT service management expertise (ITIL, ISO 20k) member of PIONIER operator of Krakow MAN home for Zeus

International projects

PL-Grid Consortium Consortium creation – January 2007 a response to requirements from Polish scientists due to ongoing Grid activities in Europe (EGEE, EGI_DS) Aim: significant extension of amount of computing resources provided to the scientific community (start of the PL-Grid Programme) Development based on: projects funded by the European Regional Development Fund as part of the Innovative Economy Program close international collaboration (EGI, ….) previous projects (5FP, 6FP, 7FP, EDA…) National Network Infrastructure available: Pionier National Project computing resources: Top500 list Polish scientific communities: ~75% highly rated Polish publications in 5 Communities PL-Grid Consortium members: 5 High Performance Computing Polish Centres, representing Communities, coordinated by ACC Cyfronet AGH

PL-Grid infrastructure Polish national IT infrastructure supporting e-Science based upon resources of most powerful academic resource centres compatible and interoperable with European Grid offering grid and cloud computing paradigms coordinated by Cyfronet Benefits for users one infrastructure instead of 5 separate compute centres unified access to software, compute and storage resources non-trivial quality of service Challenges unified monitoring, accounting, security create environment of cooperation rather than competition Federation – the key to success

PLGrid Core project Competence Centre in the Field of Distributed Computing Grid Infrastructures Budget: total ,16 PLN, including funding from the EC : ,99 PLN Duration: – Project Coordinator: Academic Computer Centre CYFRONET AGH The main objective of the project is to support the development of ACC Cyfronet AGH as a specialized competence centre in the field of distributed computing infrastructures, with particular emphasis on grid technologies, cloud computing and infrastructures supporting computations on big data.

PLGrid Core project – services Basic infrastructure services Uniform access to distributed data PaaS Cloud for scientists Applications maintenance environment of MapReduce type End-user services Technologies and environments implementing the Open Science paradigm Computing environment for interactive processing of scientific data Platform for development and execution of large-scale applications organized in a workflow Automatic selection of scientific literature Environment supporting data farming mass computations

Baribal Panda Zeus vSMP Mars Zeus FPGA Zeus GPU Zeus 2013 Platon U3 HPC at Cyfronet

Zeus over 1300 servers HP BL2x220c blades HP BL685c fat nodes (64 cores, 256 GB) HP BL490c vSMP nodes (up to 768 cores, 6 TB) HP SL390s GPGPU (2x,8x) nodes Infiniband QDR (Mellanox+Qlogic) >3 PB of disk storage (Lustre+GPFS) Scientific Linux 6, Torque/Moab

Zeus - statistics 2400 registered users >2000 jobs running simultaneously >22000 jobs per day computing hours in 2013 jobs lasting from minutes to weeks jobs from 1 core to 4000 cores

Cooling Rack 20°C 40°C Hot aisle Cold aisle

Why upgrade? Jobs growing Users hate queuing New users, new requirements Technology moving forward Power bill staying the same

New building

Requirements Petascale system Lowest TCO Energy efficient Dense Good MTBF Hardware: core count memory size network topology storage

Cooling PUE 2.0 CRAC PUE 1.6 CRAC + Precision cooling PUE 1.4 Precision cooling + Chilled water PUE 1.0 ?

Direct Liquid Cooling! Up to 1000x more efficient heat exchange than air Less energy needed to move the coolant Hardware can handle CPUs ~70C memory ~80C Hard to cool 100% of HW with liquid network switches PSUs

MTBF The less movement the better less pumps less fans less HDDs Example pump MTBF: hrs fan MTBF: hrs 1800 node system MTBF: 7 hrs

The topology 576 computing nodes services nodes Service isle storage nodes Computing isle 576 computing nodes Computing isle 576 computing nodes Computing isle Core IB switches

It should count Max jobsize ~10k cores Fastest CPUs, but compatible with old codes Two sockets are enough CPUs, not accelerators Newest memory and more than before Fast interconnect still Infiniband but no need for full CBB fat tree

The hard part Public institution, public tender Strict requirements 1.65 PFLOPS, max servers 128 GB DDR4 per node warm water cooling, no pumps inside nodes infiniband topology compute+cooling, dry-cooler only Criteria: price, power, space

And the winner is… HP Apollo 8000 Most energy efficient The only solution with 100% warm water cooling Least floor space needed Lowest TCO

Even more Apollo Focuses also on ‘1’ in PUE! Power distribution Less fans Detailed monitoring ‘energy to solution’ Safer maintenance Less cables Prefabricated piping Simplified management

System configuration 1.65 PFLOPS (first 30. of the current Top500) 1728 nodes, Intel Haswell E5-2680v cores, per island 216 TB DDR4 RAM PUE ~1.05, 680 kW total power 15 racks, m 2 System ready for undisruptive upgrade Scientific Linux 6 or 7

Prometheus Created human Gave fire to the people Accelerated innovation Defeated Zeus

Deployment plan Contract signed on Installation of the primary loop started on First delivery (service island) expected on Apollo piping should arrive before Christmas Main delivery in January Installation and acceptance in February Production work since Q2 2015

Future plans Benchmarking and Top500 submission Evaluation of Scientific Linux 7 Moving users from the previous system Tuning of applications Energy-aware scheduling First experience presented at HP-CAST 24

More information