Site report of Wigner Datacenter HEPiX 2016 Fall, LBNL Domokos Szabo facility manager Wigner Datacenter

Agenda Operation facts Cooling problem Development

Operation facts 231 racks occupied by CERN The power load reached the 40% of planed max (good but not enough ) CERN IT consumption 8.5 GWh in 2016 (est.) Support ticket cca 820 incident tickets (cca 3.2 tickets /workday) cca 50 request tickets (cca 0.2 tickets /workday) 3rd 100G link to CERN in progress

Operation facts PDU replacement campaign 220 of 600 came out from factory with production error Long story (manufacturer vs distributor vs contractor vs us) Hard task Parallel with installation campaign One round has many steps Strict schedule (Bromide contaminated capacitors)

Cooling problem History: see Szabolcs’ presentation of last year (Mostly) caused by inappropriate chillers’ control Too many failed attempts We decided to switch off the Rittal cooling system control Manual control since 2015 fall Problem was mostly a pressure problem, they couldn’t syncronize the pump start/stop We got more arguing and less solution. Even it happened they stated this problem is our fault. Last pressure drop hapenned at 2015, we decided to switch manual control

Cooling problem Solution New, independent data collection system was built Explored the behavior of coolant (esp. flow direction) Tested different chiller setups Looked up best experts in Hungary Design new preliminary regulation(!) The project to implement new regulation started in the last week (will be finished next April) Independent of our BMS. We didn’t know the show d data in BMS were reliable or not. Main suspect was the coolan flow. We realize some switching method between chillers ar turning the flow direction and it dove crazy the chillers. Test CH setup to discover the controling mechanism. There were 2 level control (inside CH (the mostly for self protection) and there upper control to control the CH farm. It made by CH1 controller. So CH 1 was dual role. We didn’t do these alone. Collecting the expereieces we strat to design the new regulation now it’s in designing phase. We kicked off the project last week

Cooling problem – New regulation New regulation of cooling system Goals: PUE optimization Fully automated Fault tolerance (& one „switch” to manual regulation in case of emergency) Already done: Opened chillers and made some changes (guarantee expired) Collected some experience with different chiller setups The Rittal chillers have wrong EEC parameter. For example there are the compressor pre-heating circuit and coolant anti-freeze heating in same breaker. So we separated these two circuit – and switch off the antifreeze circuit. It was no important, because we use 30% glycol solution and it means -15 degrees freeze point. The set point changing and switching on/off chillers gave information about behaving of cooling system and for the time constant of „regulation”

Development – HR Szabolcs (one of the ’founding fathers’) left us in this August Substitute Szabolcs Impossible mission with one person Cut his role into 3 parts Strategic Service management IT development New organization structure 2 people hired

Development – Infrastructure 5th room (originally test room) Operating on max capability Need to improve Electricity: 30kW (N) -> 400kW (2N) Cooling: 100kW (N) -> 440kW (N+1) Security, fire protection … : from zero Still in design phase Will be ready at the Q3 of 2017

Development – Cloud For Wigner RCP For MTA (Hungarian Academy of Sciences) For everybody (in case of R&D&I) Financed by MTA in the first year Self-sustainable

Development – Cloud 2 Openstack based, IaaS cloud Today 1900 vCore 24 500 GB RAM 0.8 PB storage 0,5 PB tape Main step was done: Hired new, dedicated people (2 IT engineers and 3 part-time, trainee engineers)

Thank You! Domokos Szabó Wigner Datacenter szabo.domokos@wigner.mta.hu