1. Site report of Wigner Datacenter
HEPiX 2016 Fall, LBNL
Domokos Szabo
facility manager
Wigner Datacenter

2. Agenda
Operation facts
Cooling problem
Development

3. Operation facts 231 racks occupied by CERN
The power load reached the 40% of planed max (good but not enough )
CERN IT consumption 8.5 GWh in 2016 (est.)
Support ticket
cca 820 incident tickets (cca 3.2 tickets /workday)
cca 50 request tickets (cca 0.2 tickets /workday)
3rd 100G link to CERN
in progress

4. Operation facts PDU replacement campaign
220 of 600 came out from factory with production error
Long story (manufacturer vs distributor vs contractor vs us)
Hard task
Parallel with installation campaign
One round has many steps
Strict schedule
(Bromide contaminated capacitors)

5. Cooling problem History: see Szabolcs’ presentation of last year
(Mostly) caused by inappropriate chillers’ control
Too many failed attempts
We decided to switch off the Rittal cooling system control
Manual control since 2015 fall
Problem was mostly a pressure problem, they couldn’t syncronize the pump start/stop
We got more arguing and less solution. Even it happened they stated this problem is our fault.
Last pressure drop hapenned at 2015, we decided to switch manual control

6. Cooling problem Solution
New, independent data collection system was built
Explored the behavior of coolant (esp. flow direction)
Tested different chiller setups
Looked up best experts in Hungary
Design new preliminary regulation(!)
The project to implement new regulation started in the last week (will be finished next April)
Independent of our BMS. We didn’t know the show d data in BMS were reliable or not.
Main suspect was the coolan flow. We realize some switching method between chillers ar turning the flow direction and it dove crazy the chillers.
Test CH setup to discover the controling mechanism. There were 2 level control (inside CH (the mostly for self protection) and there upper control to control the CH farm. It made by CH1 controller. So CH 1 was dual role.
We didn’t do these alone.
Collecting the expereieces we strat to design the new regulation now it’s in designing phase.
We kicked off the project last week

7. Cooling problem – New regulation
New regulation of cooling system
Goals:
PUE optimization
Fully automated
Fault tolerance (& one „switch” to manual regulation in case of emergency)
Already done:
Opened chillers and made some changes (guarantee expired)
Collected some experience with different chiller setups
The Rittal chillers have wrong EEC parameter. For example there are the compressor pre-heating circuit and coolant anti-freeze heating in same breaker. So we separated these two circuit – and switch off the antifreeze circuit. It was no important, because we use 30% glycol solution and it means -15 degrees freeze point.
The set point changing and switching on/off chillers gave information about behaving of cooling system and for the time constant of „regulation”

8. Development – HR
Szabolcs (one of the ’founding fathers’) left us in this August
Substitute Szabolcs
Impossible mission with one person
Cut his role into 3 parts
Strategic
Service management
IT development
New organization structure
2 people hired

9. Development – Infrastructure
5th room (originally test room)
Operating on max capability
Need to improve
Electricity: 30kW (N) -> 400kW (2N)
Cooling: 100kW (N) -> 440kW (N+1)
Security, fire protection … : from zero
Still in design phase
Will be ready at the Q3 of 2017

10. Development – Cloud For Wigner RCP
For MTA (Hungarian Academy of Sciences)
For everybody (in case of R&D&I)
Financed by MTA in the first year
Self-sustainable

11. Development – Cloud 2 Openstack based, IaaS cloud Today 1900 vCore
GB RAM
0.8 PB storage
0,5 PB tape
Main step was done:
Hired new, dedicated people (2 IT engineers and 3 part-time, trainee engineers)

12. Thank You!
Domokos Szabó
Wigner Datacenter