1. Technical Services:
Unavailability Root Causes, Strategy and Limitations
Data and presentation in collaboration with
Ronan LEDRU and Luigi SERIO

2. Jesper Nielsen – Luigi Serio
Introduction
What we do today and how
Systems monitored
Fault recording
Technical Infrastructure Organizational Committee
Analysis of Events in 2016 and comparisons to previous years
Unavailability causes
Shares of the faults
Major contributors and reasons
Electrical perturbations
Strategy to improve the analysis, fault tracking and root causes
Do more with AFT?
Better classification of events ?
Evian Technical Infrastructure
Jesper Nielsen – Luigi Serio

3. Technical Infrastructure Which are the systems we monitor?
Cooling
Electricity
Safety systems
Access system
IT network
Vacuum RF
Power converters
QPS
Collimation
Controls for accelerators
Etc.
Evian Technical Infrastructure
Jesper Nielsen – Luigi Serio

4. Better classification, can we easier match with AFT?
Ventilation
Failures:
- Mechanical
- PLC
- Human error
- Maintenance
- Instrumentation
Demi Water
Cooling & Ventilation
Cooling
Chilled Water
Access
Primary Water
Electricity
“Owner” can be EN-CV
Or another group
Evian Technical Infrastructure
Jesper Nielsen – Luigi Serio

5. AFT Faults – Are these systems really systems?
Some “systems” are groups of “systems”
Some “systems” are very low level
Should it be decided to follow a common standard of what is a system?
Can we introduce “groups” of systems?
Technical
Infrastructure
Cooling
Ventilation
Doors
Evian Technical Infrastructure
Jesper Nielsen – Luigi Serio

6. Major events – Where are they follow up? What is the TIOC?
Equipment groups
Experiments coordinators
Technical Infrastructure
Machine coordination
Monitor, record, analyses events related to the infrastructure systems serving the accelerator complex, the experiments and the computer centre
Recommend consolidations paths which would correct situations originating from the reduced maintenance, non-conformities or weaknesses of the technical infrastructure
Coordinate bigger technical interventions and incidents.
Evian Technical Infrastructure
Jesper Nielsen – Luigi Serio

7. Jesper Nielsen – Luigi Serio
How is a fault recorded?
Accelerator stopped due to a technical fault = Major Event is created
A major event has 1 or more “facility stop” attached.
Evian Technical Infrastructure
Jesper Nielsen – Luigi Serio

8. Are Major Events followed up?
All Major Events are presented during the weekly TIOC meeting
A report is made by each group involved.
The “responsible group”
“Users” impacted
Evian Technical Infrastructure
Jesper Nielsen – Luigi Serio

9. How are the faults distributed?
Evian Technical Infrastructure
Jesper Nielsen – Luigi Serio

10. Jesper Nielsen – Luigi Serio
Faults by failure type
Perturbations: 46%
Equipment faults: 31%
Controls, instrumentation: 12%
Equipment faults: 45%
Controls, instrumentation: 26%
Perturbations: 15%
Evian Technical Infrastructure
Jesper Nielsen – Luigi Serio

11. Fault downtimes in Controls and Instrumentation
PLC 67% less compared to 2015!
Calibration: Water circuits were not calibrated at the best time (opening valves for EPC after calibration)
Software: IT router IP tables update, and BE-CO FrontEnd Crash
Electronics: Power supply for frontend and internal voltage too low in access rack
No communication = Unhappy CRYO operators and very long downtimes…
Evian Technical Infrastructure
Jesper Nielsen – Luigi Serio

12. Jesper Nielsen – Luigi Serio
Fault downtimes in Equipment faults
Many faults caused by other equipment in short circuit
Calibration = Can we better coordinate restarts?
Not suited for usage = Can we integrate reliability in project phase?
Evian Technical Infrastructure
Jesper Nielsen – Luigi Serio

13. More or less downtime compared to 2015?
CRYO less impacted?
Down by 30%
Evian Technical Infrastructure
Jesper Nielsen – Luigi Serio

14. Electrical perturbations
23 hours
35 hours
3 hours
If we filter out all “perturbations” less than 10% in voltage dip that only stop LHC:
We reduce by 30%
Most are assigned to FMCM equipment.
Is this related to beam intensity?
These types of events are seen since second half of 2015
More time in Stable Beams?
Evian Technical Infrastructure
Jesper Nielsen – Luigi Serio

15. Thunderstorms related to number of events seen at CERN?
A general report from French analytics:
2016: 19% above the normal in instability
2015: Within the most stable years in the last 30 years
Evian Technical Infrastructure
Jesper Nielsen – Luigi Serio

16. Jesper Nielsen – Luigi Serio
Conclusion
TIOC coordination has proven very effective
Coordinating events like EDF 400kV intervention, minimized downtime
Best effort on-call in case of emergencies
Good follow-up has been done in several cases this year
Animal protection in HV areas
GSM Network follow up and many improvements
UJ33 flooding: Install racks higher, Release valve modifications, IP67 components.
We want to better class our events and would like to use the AFT tool.
Compare systems on the same level.
Make synchronization possible, if event assigned to Technical Infrastructure we would like a way to attach it to our Major Event
Perturbations
We cannot avoid them, but we can reduce the impact
Less downtime in 2016 compared to 2015
(if we disregard long downtime from weasel incident)
Evian Technical Infrastructure
Jesper Nielsen – Luigi Serio