1. Intervention Priority Management This talk will show the CERN priority list, the corresponding check list and the tools used by operators to diagnose a situation and to manage the co-ordination of activities.

2. Intervention Priorities Technical infrastructure operation Big trouble -- Major power outage Problems Solutions Experience, current situation and outlook 14/4/11P. Sollander - ARW Cape Town2

3. Operation CERN BE OP LHC SPS TI … EN EL OP CV OP TE CRG OA OD GS … IT … 14/4/11P. Sollander - ARW Cape Town3

4. It is the mission of the technical infrastructure section to minimize the impact of technical breakdowns on accelerators and other important installations at CERN. The role of the section is to manage corrective maintenance activities and to co- ordinate interventions during major breakdowns, … TI mandate 14/4/11P. Sollander - ARW Cape Town4

5. It is the mission of the technical infrastructure section to minimize the impact of technical breakdowns on accelerators and other important installations at CERN. The role of the section is to manage corrective maintenance activities and to co- ordinate interventions during major breakdowns, … TI mandate 14/4/11P. Sollander - ARW Cape Town5

6. It is the mission of the technical infrastructure section to minimize the impact of technical breakdowns on accelerators and other important installations at CERN. The role of the section is to manage corrective maintenance activities and to co- ordinate interventions during major breakdowns, … TI mandate 14/4/11P. Sollander - ARW Cape Town6

7. It is the mission of the technical infrastructure section to minimize the impact of technical breakdowns on accelerators and other important installations at CERN. The role of the section is to manage corrective maintenance activities and to co- ordinate interventions during major breakdowns, … TI mandate 14/4/11P. Sollander - ARW Cape Town7

8. Operation 24*365 One operator on shift (+1 during shutdown) Technician, 2 year university or equivalent On average, 3 years of experience Supervision of a large number of systems o Electricity, cooling, ventilation, safety detection systems, lifts, computers, … o For all accelerators, experiments and important facilities TI operation 14/4/11P. Sollander - ARW Cape Town8

9. CERN accelerators 14/4/11P. Sollander - ARW Cape Town9

10. Alarms 1’080’087 in 2010 Phone Calls 150-300/day TI First Line 4% Contractor & CERN Interventions Work Orders 6960 in 2010 14/4/11 10 P. Sollander - ARW Cape Town Corrective maintenance

11. Technical Infrastructure desk in the CCC 14/4/11P. Sollander - ARW Cape Town11

12. Big Bad Breakdowns 11/4/11P. Sollander - ARW Cape Town12

13. Thunderstorm 14/4/11P. Sollander - ARW Cape Town13

14. 14/4/11P. Sollander - ARW Cape Town14

15. Panic! 14/4/11P. Sollander - ARW Cape Town15

16. CERN panel for intervention priorities o Set up in 2008 o Representatives from different departments o Mandate to decide on priorities Provide a logically structured list of the facilities at CERN and examine the criticality of the equipment and systems in each facility. Arrange the list in order of priority of intervention as a function of time and as a function of the operational cycle of the accelerator complex and computing facilities. Provide new versions of the priority list as needed to the technical infrastructure operations team. We need a plan! 14/4/11P. Sollander - ARW Cape Town16

17. Improve communication Installation of new Call Centre to handle calls o Queues incoming calls o Voice message explaining situation o Allows VIP list Technical Infrastructure Operation information feed o Up-to-date information on ongoing interventions o Works if networks and servers are up! Information meeting same day o Exchange information between operation, equipment groups and users 14/4/11P. Sollander - ARW Cape Town17

18. Agreed by panel Can change as often as needed Contains high level priorities to check, today: 1.Safety systems 2.Environmental Protection 3.Computer Centre 4.Equipment protection LAr cryo, LKr cryo, RF electronics, … 5.LHC cryogenics 6.ATLAS 7.CMS 8.Accelerator backbone 9.CNGS target areas 10.ALICE 11.LHCb 12.… Priority list 14/4/11P. Sollander - ARW Cape Town18

19. Help; single operator not enough Priority list For each item on the priority list o A means to know current state o Information on what to do in case of problem  Need for a comprehensible CHECK LIST TI operator needs 14/4/11P. Sollander - ARW Cape Town19

20. Checklist 14/4/11P. Sollander - ARW Cape Town20

21. For each item on list o Verification o Actions o Contacts o Help InformationChecklist 14/4/11P. Sollander - ARW Cape Town21

22. High level monitoring 14/4/11P. Sollander - ARW Cape Town22

23. High level monitoring 14/4/11P. Sollander - ARW Cape Town23

24. Experience and Outlook Lists in use since 2008 Priorities change very little (Fortunately) not often used, but very few opportunities for training Priority list useful also for smaller incidents o Avoids overlooking anything, should be used every time Learning by checklist creation o Verification, action, contact: many clarifications obtained. Supervision and Monitoring o Still being implemented o Distills data AND ….. errors Perfectible but necessary, even partly complete Escalation, management, outside communication not included here, dealt with elsewhere. Future vision of an “animated check-list” o Very High Level Monitoring 14/4/11P. Sollander - ARW Cape Town24

25. VHLM Electrical networksSafety systemsEnvironmental ProtectionComputer Centre 14/4/11P. Sollander - ARW Cape Town25

26. Summary TI operation: 1 technician does it all Big trouble: major power outage, 1/year Problems: communication, priorities, tools Solutions: call centre, checklist, high level monitoring Experience: useful, priorities easy to find, checklist difficult to put in place, not enough training possible for the time being 14/4/11P. Sollander - ARW Cape Town26 Questions?

27. 14/4/11P. Sollander - ARW Cape Town27

28. Alarms by year 14/4/11P. Sollander - ARW Cape Town28

29. Phone calls 2010 14/4/11P. Sollander - ARW Cape Town29