6. Shift Leader 1. Duties 2. Golden Rules 3. Cold Start 4. FSM Concepts 5. Novelties Fill Preparation LHCb State Control 8. Operational Procedure 9. Mode Handshakes 10. Injection State 11. HV Status 12. Luminosity Leveling 13. Background Monitoring 14. Protective Beam Dump 15. Timing Monitoring 16. VELO Closure 17. Farm Handling 18. Run Configuration 19. Error and Problems 20. Dead Time 21. Access Handling 22. Level 3 Alarm 23. Power Cut R. Jacobsson 6. Shift Leader “Like driving a car – Maneuvers may feel like routine, but never take the eyes off the road, the instruments, and the mirrors(=DQ)!!” 1

6. Shift Leader 1. Duties 2. Golden Rules 3. Cold Start 4. FSM Concepts 5. Novelties Fill Preparation LHCb State Control 8. Operational Procedure 9. Mode Handshakes 10. Injection State 11. HV Status 12. Luminosity Leveling 13. Background Monitoring 14. Protective Beam Dump 15. Timing Monitoring 16. VELO Closure 17. Farm Handling 18. Run Configuration 19. Error and Problems 20. Dead Time 21. Access Handling 22. Level 3 Alarm 23. Power Cut R. Jacobsson  The LHCb Shift Crew training consists of several courses 1.Introduction 2.Safety 3.Basic Concepts 4.Running LHCb 5.Data Manager 6.Shift Leader  All of these course are subjects to changes  This course is intended to describe the work of the Shift Leader Tasks Procedures Tools  This is a conceptual course not a tutorial, hands-on training is the only way to learn how to run the system  Shift Leaders should also look through the Data Manager slides  Please comment on missing or unclear topics, errors. Don’t hesitate to ask questions 2

6. Shift Leader 1. Duties 2. Golden Rules 3. Cold Start 4. FSM Concepts 5. Novelties Fill Preparation LHCb State Control 8. Operational Procedure 9. Mode Handshakes 10. Injection State 11. HV Status 12. Luminosity Leveling 13. Background Monitoring 14. Protective Beam Dump 15. Timing Monitoring 16. VELO Closure 17. Farm Handling 18. Run Configuration 19. Error and Problems 20. Dead Time 21. Access Handling 22. Level 3 Alarm 23. Power Cut R. Jacobsson  Energy 4 TeV  Intensity ramp up reduced to 7 steps in 2012  3 bunches for Machine Protection validation (below Safe Beam Limit)  2-3 fills and 4-6 hours with 48b, 84b, 264b and 624b (cycle validation)  LHCb recommissioning with beam  3 fills and 20 hours with 840b, 1092b, 1380b (lumi related problems)  At I b ~1.6E11,  ~2.5mm,  *=3m, 840(816) bunches: L~ 3.7E32 /  2  separation  Ultimate 2012 luminosity and rate at (almost) “Day 1”  Requires full attention and INITIAL stable operation/good data quality  “5 min lost in April is equal to 5min lost in October” 1.5 fb -1 is a worthy challenge 3 3.5E32 4E32 1 MHz

6. Shift Leader 1. Duties 2. Golden Rules 3. Cold Start 4. FSM Concepts 5. Novelties Fill Preparation LHCb State Control 8. Operational Procedure 9. Mode Handshakes 10. Injection State 11. HV Status 12. Luminosity Leveling 13. Background Monitoring 14. Protective Beam Dump 15. Timing Monitoring 16. VELO Closure 17. Farm Handling 18. Run Configuration 19. Error and Problems 20. Dead Time 21. Access Handling 22. Level 3 Alarm 23. Power Cut R. Jacobsson  Assist Data Manager Your first task on shift is to get to know the DM and to estimate his experience/quality  Cmp racing map reader! Do the first round of DQ checking together on EVERY FILL! In general, watch his displays and help judging data quality based on your knowledge about the running  Attend Run Meeting (send DM if you are busy)  Follow running during your period of shifts (Logbook, Run News)  Every Start of Shift: Talk to previous shifter and read his Shifter Summary before he leaves Read Daily Shifter Instructions Read Common Problems and Remedies Read logbook of last 24h Make a tour of all screens to make a cerebral snap shot  Operations Twiki Webpages: 4

6. Shift Leader 1. Duties 2. Golden Rules 3. Cold Start 4. FSM Concepts 5. Novelties Fill Preparation LHCb State Control 8. Operational Procedure 9. Mode Handshakes 10. Injection State 11. HV Status 12. Luminosity Leveling 13. Background Monitoring 14. Protective Beam Dump 15. Timing Monitoring 16. VELO Closure 17. Farm Handling 18. Run Configuration 19. Error and Problems 20. Dead Time 21. Access Handling 22. Level 3 Alarm 23. Power Cut R. Jacobsson  Calling piquet/experts 1. Make clear who you are 2. General situation in a handful of words (“We are in physics”/We just dumped”) 3. Summarize main point/problem/incident in one or two sentences 4. In most circumstances, experts will already have a clear list of questions to narrow down the origin 5. Let yourself be guided by experts’ questions and fill in with surrounding relevant information at secondary level  After interventions, invite the experts to put a brief entry with the problem/action/recipe in the central logbook, and also link to local piquet logbook entry if existing  As Shift Leader you will mainly use the following tools LHCb Big Brother Run Control Farm Monitoring Trigger Rate Monitoring Alarm and Log screens Run Database LHC Operational Monitoring Detector Safety System Access Control System Elog 5

6. Shift Leader 1. Duties 2. Golden Rules 3. Cold Start 4. FSM Concepts 5. Novelties Fill Preparation LHCb State Control 8. Operational Procedure 9. Mode Handshakes 10. Injection State 11. HV Status 12. Luminosity Leveling 13. Background Monitoring 14. Protective Beam Dump 15. Timing Monitoring 16. VELO Closure 17. Farm Handling 18. Run Configuration 19. Error and Problems 20. Dead Time 21. Access Handling 22. Level 3 Alarm 23. Power Cut R. Jacobsson  Always keep acquisition together running Take data even without collisions (Destination ‘Local’)  In case a detector is excluded to fix a problem, run the system with the rest to detect other problems EVEN IF YOU THINK DATA WILL NOT BE USEFUL (ex. M1, DQ nuancing)  Before stopping the run while data taking to fix a problem, think twice Is it necessary? Can another pending problem be fixed in parallel? Is the plan clear, who does what and in which order?  If you discover a problem or an anomaly in data, restart run (“fast run change”)  Recovery 3 attempts/10 minutes before calling expert  Don’t panic!  Click calmly – doubts about whether you did the action or not is a potential source of disaster..! Mouse at the centre of object Firm single click Wait until next action Observe all that is happening on the panel  Message Box 6

6. Shift Leader 1. Duties 2. Golden Rules 3. Cold Start 4. FSM Concepts 5. Novelties Fill Preparation LHCb State Control 8. Operational Procedure 9. Mode Handshakes 10. Injection State 11. HV Status 12. Luminosity Leveling 13. Background Monitoring 14. Protective Beam Dump 15. Timing Monitoring 16. VELO Closure 17. Farm Handling 18. Run Configuration 19. Error and Problems 20. Dead Time 21. Access Handling 22. Level 3 Alarm 23. Power Cut R. Jacobsson  Every start of shift 7

6. Shift Leader 1. Duties 2. Golden Rules 3. Cold Start 4. FSM Concepts 5. Novelties Fill Preparation LHCb State Control 8. Operational Procedure 9. Mode Handshakes 10. Injection State 11. HV Status 12. Luminosity Leveling 13. Background Monitoring 14. Protective Beam Dump 15. Timing Monitoring 16. VELO Closure 17. Farm Handling 18. Run Configuration 19. Error and Problems 20. Dead Time 21. Access Handling 22. Level 3 Alarm 23. Power Cut R. Jacobsson  System control tree based on a Finite State Machine (‘FSM Tree’) Control from ‘top’, commands passed down and status up Sub-node status according to rules (“device unit”/”control unit”) Ownership concept: included/excluded Graphics User Interface associated with each sub-node 8

6. Shift Leader 1. Duties 2. Golden Rules 3. Cold Start 4. FSM Concepts 5. Novelties Fill Preparation LHCb State Control 8. Operational Procedure 9. Mode Handshakes 10. Injection State 11. HV Status 12. Luminosity Leveling 13. Background Monitoring 14. Protective Beam Dump 15. Timing Monitoring 16. VELO Closure 17. Farm Handling 18. Run Configuration 19. Error and Problems 20. Dead Time 21. Access Handling 22. Level 3 Alarm 23. Power Cut R. Jacobsson  Make sure everything is included in Run Control FSM Tree and LHCb State Control Tree At Start of Shift and Start of Fill  If something is excluded Investigate and get confirmation from expert 9 Incomplete Some CU excluded Some devices disabled Dead FSM in the tree below

6. Shift Leader 1. Duties 2. Golden Rules 3. Cold Start 4. FSM Concepts 5. Novelties Fill Preparation LHCb State Control 8. Operational Procedure 9. Mode Handshakes 10. Injection State 11. HV Status 12. Luminosity Leveling 13. Background Monitoring 14. Protective Beam Dump 15. Timing Monitoring 16. VELO Closure 17. Farm Handling 18. Run Configuration 19. Error and Problems 20. Dead Time 21. Access Handling 22. Level 3 Alarm 23. Power Cut R. Jacobsson Excluded & Locked Out Unlocked Locked Included By You By Someone else By Nobody 10 Incomplete Some CU excluded Some devices disabled Dead FSM in the tree below Shared By You By Someone else Devices Enabled Disabled  Golden Rule n+1: At beginning of shift, go through system to check what is excluded/disabled  Golden Rule n+2 :Handle padlocks cleanly, lock in global/unlock/lock in local/ask what is going on if padlock is left open, it’s likely the detector has fallen into oblivion

6. Shift Leader 1. Duties 2. Golden Rules 3. Cold Start 4. FSM Concepts 5. Novelties Fill Preparation LHCb State Control 8. Operational Procedure 9. Mode Handshakes 10. Injection State 11. HV Status 12. Luminosity Leveling 13. Background Monitoring 14. Protective Beam Dump 15. Timing Monitoring 16. VELO Closure 17. Farm Handling 18. Run Configuration 19. Error and Problems 20. Dead Time 21. Access Handling 22. Level 3 Alarm 23. Power Cut R. Jacobsson  Deferred HLT trigger  End-Of-Fill calibrations  Data taking until DUMP, VELO IN until last proton leaves the machine  Tilted crossing plane  Luminosity Leveling revised  Source of trigger dead-time and how it is counted  Automatic recoveries (HV, and DAQ)  Autopilot ON  High Voltage status and automatic recovery of channel trips  Event Filter Farm (HLT Farm) Majority mechanism to Start Run On the fly removal/inclusion of entire subfarm or individual nodes HLT task automatic recovery  Ignore / Include feature on the fly of Monitoring, Reconstruction, Calibration 11

6. Shift Leader 1. Duties 2. Golden Rules 3. Cold Start 4. FSM Concepts 5. Novelties Fill Preparation LHCb State Control 8. Operational Procedure 9. Mode Handshakes 10. Injection State 11. HV Status 12. Luminosity Leveling 13. Background Monitoring 14. Protective Beam Dump 15. Timing Monitoring 16. VELO Closure 17. Farm Handling 18. Run Configuration 19. Error and Problems 20. Dead Time 21. Access Handling 22. Level 3 Alarm 23. Power Cut R. Jacobsson  Farm Log Viewer Less messages, hopefully… Message level and filtering  Run Configuration DCS must be READY! Partition / Activity / Trigger Configuration / HLT+L0 TCK / Internal ODIN Triggers Data Type and Data Destination Fast Run Change  Level 3 Alarm  Power Cut  Access Handling and Access Application documented carefully on Shifter Instructions  Homework: Try and explore all these features in interfill time 12

6. Shift Leader 1. Duties 2. Golden Rules 3. Cold Start 4. FSM Concepts 5. Novelties Fill Preparation LHCb State Control 8. Operational Procedure 9. Mode Handshakes 10. Injection State 11. HV Status 12. Luminosity Leveling 13. Background Monitoring 14. Protective Beam Dump 15. Timing Monitoring 16. VELO Closure 17. Farm Handling 18. Run Configuration 19. Error and Problems 20. Dead Time 21. Access Handling 22. Level 3 Alarm 23. Power Cut R. Jacobsson Preparing LHCb for Data Taking 13

6. Shift Leader 1. Duties 2. Golden Rules 3. Cold Start 4. FSM Concepts 5. Novelties Fill Preparation LHCb State Control 8. Operational Procedure 9. Mode Handshakes 10. Injection State 11. HV Status 12. Luminosity Leveling 13. Background Monitoring 14. Protective Beam Dump 15. Timing Monitoring 16. VELO Closure 17. Farm Handling 18. Run Configuration 19. Error and Problems 20. Dead Time 21. Access Handling 22. Level 3 Alarm 23. Power Cut R. Jacobsson 14 Run Control LHCb State Control

6. Shift Leader 1. Duties 2. Golden Rules 3. Cold Start 4. FSM Concepts 5. Novelties Fill Preparation LHCb State Control 8. Operational Procedure 9. Mode Handshakes 10. Injection State 11. HV Status 12. Luminosity Leveling 13. Background Monitoring 14. Protective Beam Dump 15. Timing Monitoring 16. VELO Closure 17. Farm Handling 18. Run Configuration 19. Error and Problems 20. Dead Time 21. Access Handling 22. Level 3 Alarm 23. Power Cut R. Jacobsson 15 LHC State LHCb State Handshake HV/LV State VELO Closure BCM Magnet Global clock HV Configuration

6. Shift Leader 1. Duties 2. Golden Rules 3. Cold Start 4. FSM Concepts 5. Novelties Fill Preparation LHCb State Control 8. Operational Procedure 9. Mode Handshakes 10. Injection State 11. HV Status 12. Luminosity Leveling 13. Background Monitoring 14. Protective Beam Dump 15. Timing Monitoring 16. VELO Closure 17. Farm Handling 18. Run Configuration 19. Error and Problems 20. Dead Time 21. Access Handling 22. Level 3 Alarm 23. Power Cut R. Jacobsson Fill sequence for physics fill in LHCb language: NO_BEAM (Injection Permit = FALSE, Any state of LHCb, Internal clock) INJECTION (Injection Permit = TRUE, VELO out) RAMP (Injection Permit = FALSE) PHYS_ADJUST PHYSICS (VELO in) (ADJUST) ( VELO out) DUMP EOF (Internal Clock, Calibrations)  All state changes require confirmation on LHCb State Control from the shifter! 16 Handshake for Injection OR TI8_Setup Beam Dump Warning (Only when directly from PHYSICS) Handshake for Adjust (LHC: FLAT TOP  Reset of entire DAQ (inclusion of “ignored FE and switch off data generators))

6. Shift Leader 1. Duties 2. Golden Rules 3. Cold Start 4. FSM Concepts 5. Novelties Fill Preparation LHCb State Control 8. Operational Procedure 9. Mode Handshakes 10. Injection State 11. HV Status 12. Luminosity Leveling 13. Background Monitoring 14. Protective Beam Dump 15. Timing Monitoring 16. VELO Closure 17. Farm Handling 18. Run Configuration 19. Error and Problems 20. Dead Time 21. Access Handling 22. Level 3 Alarm 23. Power Cut R. Jacobsson 17 PROBLEM WARNING READY PREPARE STANDBY VETO READY OK LHC: NO_BEAM INJECTION LHCb: STANDBY Confirm VETO ECS GET READY READY  5 minutes timeout Confirm RAMP Voice assistant warns repeatedly about the timeout after 3min

6. Shift Leader 1. Duties 2. Golden Rules 3. Cold Start 4. FSM Concepts 5. Novelties Fill Preparation LHCb State Control 8. Operational Procedure 9. Mode Handshakes 10. Injection State 11. HV Status 12. Luminosity Leveling 13. Background Monitoring 14. Protective Beam Dump 15. Timing Monitoring 16. VELO Closure 17. Farm Handling 18. Run Configuration 19. Error and Problems 20. Dead Time 21. Access Handling 22. Level 3 Alarm 23. Power Cut R. Jacobsson  TI8_SETUP handshake: Allow LHC to work with transfer line for beam 2 (TI8) Only requires injection permit for beam 2 ONLY from us TED beam stopper is IN (prevents beam from entering LHC) LHCb State goes to INJECTION  INJECTION and ADJUST handshake no changes  5 min automatic timeout in place counting from LHC WARNING  If LHCb is not yet ready, we publish PROBLEM automatically  Voice warns before  React swiftly to handshake!  If in PROBLEM 1.Slow prepare  Once the subsystem is ready PROBLEM switches to READY automatically 2.Subdetector in error  As soon as error is solved or excluded (careful!!) PROBLEM switches to READY automatically  You might be contacted by LHC for explanation and time estimate  BEAM_DUMP handshake (Fully automatic in LHCb) LHC issues WARNING 5min before dumping beam unconditionally LHCb continues data taking until the beam dump (after first two weeks) At beam dump, automatically stops run, moves VELO out, and ramps down everything which should NOT participate to End-Of-Fill calibration. A lot of mechanism and systems now depend on the LHC internal state  Always make sure you confirm from the big box! (Don’t use cancel!) 18

6. Shift Leader 1. Duties 2. Golden Rules 3. Cold Start 4. FSM Concepts 5. Novelties Fill Preparation LHCb State Control 8. Operational Procedure 9. Mode Handshakes 10. Injection State 11. HV Status 12. Luminosity Leveling 13. Background Monitoring 14. Protective Beam Dump 15. Timing Monitoring 16. VELO Closure 17. Farm Handling 18. Run Configuration 19. Error and Problems 20. Dead Time 21. Access Handling 22. Level 3 Alarm 23. Power Cut R. Jacobsson 19 Status of all LHCb-LHC communication (on all LHC/LHCb monitoring displays)

6. Shift Leader 1. Duties 2. Golden Rules 3. Cold Start 4. FSM Concepts 5. Novelties Fill Preparation LHCb State Control 8. Operational Procedure 9. Mode Handshakes 10. Injection State 11. HV Status 12. Luminosity Leveling 13. Background Monitoring 14. Protective Beam Dump 15. Timing Monitoring 16. VELO Closure 17. Farm Handling 18. Run Configuration 19. Error and Problems 20. Dead Time 21. Access Handling 22. Level 3 Alarm 23. Power Cut R. Jacobsson 20

6. Shift Leader 1. Duties 2. Golden Rules 3. Cold Start 4. FSM Concepts 5. Novelties Fill Preparation LHCb State Control 8. Operational Procedure 9. Mode Handshakes 10. Injection State 11. HV Status 12. Luminosity Leveling 13. Background Monitoring 14. Protective Beam Dump 15. Timing Monitoring 16. VELO Closure 17. Farm Handling 18. Run Configuration 19. Error and Problems 20. Dead Time 21. Access Handling 22. Level 3 Alarm 23. Power Cut R. Jacobsson 21

6. Shift Leader 1. Duties 2. Golden Rules 3. Cold Start 4. FSM Concepts 5. Novelties Fill Preparation LHCb State Control 8. Operational Procedure 9. Mode Handshakes 10. Injection State 11. HV Status 12. Luminosity Leveling 13. Background Monitoring 14. Protective Beam Dump 15. Timing Monitoring 16. VELO Closure 17. Farm Handling 18. Run Configuration 19. Error and Problems 20. Dead Time 21. Access Handling 22. Level 3 Alarm 23. Power Cut R. Jacobsson 22

6. Shift Leader 1. Duties 2. Golden Rules 3. Cold Start 4. FSM Concepts 5. Novelties Fill Preparation LHCb State Control 8. Operational Procedure 9. Mode Handshakes 10. Injection State 11. HV Status 12. Luminosity Leveling 13. Background Monitoring 14. Protective Beam Dump 15. Timing Monitoring 16. VELO Closure 17. Farm Handling 18. Run Configuration 19. Error and Problems 20. Dead Time 21. Access Handling 22. Level 3 Alarm 23. Power Cut R. Jacobsson 23

6. Shift Leader 1. Duties 2. Golden Rules 3. Cold Start 4. FSM Concepts 5. Novelties Fill Preparation LHCb State Control 8. Operational Procedure 9. Mode Handshakes 10. Injection State 11. HV Status 12. Luminosity Leveling 13. Background Monitoring 14. Protective Beam Dump 15. Timing Monitoring 16. VELO Closure 17. Farm Handling 18. Run Configuration 19. Error and Problems 20. Dead Time 21. Access Handling 22. Level 3 Alarm 23. Power Cut R. Jacobsson  Timeout after 5 min! Get the system ready and PROBLEM moves to READY automatically 24 Timer will be added

6. Shift Leader 1. Duties 2. Golden Rules 3. Cold Start 4. FSM Concepts 5. Novelties Fill Preparation LHCb State Control 8. Operational Procedure 9. Mode Handshakes 10. Injection State 11. HV Status 12. Luminosity Leveling 13. Background Monitoring 14. Protective Beam Dump 15. Timing Monitoring 16. VELO Closure 17. Farm Handling 18. Run Configuration 19. Error and Problems 20. Dead Time 21. Access Handling 22. Level 3 Alarm 23. Power Cut R. Jacobsson  Following e.g. CANCEL of request to change LHCb state or ERROR 25

6. Shift Leader 1. Duties 2. Golden Rules 3. Cold Start 4. FSM Concepts 5. Novelties Fill Preparation LHCb State Control 8. Operational Procedure 9. Mode Handshakes 10. Injection State 11. HV Status 12. Luminosity Leveling 13. Background Monitoring 14. Protective Beam Dump 15. Timing Monitoring 16. VELO Closure 17. Farm Handling 18. Run Configuration 19. Error and Problems 20. Dead Time 21. Access Handling 22. Level 3 Alarm 23. Power Cut R. Jacobsson 26 Injection is automatically VETOed when Beam Mode leaves INJECTION* If not yet terminated

6. Shift Leader 1. Duties 2. Golden Rules 3. Cold Start 4. FSM Concepts 5. Novelties Fill Preparation LHCb State Control 8. Operational Procedure 9. Mode Handshakes 10. Injection State 11. HV Status 12. Luminosity Leveling 13. Background Monitoring 14. Protective Beam Dump 15. Timing Monitoring 16. VELO Closure 17. Farm Handling 18. Run Configuration 19. Error and Problems 20. Dead Time 21. Access Handling 22. Level 3 Alarm 23. Power Cut R. Jacobsson 27 Ramp progress

6. Shift Leader 1. Duties 2. Golden Rules 3. Cold Start 4. FSM Concepts 5. Novelties Fill Preparation LHCb State Control 8. Operational Procedure 9. Mode Handshakes 10. Injection State 11. HV Status 12. Luminosity Leveling 13. Background Monitoring 14. Protective Beam Dump 15. Timing Monitoring 16. VELO Closure 17. Farm Handling 18. Run Configuration 19. Error and Problems 20. Dead Time 21. Access Handling 22. Level 3 Alarm 23. Power Cut R. Jacobsson 28

6. Shift Leader 1. Duties 2. Golden Rules 3. Cold Start 4. FSM Concepts 5. Novelties Fill Preparation LHCb State Control 8. Operational Procedure 9. Mode Handshakes 10. Injection State 11. HV Status 12. Luminosity Leveling 13. Background Monitoring 14. Protective Beam Dump 15. Timing Monitoring 16. VELO Closure 17. Farm Handling 18. Run Configuration 19. Error and Problems 20. Dead Time 21. Access Handling 22. Level 3 Alarm 23. Power Cut R. Jacobsson  Two activities to take care of: 1. Process the deferred HLT Trigger 2. End of Fill calibration  This will be manual during the first weeks of the year. 29

6. Shift Leader 1. Duties 2. Golden Rules 3. Cold Start 4. FSM Concepts 5. Novelties Fill Preparation LHCb State Control 8. Operational Procedure 9. Mode Handshakes 10. Injection State 11. HV Status 12. Luminosity Leveling 13. Background Monitoring 14. Protective Beam Dump 15. Timing Monitoring 16. VELO Closure 17. Farm Handling 18. Run Configuration 19. Error and Problems 20. Dead Time 21. Access Handling 22. Level 3 Alarm 23. Power Cut R. Jacobsson  Idea: 1. Farm is essentially idling during interfill time 2. During fill: Overrun farm processing capacity and store rate overflow on local disk 3. During interfill: Process overflow and close runs Each farm nodes has 200 Gbyte of disk space allocated to L0 Events  Space for >5E9 events  Example: 10% over-commitment  15 – 20h, i.e 1.5 – 2h processing in interfill  Instead of throttling the L0 rate via the Event Request scheme, write overflow L0 events temporarily on local disk (“OverflowWriter”)  Special “Deferred HLT Trigger” partition processes the events during interfill  Thus, Event Requests as long as disk space is available  If disks get full, system will fall-back to throttle L0 rate by “less Event Requests” as 2011  Two alternatives to deal with deadtime  Luminosity or alternative trigger configuration  If not all processing done by next fill, simply start with less buffer and process in next interfill Interfill processing performed on all subfarms but one  One subfarm sufficient for End-of-Fill calibration, and global and local tests during interfill time 30

6. Shift Leader 1. Duties 2. Golden Rules 3. Cold Start 4. FSM Concepts 5. Novelties Fill Preparation LHCb State Control 8. Operational Procedure 9. Mode Handshakes 10. Injection State 11. HV Status 12. Luminosity Leveling 13. Background Monitoring 14. Protective Beam Dump 15. Timing Monitoring 16. VELO Closure 17. Farm Handling 18. Run Configuration 19. Error and Problems 20. Dead Time 21. Access Handling 22. Level 3 Alarm 23. Power Cut R. Jacobsson 31 Deferred HLT Overflow Storage...  Rate regulated system at many levels: “trigger throttling” by TFC

6. Shift Leader 1. Duties 2. Golden Rules 3. Cold Start 4. FSM Concepts 5. Novelties Fill Preparation LHCb State Control 8. Operational Procedure 9. Mode Handshakes 10. Injection State 11. HV Status 12. Luminosity Leveling 13. Background Monitoring 14. Protective Beam Dump 15. Timing Monitoring 16. VELO Closure 17. Farm Handling 18. Run Configuration 19. Error and Problems 20. Dead Time 21. Access Handling 22. Level 3 Alarm 23. Power Cut R. Jacobsson  In LHCb (before starting the run) Tick the Defer HLT box When deferring Runs/Files will start increasing 32

6. Shift Leader 1. Duties 2. Golden Rules 3. Cold Start 4. FSM Concepts 5. Novelties Fill Preparation LHCb State Control 8. Operational Procedure 9. Mode Handshakes 10. Injection State 11. HV Status 12. Luminosity Leveling 13. Background Monitoring 14. Protective Beam Dump 15. Timing Monitoring 16. VELO Closure 17. Farm Handling 18. Run Configuration 19. Error and Problems 20. Dead Time 21. Access Handling 22. Level 3 Alarm 23. Power Cut R. Jacobsson  At End-of-fill Choose “Start” Deferred HLT  Stops and Deallocates LHCb  Allocates and Starts LHCb_Deferred 33

6. Shift Leader 1. Duties 2. Golden Rules 3. Cold Start 4. FSM Concepts 5. Novelties Fill Preparation LHCb State Control 8. Operational Procedure 9. Mode Handshakes 10. Injection State 11. HV Status 12. Luminosity Leveling 13. Background Monitoring 14. Protective Beam Dump 15. Timing Monitoring 16. VELO Closure 17. Farm Handling 18. Run Configuration 19. Error and Problems 20. Dead Time 21. Access Handling 22. Level 3 Alarm 23. Power Cut R. Jacobsson  Processing the Deferred HLT LHCb_Deferred should be RUNNING Runs/Files should decrease Processing bar will increase LHCb can be used on parallel (but only 10 A farms available) 34

6. Shift Leader 1. Duties 2. Golden Rules 3. Cold Start 4. FSM Concepts 5. Novelties Fill Preparation LHCb State Control 8. Operational Procedure 9. Mode Handshakes 10. Injection State 11. HV Status 12. Luminosity Leveling 13. Background Monitoring 14. Protective Beam Dump 15. Timing Monitoring 16. VELO Closure 17. Farm Handling 18. Run Configuration 19. Error and Problems 20. Dead Time 21. Access Handling 22. Level 3 Alarm 23. Power Cut R. Jacobsson  When Finished (Runs/Files = 0) Choose “Stop” Deferred HLT  Stops and Deallocates LHCb_Deferred  At the moment also Stops and Deallocates LHCb  Can then Re-Allocate LHCb to take all farms back 35

6. Shift Leader 1. Duties 2. Golden Rules 3. Cold Start 4. FSM Concepts 5. Novelties Fill Preparation LHCb State Control 8. Operational Procedure 9. Mode Handshakes 10. Injection State 11. HV Status 12. Luminosity Leveling 13. Background Monitoring 14. Protective Beam Dump 15. Timing Monitoring 16. VELO Closure 17. Farm Handling 18. Run Configuration 19. Error and Problems 20. Dead Time 21. Access Handling 22. Level 3 Alarm 23. Power Cut R. Jacobsson  LHCb_Deferred is like any other partition If in ERROR, Recover, etc. 36

6. Shift Leader 1. Duties 2. Golden Rules 3. Cold Start 4. FSM Concepts 5. Novelties Fill Preparation LHCb State Control 8. Operational Procedure 9. Mode Handshakes 10. Injection State 11. HV Status 12. Luminosity Leveling 13. Background Monitoring 14. Protective Beam Dump 15. Timing Monitoring 16. VELO Closure 17. Farm Handling 18. Run Configuration 19. Error and Problems 20. Dead Time 21. Access Handling 22. Level 3 Alarm 23. Power Cut R. Jacobsson  VELO and ST would like to take some special runs after each fill in global  At End-of-fill (after starting the Deferred HLT processing) Apply the End-of-fill Activity (“EOF_CALIB”) Switch_On the LHCb autopilot  When both the End-of-fill and the Deferred HLT Processing are done: “Stop” the Deferred HLT Select the Physics Activity Switch_On LHCb Autopilot  Deferred HLT procedure and EOF calibration procedure to be automated after some experience Your feedback is very welcome 37

6. Shift Leader 1. Duties 2. Golden Rules 3. Cold Start 4. FSM Concepts 5. Novelties Fill Preparation LHCb State Control 8. Operational Procedure 9. Mode Handshakes 10. Injection State 11. HV Status 12. Luminosity Leveling 13. Background Monitoring 14. Protective Beam Dump 15. Timing Monitoring 16. VELO Closure 17. Farm Handling 18. Run Configuration 19. Error and Problems 20. Dead Time 21. Access Handling 22. Level 3 Alarm 23. Power Cut R. Jacobsson  Beam commissioning, beam studies etc  Exception to be handled according to planning  separate instructions INJECTION state safe for most activities Use of MD may change in future  In MD, only one single handshake for entire duration of MD (injection) Stay on Internal Clock General rule, if already in MD stay in MD at each injection handshake until next physics fill 38

6. Shift Leader 1. Duties 2. Golden Rules 3. Cold Start 4. FSM Concepts 5. Novelties Fill Preparation LHCb State Control 8. Operational Procedure 9. Mode Handshakes 10. Injection State 11. HV Status 12. Luminosity Leveling 13. Background Monitoring 14. Protective Beam Dump 15. Timing Monitoring 16. VELO Closure 17. Farm Handling 18. Run Configuration 19. Error and Problems 20. Dead Time 21. Access Handling 22. Level 3 Alarm 23. Power Cut R. Jacobsson  The LHCb clock is switched automatically to the machine clocks when LHC Beam Mode changes to INJECTION. As a brutal change in the clock has consequences for electronics, it is then advisable NOT TO RUN at this time. Currently there is no automatic switch to internal clock  Manually SET_INTERNAL/SET_EXTERNAL may be done on LHC State Control Panel  Stop the run before 39

6. Shift Leader 1. Duties 2. Golden Rules 3. Cold Start 4. FSM Concepts 5. Novelties Fill Preparation LHCb State Control 8. Operational Procedure 9. Mode Handshakes 10. Injection State 11. HV Status 12. Luminosity Leveling 13. Background Monitoring 14. Protective Beam Dump 15. Timing Monitoring 16. VELO Closure 17. Farm Handling 18. Run Configuration 19. Error and Problems 20. Dead Time 21. Access Handling 22. Level 3 Alarm 23. Power Cut R. Jacobsson  The LHC clock may have jumps due to resynchronization between SPS and LHC. LHC Clocks only guaranteed to be stable INJECTION  DUMP 40

6. Shift Leader 1. Duties 2. Golden Rules 3. Cold Start 4. FSM Concepts 5. Novelties Fill Preparation LHCb State Control 8. Operational Procedure 9. Mode Handshakes 10. Injection State 11. HV Status 12. Luminosity Leveling 13. Background Monitoring 14. Protective Beam Dump 15. Timing Monitoring 16. VELO Closure 17. Farm Handling 18. Run Configuration 19. Error and Problems 20. Dead Time 21. Access Handling 22. Level 3 Alarm 23. Power Cut R. Jacobsson  HV Status = % Operational Channels indicated on LHCb Big Brother  According to preconfigured thresholds, the HV status drives the Detector Status bits in the ODIN data bank  MUON has automatic recovery mechanism of tripping channels 41

6. Shift Leader 1. Duties 2. Golden Rules 3. Cold Start 4. FSM Concepts 5. Novelties Fill Preparation LHCb State Control 8. Operational Procedure 9. Mode Handshakes 10. Injection State 11. HV Status 12. Luminosity Leveling 13. Background Monitoring 14. Protective Beam Dump 15. Timing Monitoring 16. VELO Closure 17. Farm Handling 18. Run Configuration 19. Error and Problems 20. Dead Time 21. Access Handling 22. Level 3 Alarm 23. Power Cut R. Jacobsson  Same crossing angle at IP8 with both polarities  Remove external horizontal crossing angle, apply vertical angle  Crossing plane tilted! 42

6. Shift Leader 1. Duties 2. Golden Rules 3. Cold Start 4. FSM Concepts 5. Novelties Fill Preparation LHCb State Control 8. Operational Procedure 9. Mode Handshakes 10. Injection State 11. HV Status 12. Luminosity Leveling 13. Background Monitoring 14. Protective Beam Dump 15. Timing Monitoring 16. VELO Closure 17. Farm Handling 18. Run Configuration 19. Error and Problems 20. Dead Time 21. Access Handling 22. Level 3 Alarm 23. Power Cut R. Jacobsson  Optimum luminosity is a function of several parameters monitored real-time Full event readout rate (max 1 MHz) Average number of interactions per crossing (max 2.5) Max readout network through-put (max 70 GB/s) Physics trigger overall dead-time ( <5-6%) High Level Trigger output rate to storage (max ~4-5 kHz) Instantaneous luminosity ( max ?  = Detector/operational stability, still exploring) Typical bunch parameters: intensity=1.2E11, emittance=2.5um, beta*=3m   =3 ! Also to be used for special runs requiring low rates  Behaviour and limits of parameters depend on experimental conditions (background/beam-gas/…) and system status  We drive the LHCb luminosity via special application and communication with LHC “Continuous” leveling by vertical offset between beams, e.g. 43

6. Shift Leader 1. Duties 2. Golden Rules 3. Cold Start 4. FSM Concepts 5. Novelties Fill Preparation LHCb State Control 8. Operational Procedure 9. Mode Handshakes 10. Injection State 11. HV Status 12. Luminosity Leveling 13. Background Monitoring 14. Protective Beam Dump 15. Timing Monitoring 16. VELO Closure 17. Farm Handling 18. Run Configuration 19. Error and Problems 20. Dead Time 21. Access Handling 22. Level 3 Alarm 23. Power Cut R. Jacobsson  SQUEEZE  ADJUST 1. Collapse separation bumps in Atlas/CMS/Alice (65 s) 2. Tilt LHCb in parallel (180 s  adds 1.5 – 2min) 3. Arrive after tilt to a separation in leveling plane of ~3  L < 1.5x10 32 cm -2 s -1 )  Watch luminosity carefully!  STABLE BEAMS Full optimization of Atlas/CMS Optimization of LHCb in crossing plane in parallel Luminosity leveling function in LHCb gets automatically triggered after the optimization Luminosity ramp starts off Closing VELO in parallel  Luminosity ramp starts before VELO is closed, normally  Shorter plateaus which means faster ramp  Coast Lumi Leveling requested when current luminosity and target different by > 3%  Automatic emergency trigger implemented including button in LHCb & CCC Faster reaction in case of excessive luminosity  After run-in of new procedure, more specific SL instructions 44

6. Shift Leader 1. Duties 2. Golden Rules 3. Cold Start 4. FSM Concepts 5. Novelties Fill Preparation LHCb State Control 8. Operational Procedure 9. Mode Handshakes 10. Injection State 11. HV Status 12. Luminosity Leveling 13. Background Monitoring 14. Protective Beam Dump 15. Timing Monitoring 16. VELO Closure 17. Farm Handling 18. Run Configuration 19. Error and Problems 20. Dead Time 21. Access Handling 22. Level 3 Alarm 23. Power Cut R. Jacobsson 45 Mu per bunch Specific luminosity per bunch Rates of L0 and luminosity counters Luminosity and pileup trends Vertical separation trend

6. Shift Leader 1. Duties 2. Golden Rules 3. Cold Start 4. FSM Concepts 5. Novelties Fill Preparation LHCb State Control 8. Operational Procedure 9. Mode Handshakes 10. Injection State 11. HV Status 12. Luminosity Leveling 13. Background Monitoring 14. Protective Beam Dump 15. Timing Monitoring 16. VELO Closure 17. Farm Handling 18. Run Configuration 19. Error and Problems 20. Dead Time 21. Access Handling 22. Level 3 Alarm 23. Power Cut R. Jacobsson 46

6. Shift Leader 1. Duties 2. Golden Rules 3. Cold Start 4. FSM Concepts 5. Novelties Fill Preparation LHCb State Control 8. Operational Procedure 9. Mode Handshakes 10. Injection State 11. HV Status 12. Luminosity Leveling 13. Background Monitoring 14. Protective Beam Dump 15. Timing Monitoring 16. VELO Closure 17. Farm Handling 18. Run Configuration 19. Error and Problems 20. Dead Time 21. Access Handling 22. Level 3 Alarm 23. Power Cut R. Jacobsson  High radiation levels should be avoided. However, some radiation is normal, and is just due to proton-proton collisions, beam gas and halo particles. High radiation levels for a long enough time can trigger a beam dump to avoid serious damages to the detector. Background is the normal life around an accelerator but it can be unacceptably high. This affects the quality of the data. If high, it can decrease the expected life of the detectors  Guidelines for BCM: Injection: Beam losses may reach 30% of dump threshold on BCM  May happen a few times but should not be systematically above  inform LHC Circulating beam: Beam losses up to 10% is already an anomaly  Up to 2% in 2010 purely due to collisions  Single spike OK, repeatedly or constant, inform LHC In case of doubt call Run Chief or Run Coordinator  BLS and RMS guidelines to come 47

6. Shift Leader 1. Duties 2. Golden Rules 3. Cold Start 4. FSM Concepts 5. Novelties Fill Preparation LHCb State Control 8. Operational Procedure 9. Mode Handshakes 10. Injection State 11. HV Status 12. Luminosity Leveling 13. Background Monitoring 14. Protective Beam Dump 15. Timing Monitoring 16. VELO Closure 17. Farm Handling 18. Run Configuration 19. Error and Problems 20. Dead Time 21. Access Handling 22. Level 3 Alarm 23. Power Cut R. Jacobsson 48 BCM Dumped BCM Injection Mode

6. Shift Leader 1. Duties 2. Golden Rules 3. Cold Start 4. FSM Concepts 5. Novelties Fill Preparation LHCb State Control 8. Operational Procedure 9. Mode Handshakes 10. Injection State 11. HV Status 12. Luminosity Leveling 13. Background Monitoring 14. Protective Beam Dump 15. Timing Monitoring 16. VELO Closure 17. Farm Handling 18. Run Configuration 19. Error and Problems 20. Dead Time 21. Access Handling 22. Level 3 Alarm 23. Power Cut R. Jacobsson 49 Post Mortem Trigger

6. Shift Leader 1. Duties 2. Golden Rules 3. Cold Start 4. FSM Concepts 5. Novelties Fill Preparation LHCb State Control 8. Operational Procedure 9. Mode Handshakes 10. Injection State 11. HV Status 12. Luminosity Leveling 13. Background Monitoring 14. Protective Beam Dump 15. Timing Monitoring 16. VELO Closure 17. Farm Handling 18. Run Configuration 19. Error and Problems 20. Dead Time 21. Access Handling 22. Level 3 Alarm 23. Power Cut R. Jacobsson  A protective dump is always followed by transmission of “Post Mortem Trigger” Requests all systems to readout history buffers and send to LHC for post mortem analysis  Beam Condition Monitor Rearms automatically if dump was not by itself  FSM state changes ABORT_PMT  PM_READY  READY Dump by BCM: Spike exceeding one of the dump thresholds  Not necessarily over 1000 per mille on display depending on which logic triggered  BCM trend on Background Monitor Display blinks “BCM Dumped” in red  FSM stays in PM_READY waiting for manual ‘Rearm’ command  Call CCC to tell them we have probably dumped and that we are investigating  Call Run Chief or Run Coordinator before rearming 50

6. Shift Leader 1. Duties 2. Golden Rules 3. Cold Start 4. FSM Concepts 5. Novelties Fill Preparation LHCb State Control 8. Operational Procedure 9. Mode Handshakes 10. Injection State 11. HV Status 12. Luminosity Leveling 13. Background Monitoring 14. Protective Beam Dump 15. Timing Monitoring 16. VELO Closure 17. Farm Handling 18. Run Configuration 19. Error and Problems 20. Dead Time 21. Access Handling 22. Level 3 Alarm 23. Power Cut R. Jacobsson  Check that filling scheme is OK and no ghost bunches  Phase of Beam 1 must stay within +/- 0.5ns  DeltaT (difference in arrival time IP) within 100ps 51 Acceleration

6. Shift Leader 1. Duties 2. Golden Rules 3. Cold Start 4. FSM Concepts 5. Novelties Fill Preparation LHCb State Control 8. Operational Procedure 9. Mode Handshakes 10. Injection State 11. HV Status 12. Luminosity Leveling 13. Background Monitoring 14. Protective Beam Dump 15. Timing Monitoring 16. VELO Closure 17. Farm Handling 18. Run Configuration 19. Error and Problems 20. Dead Time 21. Access Handling 22. Level 3 Alarm 23. Power Cut R. Jacobsson VELO Closure Slide material from Karol Hennessy and Malcolm John 52

6. Shift Leader 1. Duties 2. Golden Rules 3. Cold Start 4. FSM Concepts 5. Novelties Fill Preparation LHCb State Control 8. Operational Procedure 9. Mode Handshakes 10. Injection State 11. HV Status 12. Luminosity Leveling 13. Background Monitoring 14. Protective Beam Dump 15. Timing Monitoring 16. VELO Closure 17. Farm Handling 18. Run Configuration 19. Error and Problems 20. Dead Time 21. Access Handling 22. Level 3 Alarm 23. Power Cut R. Jacobsson  Closing and opening the VELO is fully automatic  Requires ONLY confirmations and your attentiveness VELO is only fully in the shadow of machine elements in garage position…  You will be requested to confirm the closure action at 29mm VELO will pause on all steps and reconstruct a new beam position Stops and measures at 20, 10, 5, 1mm  The beam monitoring requires 400 vertices to reconstruct the luminous region  Make sure the trigger is working and select physics events  Call VELO piquet in case of doubt or problems 53

6. Shift Leader 1. Duties 2. Golden Rules 3. Cold Start 4. FSM Concepts 5. Novelties Fill Preparation LHCb State Control 8. Operational Procedure 9. Mode Handshakes 10. Injection State 11. HV Status 12. Luminosity Leveling 13. Background Monitoring 14. Protective Beam Dump 15. Timing Monitoring 16. VELO Closure 17. Farm Handling 18. Run Configuration 19. Error and Problems 20. Dead Time 21. Access Handling 22. Level 3 Alarm 23. Power Cut R. Jacobsson  Make sure you are the owner of the Closure FSM Only one user can operate the VELO at time  Read carefully the information that is displayed in the “Status Box” 54

6. Shift Leader 1. Duties 2. Golden Rules 3. Cold Start 4. FSM Concepts 5. Novelties Fill Preparation LHCb State Control 8. Operational Procedure 9. Mode Handshakes 10. Injection State 11. HV Status 12. Luminosity Leveling 13. Background Monitoring 14. Protective Beam Dump 15. Timing Monitoring 16. VELO Closure 17. Farm Handling 18. Run Configuration 19. Error and Problems 20. Dead Time 21. Access Handling 22. Level 3 Alarm 23. Power Cut R. Jacobsson 55 Open/Close Position of Motion system in absolute frame Absolute Position of beam as calculated by VELO beam monitoring Auxiliary support panels FSM “take” Criteria Checks Log Messages

6. Shift Leader 1. Duties 2. Golden Rules 3. Cold Start 4. FSM Concepts 5. Novelties Fill Preparation LHCb State Control 8. Operational Procedure 9. Mode Handshakes 10. Injection State 11. HV Status 12. Luminosity Leveling 13. Background Monitoring 14. Protective Beam Dump 15. Timing Monitoring 16. VELO Closure 17. Farm Handling 18. Run Configuration 19. Error and Problems 20. Dead Time 21. Access Handling 22. Level 3 Alarm 23. Power Cut R. Jacobsson  Several criteria must be fulfilled for the VELO to be allowed to leave garage position  External via hardware: 1. STABLE BEAM and MOVABLE DEVICES ALLOWED declared by the machine  Internal via software:  LHCb DAQ  Required to be running in order to monitor luminous region  VELO HV  High Voltage Currents should be below threshold  BCM  Beam Condition Monitor background should be below threshold  BPM  Beam Position Monitors should show stable orbit  VTX  Luminous region stable and well centered  Actual list of criteria and thresholds detailed in “Criteria Panel”  Criteria reflected by the panel LEDs  Blue = not checked  Orange = check failed  Green = OK  All LEDs must be green to allow movement  If a criteria fails during closure, VELO requests opening. Auto-aborts after 2 minutes  In addition you should check general beam stability:  Trigger rates, luminosity and specific luminosity trends  Background trends 56

6. Shift Leader 1. Duties 2. Golden Rules 3. Cold Start 4. FSM Concepts 5. Novelties Fill Preparation LHCb State Control 8. Operational Procedure 9. Mode Handshakes 10. Injection State 11. HV Status 12. Luminosity Leveling 13. Background Monitoring 14. Protective Beam Dump 15. Timing Monitoring 16. VELO Closure 17. Farm Handling 18. Run Configuration 19. Error and Problems 20. Dead Time 21. Access Handling 22. Level 3 Alarm 23. Power Cut R. Jacobsson  Automatically prompted to confirm closure/next step when all criteria fulfilled 57 All OK

6. Shift Leader 1. Duties 2. Golden Rules 3. Cold Start 4. FSM Concepts 5. Novelties Fill Preparation LHCb State Control 8. Operational Procedure 9. Mode Handshakes 10. Injection State 11. HV Status 12. Luminosity Leveling 13. Background Monitoring 14. Protective Beam Dump 15. Timing Monitoring 16. VELO Closure 17. Farm Handling 18. Run Configuration 19. Error and Problems 20. Dead Time 21. Access Handling 22. Level 3 Alarm 23. Power Cut R. Jacobsson  Beam Dump or beams lost  Fully automatic  Criteria failed during closure or while closed  Requests opening  There is no reason for stress or panic  If shifter does not confirm in 2min, opens automatically If criteria is fulfilled again, clears request or if already open, requests closing  Thus, in most cases investigate reason for failed criteria, it may be temporary Exception to the 2min rule:  If DAQ is stopped, there are no new vertices, position monitoring uses BPM for a 15min grace period. After 20 min VELO will ask to open  Should be no reason but closing and opening is available from FSM Object 58 NOT OK Countdown (2mins)

6. Shift Leader 1. Duties 2. Golden Rules 3. Cold Start 4. FSM Concepts 5. Novelties Fill Preparation LHCb State Control 8. Operational Procedure 9. Mode Handshakes 10. Injection State 11. HV Status 12. Luminosity Leveling 13. Background Monitoring 14. Protective Beam Dump 15. Timing Monitoring 16. VELO Closure 17. Farm Handling 18. Run Configuration 19. Error and Problems 20. Dead Time 21. Access Handling 22. Level 3 Alarm 23. Power Cut R. Jacobsson  If the closing panel is frozen or the FSM is in error (VELO_DCS -> VELO_MOTION) call the VELO piquet  If the VELO will not close or conditions (LEDs) fail more than once, call the VELO piquet  Emergency Out button 59

6. Shift Leader 1. Duties 2. Golden Rules 3. Cold Start 4. FSM Concepts 5. Novelties Fill Preparation LHCb State Control 8. Operational Procedure 9. Mode Handshakes 10. Injection State 11. HV Status 12. Luminosity Leveling 13. Background Monitoring 14. Protective Beam Dump 15. Timing Monitoring 16. VELO Closure 17. Farm Handling 18. Run Configuration 19. Error and Problems 20. Dead Time 21. Access Handling 22. Level 3 Alarm 23. Power Cut R. Jacobsson Data Taking 60

6. Shift Leader 1. Duties 2. Golden Rules 3. Cold Start 4. FSM Concepts 5. Novelties Fill Preparation LHCb State Control 8. Operational Procedure 9. Mode Handshakes 10. Injection State 11. HV Status 12. Luminosity Leveling 13. Background Monitoring 14. Protective Beam Dump 15. Timing Monitoring 16. VELO Closure 17. Farm Handling 18. Run Configuration 19. Error and Problems 20. Dead Time 21. Access Handling 22. Level 3 Alarm 23. Power Cut R. Jacobsson 61 Run Control LHCb State Control

6. Shift Leader 1. Duties 2. Golden Rules 3. Cold Start 4. FSM Concepts 5. Novelties Fill Preparation LHCb State Control 8. Operational Procedure 9. Mode Handshakes 10. Injection State 11. HV Status 12. Luminosity Leveling 13. Background Monitoring 14. Protective Beam Dump 15. Timing Monitoring 16. VELO Closure 17. Farm Handling 18. Run Configuration 19. Error and Problems 20. Dead Time 21. Access Handling 22. Level 3 Alarm 23. Power Cut R. Jacobsson 62 Top Node Click to get breakdown Important to check outcome of commands Concept of system ownership Subdetector may be included/excluded

6. Shift Leader 1. Duties 2. Golden Rules 3. Cold Start 4. FSM Concepts 5. Novelties Fill Preparation LHCb State Control 8. Operational Procedure 9. Mode Handshakes 10. Injection State 11. HV Status 12. Luminosity Leveling 13. Background Monitoring 14. Protective Beam Dump 15. Timing Monitoring 16. VELO Closure 17. Farm Handling 18. Run Configuration 19. Error and Problems 20. Dead Time 21. Access Handling 22. Level 3 Alarm 23. Power Cut R. Jacobsson  Always send the Configure command from Top! Reset command may be sent from anywhere top or local subsystem 63

6. Shift Leader 1. Duties 2. Golden Rules 3. Cold Start 4. FSM Concepts 5. Novelties Fill Preparation LHCb State Control 8. Operational Procedure 9. Mode Handshakes 10. Injection State 11. HV Status 12. Luminosity Leveling 13. Background Monitoring 14. Protective Beam Dump 15. Timing Monitoring 16. VELO Closure 17. Farm Handling 18. Run Configuration 19. Error and Problems 20. Dead Time 21. Access Handling 22. Level 3 Alarm 23. Power Cut R. Jacobsson Setting up LHCb for a run consist of choosing and configuring:  Partition composition (subdetectors and common resources)  Activity  Farm size  Trigger Configuration  L0+HLT TCK + ODIN Triggers  Data Type  Data Destination  Technical Triggers in addition to HV/LV obviously… 64

6. Shift Leader 1. Duties 2. Golden Rules 3. Cold Start 4. FSM Concepts 5. Novelties Fill Preparation LHCb State Control 8. Operational Procedure 9. Mode Handshakes 10. Injection State 11. HV Status 12. Luminosity Leveling 13. Background Monitoring 14. Protective Beam Dump 15. Timing Monitoring 16. VELO Closure 17. Farm Handling 18. Run Configuration 19. Error and Problems 20. Dead Time 21. Access Handling 22. Level 3 Alarm 23. Power Cut R. Jacobsson 65  Include subdetectors and make sure all common resources are included  NOTE: Trigger components and associated subdetectors must be excluded/included together!  Make sure nothing is excluded/disabled below top FSM objects!

6. Shift Leader 1. Duties 2. Golden Rules 3. Cold Start 4. FSM Concepts 5. Novelties Fill Preparation LHCb State Control 8. Operational Procedure 9. Mode Handshakes 10. Injection State 11. HV Status 12. Luminosity Leveling 13. Background Monitoring 14. Protective Beam Dump 15. Timing Monitoring 16. VELO Closure 17. Farm Handling 18. Run Configuration 19. Error and Problems 20. Dead Time 21. Access Handling 22. Level 3 Alarm 23. Power Cut R. Jacobsson  “Activity” defines the “recipe” which will be applied by all subsystems on CONFIGURE  If no special recipe is required for a particular activity, subsystem applies recipe DEFAULT  Activity contains the global run settings (#subfarms, TAE, max events, steps Trigger Configuration by default, Data type by default) 66

6. Shift Leader 1. Duties 2. Golden Rules 3. Cold Start 4. FSM Concepts 5. Novelties Fill Preparation LHCb State Control 8. Operational Procedure 9. Mode Handshakes 10. Injection State 11. HV Status 12. Luminosity Leveling 13. Background Monitoring 14. Protective Beam Dump 15. Timing Monitoring 16. VELO Closure 17. Farm Handling 18. Run Configuration 19. Error and Problems 20. Dead Time 21. Access Handling 22. Level 3 Alarm 23. Power Cut R. Jacobsson  You may check e.g. the farm and other resources you have allocated in “RunInfo” 67

6. Shift Leader 1. Duties 2. Golden Rules 3. Cold Start 4. FSM Concepts 5. Novelties Fill Preparation LHCb State Control 8. Operational Procedure 9. Mode Handshakes 10. Injection State 11. HV Status 12. Luminosity Leveling 13. Background Monitoring 14. Protective Beam Dump 15. Timing Monitoring 16. VELO Closure 17. Farm Handling 18. Run Configuration 19. Error and Problems 20. Dead Time 21. Access Handling 22. Level 3 Alarm 23. Power Cut R. Jacobsson  If allocation fails because insufficient farm nodes available or if you want to change size of farm  Number of farm nodes to be allocated is configured in “RunInfo:Edit” 68

6. Shift Leader 1. Duties 2. Golden Rules 3. Cold Start 4. FSM Concepts 5. Novelties Fill Preparation LHCb State Control 8. Operational Procedure 9. Mode Handshakes 10. Injection State 11. HV Status 12. Luminosity Leveling 13. Background Monitoring 14. Protective Beam Dump 15. Timing Monitoring 16. VELO Closure 17. Farm Handling 18. Run Configuration 19. Error and Problems 20. Dead Time 21. Access Handling 22. Level 3 Alarm 23. Power Cut R. Jacobsson  Check the available number of subfarms in “ServiceDomains:FARM” 69

6. Shift Leader 1. Duties 2. Golden Rules 3. Cold Start 4. FSM Concepts 5. Novelties Fill Preparation LHCb State Control 8. Operational Procedure 9. Mode Handshakes 10. Injection State 11. HV Status 12. Luminosity Leveling 13. Background Monitoring 14. Protective Beam Dump 15. Timing Monitoring 16. VELO Closure 17. Farm Handling 18. Run Configuration 19. Error and Problems 20. Dead Time 21. Access Handling 22. Level 3 Alarm 23. Power Cut R. Jacobsson  Make sure “Detector Control System” is READY ! 70

6. Shift Leader 1. Duties 2. Golden Rules 3. Cold Start 4. FSM Concepts 5. Novelties Fill Preparation LHCb State Control 8. Operational Procedure 9. Mode Handshakes 10. Injection State 11. HV Status 12. Luminosity Leveling 13. Background Monitoring 14. Protective Beam Dump 15. Timing Monitoring 16. VELO Closure 17. Farm Handling 18. Run Configuration 19. Error and Problems 20. Dead Time 21. Access Handling 22. Level 3 Alarm 23. Power Cut R. Jacobsson  “Trigger Configurations” define L0 and an HLT setup (“Trigger Configuration Key - TCK”) together ODIN triggers for luminosity and beam-gas  Trigger Configurations may be changed on the fly  “Fast Run Change” 71

6. Shift Leader 1. Duties 2. Golden Rules 3. Cold Start 4. FSM Concepts 5. Novelties Fill Preparation LHCb State Control 8. Operational Procedure 9. Mode Handshakes 10. Injection State 11. HV Status 12. Luminosity Leveling 13. Background Monitoring 14. Protective Beam Dump 15. Timing Monitoring 16. VELO Closure 17. Farm Handling 18. Run Configuration 19. Error and Problems 20. Dead Time 21. Access Handling 22. Level 3 Alarm 23. Power Cut R. Jacobsson 72

6. Shift Leader 1. Duties 2. Golden Rules 3. Cold Start 4. FSM Concepts 5. Novelties Fill Preparation LHCb State Control 8. Operational Procedure 9. Mode Handshakes 10. Injection State 11. HV Status 12. Luminosity Leveling 13. Background Monitoring 14. Protective Beam Dump 15. Timing Monitoring 16. VELO Closure 17. Farm Handling 18. Run Configuration 19. Error and Problems 20. Dead Time 21. Access Handling 22. Level 3 Alarm 23. Power Cut R. Jacobsson  “Data Type” used OFFLINE to identify data 73

6. Shift Leader 1. Duties 2. Golden Rules 3. Cold Start 4. FSM Concepts 5. Novelties Fill Preparation LHCb State Control 8. Operational Procedure 9. Mode Handshakes 10. Injection State 11. HV Status 12. Luminosity Leveling 13. Background Monitoring 14. Protective Beam Dump 15. Timing Monitoring 16. VELO Closure 17. Farm Handling 18. Run Configuration 19. Error and Problems 20. Dead Time 21. Access Handling 22. Level 3 Alarm 23. Power Cut R. Jacobsson  “Data Destination” is normally handled automatically Switches to OFFLINE as soon as physics conditions established 74

6. Shift Leader 1. Duties 2. Golden Rules 3. Cold Start 4. FSM Concepts 5. Novelties Fill Preparation LHCb State Control 8. Operational Procedure 9. Mode Handshakes 10. Injection State 11. HV Status 12. Luminosity Leveling 13. Background Monitoring 14. Protective Beam Dump 15. Timing Monitoring 16. VELO Closure 17. Farm Handling 18. Run Configuration 19. Error and Problems 20. Dead Time 21. Access Handling 22. Level 3 Alarm 23. Power Cut R. Jacobsson  Open by clicking on “RunDB”  Usually good praxis to check that files are opened at the beginning of fill 75 Allows changing destination manually

6. Shift Leader 1. Duties 2. Golden Rules 3. Cold Start 4. FSM Concepts 5. Novelties Fill Preparation LHCb State Control 8. Operational Procedure 9. Mode Handshakes 10. Injection State 11. HV Status 12. Luminosity Leveling 13. Background Monitoring 14. Protective Beam Dump 15. Timing Monitoring 16. VELO Closure 17. Farm Handling 18. Run Configuration 19. Error and Problems 20. Dead Time 21. Access Handling 22. Level 3 Alarm 23. Power Cut R. Jacobsson  Technical triggers may be added for test purposes from “TFC Control” panel  TFC settings may be modified after “Stop Trigger” in state ACTIVE 76

6. Shift Leader 1. Duties 2. Golden Rules 3. Cold Start 4. FSM Concepts 5. Novelties Fill Preparation LHCb State Control 8. Operational Procedure 9. Mode Handshakes 10. Injection State 11. HV Status 12. Luminosity Leveling 13. Background Monitoring 14. Protective Beam Dump 15. Timing Monitoring 16. VELO Closure 17. Farm Handling 18. Run Configuration 19. Error and Problems 20. Dead Time 21. Access Handling 22. Level 3 Alarm 23. Power Cut R. Jacobsson  Always send the Configure command from Top! Reset command may be sent from anywhere top or local subsystem 77

6. Shift Leader 1. Duties 2. Golden Rules 3. Cold Start 4. FSM Concepts 5. Novelties Fill Preparation LHCb State Control 8. Operational Procedure 9. Mode Handshakes 10. Injection State 11. HV Status 12. Luminosity Leveling 13. Background Monitoring 14. Protective Beam Dump 15. Timing Monitoring 16. VELO Closure 17. Farm Handling 18. Run Configuration 19. Error and Problems 20. Dead Time 21. Access Handling 22. Level 3 Alarm 23. Power Cut R. Jacobsson 78

6. Shift Leader 1. Duties 2. Golden Rules 3. Cold Start 4. FSM Concepts 5. Novelties Fill Preparation LHCb State Control 8. Operational Procedure 9. Mode Handshakes 10. Injection State 11. HV Status 12. Luminosity Leveling 13. Background Monitoring 14. Protective Beam Dump 15. Timing Monitoring 16. VELO Closure 17. Farm Handling 18. Run Configuration 19. Error and Problems 20. Dead Time 21. Access Handling 22. Level 3 Alarm 23. Power Cut R. Jacobsson  The HLT farm now consist of 50 Subfarms, each with 27 nodes, each with either 8 or 20 Moore tasks = HLT tasks!  A lot of work has been put into improving configuration/startup speed New forking mechanism of tasks (being commissioned) Majority Ready mechanism to Start Run as soon as 70% of farm is READY Automatic recovery of HLT tasks  HLT FSM object gets grey and message in Message Box in Run Control panel Removing on the fly entire subfarms / individual nodes which do not figure 79  Opened by clicking on “HLT” in Run Control  Advice: keep open at crucial moments!  Majority mechanism uses in practice removal mechanism as soon as 70% READY  1 minute after Start Run, the HLT Automatic Recovery will try to include them  Manual removal  Removed farms are excluded until included  Mail is sent automatically to Online Team  Try to get them running while run is running if all is stable by including them again HLT Automatic Recovery will try to get them to state Running 

6. Shift Leader 1. Duties 2. Golden Rules 3. Cold Start 4. FSM Concepts 5. Novelties Fill Preparation LHCb State Control 8. Operational Procedure 9. Mode Handshakes 10. Injection State 11. HV Status 12. Luminosity Leveling 13. Background Monitoring 14. Protective Beam Dump 15. Timing Monitoring 16. VELO Closure 17. Farm Handling 18. Run Configuration 19. Error and Problems 20. Dead Time 21. Access Handling 22. Level 3 Alarm 23. Power Cut R. Jacobsson 80

6. Shift Leader 1. Duties 2. Golden Rules 3. Cold Start 4. FSM Concepts 5. Novelties Fill Preparation LHCb State Control 8. Operational Procedure 9. Mode Handshakes 10. Injection State 11. HV Status 12. Luminosity Leveling 13. Background Monitoring 14. Protective Beam Dump 15. Timing Monitoring 16. VELO Closure 17. Farm Handling 18. Run Configuration 19. Error and Problems 20. Dead Time 21. Access Handling 22. Level 3 Alarm 23. Power Cut R. Jacobsson 81

6. Shift Leader 1. Duties 2. Golden Rules 3. Cold Start 4. FSM Concepts 5. Novelties Fill Preparation LHCb State Control 8. Operational Procedure 9. Mode Handshakes 10. Injection State 11. HV Status 12. Luminosity Leveling 13. Background Monitoring 14. Protective Beam Dump 15. Timing Monitoring 16. VELO Closure 17. Farm Handling 18. Run Configuration 19. Error and Problems 20. Dead Time 21. Access Handling 22. Level 3 Alarm 23. Power Cut R. Jacobsson  ONLY DCS, DAQ, TFC, HLT, STORAGE needed to take data Monitoring, Reconstruction and Calibration Farms needed to take GOOD data  Mechanism to ignore temporarily these FSM objects on the fly 82

6. Shift Leader 1. Duties 2. Golden Rules 3. Cold Start 4. FSM Concepts 5. Novelties Fill Preparation LHCb State Control 8. Operational Procedure 9. Mode Handshakes 10. Injection State 11. HV Status 12. Luminosity Leveling 13. Background Monitoring 14. Protective Beam Dump 15. Timing Monitoring 16. VELO Closure 17. Farm Handling 18. Run Configuration 19. Error and Problems 20. Dead Time 21. Access Handling 22. Level 3 Alarm 23. Power Cut R. Jacobsson 83 Click to get breakdown

6. Shift Leader 1. Duties 2. Golden Rules 3. Cold Start 4. FSM Concepts 5. Novelties Fill Preparation LHCb State Control 8. Operational Procedure 9. Mode Handshakes 10. Injection State 11. HV Status 12. Luminosity Leveling 13. Background Monitoring 14. Protective Beam Dump 15. Timing Monitoring 16. VELO Closure 17. Farm Handling 18. Run Configuration 19. Error and Problems 20. Dead Time 21. Access Handling 22. Level 3 Alarm 23. Power Cut R. Jacobsson  Many sources may generate DAQ deadtime Technically meaning that trigger is ‘throttled’ to some level Normal sources (ODIN)  FE derandomizer occupancy  Standard emulator  OTIS Emulator (OT)  Beetle Emulator (VELO, ST)  Trigger gap due to RICH  Synchronous Front-End reset  + other non-conformities Technical  TELL1s problem in receiving data from FE or sending data to FARM  Desynchronized or incomplete events detected in Event Filter Farm  Data congestion in Event Filter Farm or Storage 84

6. Shift Leader 1. Duties 2. Golden Rules 3. Cold Start 4. FSM Concepts 5. Novelties Fill Preparation LHCb State Control 8. Operational Procedure 9. Mode Handshakes 10. Injection State 11. HV Status 12. Luminosity Leveling 13. Background Monitoring 14. Protective Beam Dump 15. Timing Monitoring 16. VELO Closure 17. Farm Handling 18. Run Configuration 19. Error and Problems 20. Dead Time 21. Access Handling 22. Level 3 Alarm 23. Power Cut R. Jacobsson 85 Click to find out which subdetector is throttling

6. Shift Leader 1. Duties 2. Golden Rules 3. Cold Start 4. FSM Concepts 5. Novelties Fill Preparation LHCb State Control 8. Operational Procedure 9. Mode Handshakes 10. Injection State 11. HV Status 12. Luminosity Leveling 13. Background Monitoring 14. Protective Beam Dump 15. Timing Monitoring 16. VELO Closure 17. Farm Handling 18. Run Configuration 19. Error and Problems 20. Dead Time 21. Access Handling 22. Level 3 Alarm 23. Power Cut R. Jacobsson  Status of throttle signals sampled every 3 seconds 86

6. Shift Leader 1. Duties 2. Golden Rules 3. Cold Start 4. FSM Concepts 5. Novelties Fill Preparation LHCb State Control 8. Operational Procedure 9. Mode Handshakes 10. Injection State 11. HV Status 12. Luminosity Leveling 13. Background Monitoring 14. Protective Beam Dump 15. Timing Monitoring 16. VELO Closure 17. Farm Handling 18. Run Configuration 19. Error and Problems 20. Dead Time 21. Access Handling 22. Level 3 Alarm 23. Power Cut R. Jacobsson  Most problems are recovered by a (well-chosen) sequence of RECOVER from top node RESET often on only system which is in error (sometimes more than one…) CONFIGURE from top (sometimes more than one…)  FARM (single subfarms, single nodes), MONITORING, RECONSTRUCTION, CALIBRATION FARM may be manipulated while run is taking data perfectly well!  Recovering a subdetector from error rarely requires “STOP_RUN” Only “STOP_TRIGGER” and then reset subdetector/configure from top/start_run again  Again, failing FARM nodes/SUBFARMs may be excluded(included) on the fly 87

6. Shift Leader 1. Duties 2. Golden Rules 3. Cold Start 4. FSM Concepts 5. Novelties Fill Preparation LHCb State Control 8. Operational Procedure 9. Mode Handshakes 10. Injection State 11. HV Status 12. Luminosity Leveling 13. Background Monitoring 14. Protective Beam Dump 15. Timing Monitoring 16. VELO Closure 17. Farm Handling 18. Run Configuration 19. Error and Problems 20. Dead Time 21. Access Handling 22. Level 3 Alarm 23. Power Cut R. Jacobsson  Surely most difficult problem is not responding FSMs… Most difficult to identify May be deep down the tree No magic, restart FSM, often starting from lowest level in the tree NOTE: After restart, the FSM object returns to UNLOCKED  Take it  May also be done from the very top of Run Control if many things should be taken  If FSM is “DEAD”, command restart is available from FSM  If not, right-click on FSM Object: 88

6. Shift Leader 1. Duties 2. Golden Rules 3. Cold Start 4. FSM Concepts 5. Novelties Fill Preparation LHCb State Control 8. Operational Procedure 9. Mode Handshakes 10. Injection State 11. HV Status 12. Luminosity Leveling 13. Background Monitoring 14. Protective Beam Dump 15. Timing Monitoring 16. VELO Closure 17. Farm Handling 18. Run Configuration 19. Error and Problems 20. Dead Time 21. Access Handling 22. Level 3 Alarm 23. Power Cut R. Jacobsson  Many automatic recoveries Some even with “AutoPilot OFF”  Do not interfere while these are in progress as seen in Message box at the bottom of the Run Control 89

6. Shift Leader 1. Duties 2. Golden Rules 3. Cold Start 4. FSM Concepts 5. Novelties Fill Preparation LHCb State Control 8. Operational Procedure 9. Mode Handshakes 10. Injection State 11. HV Status 12. Luminosity Leveling 13. Background Monitoring 14. Protective Beam Dump 15. Timing Monitoring 16. VELO Closure 17. Farm Handling 18. Run Configuration 19. Error and Problems 20. Dead Time 21. Access Handling 22. Level 3 Alarm 23. Power Cut R. Jacobsson  To diagnose problems with errors from TELL1 open overview panel from Run Control Best to check the counters when the trigger is stopped 90

6. Shift Leader 1. Duties 2. Golden Rules 3. Cold Start 4. FSM Concepts 5. Novelties Fill Preparation LHCb State Control 8. Operational Procedure 9. Mode Handshakes 10. Injection State 11. HV Status 12. Luminosity Leveling 13. Background Monitoring 14. Protective Beam Dump 15. Timing Monitoring 16. VELO Closure 17. Farm Handling 18. Run Configuration 19. Error and Problems 20. Dead Time 21. Access Handling 22. Level 3 Alarm 23. Power Cut R. Jacobsson  The LHCb Slow Control System is a large ensemble of various pieces of hardware and software Controls "stable" parameters like voltages and crate status, and constantly monitors that all values are within tolerances  DCS (Detector Control System) Handles the powering of the front-end electronics, but also any possible gas system, cooling, temperature monitor, and more for some detectors like the VELO with vacuum and position systems. Its state has to be READY before a run can be started  The CONFIGURE command first tries to do that if needed  Problems in this area requires sub-detector experts  DAI (DAq Infrastructure) Handles the crates housing readout modules in D3, and in the cavern. Here also it has to be READY before to configure the system 91

6. Shift Leader 1. Duties 2. Golden Rules 3. Cold Start 4. FSM Concepts 5. Novelties Fill Preparation LHCb State Control 8. Operational Procedure 9. Mode Handshakes 10. Injection State 11. HV Status 12. Luminosity Leveling 13. Background Monitoring 14. Protective Beam Dump 15. Timing Monitoring 16. VELO Closure 17. Farm Handling 18. Run Configuration 19. Error and Problems 20. Dead Time 21. Access Handling 22. Level 3 Alarm 23. Power Cut R. Jacobsson  Most crates are controlled from a central INFDAI system Don’t power up/down without clear sub-detector instructions 92

6. Shift Leader 1. Duties 2. Golden Rules 3. Cold Start 4. FSM Concepts 5. Novelties Fill Preparation LHCb State Control 8. Operational Procedure 9. Mode Handshakes 10. Injection State 11. HV Status 12. Luminosity Leveling 13. Background Monitoring 14. Protective Beam Dump 15. Timing Monitoring 16. VELO Closure 17. Farm Handling 18. Run Configuration 19. Error and Problems 20. Dead Time 21. Access Handling 22. Level 3 Alarm 23. Power Cut R. Jacobsson 93

6. Shift Leader 1. Duties 2. Golden Rules 3. Cold Start 4. FSM Concepts 5. Novelties Fill Preparation LHCb State Control 8. Operational Procedure 9. Mode Handshakes 10. Injection State 11. HV Status 12. Luminosity Leveling 13. Background Monitoring 14. Protective Beam Dump 15. Timing Monitoring 16. VELO Closure 17. Farm Handling 18. Run Configuration 19. Error and Problems 20. Dead Time 21. Access Handling 22. Level 3 Alarm 23. Power Cut R. Jacobsson  Access procedure and application…. Instructions on Shifter Twiki 94 To be updated with ew IMPACT procedure

6. Shift Leader 1. Duties 2. Golden Rules 3. Cold Start 4. FSM Concepts 5. Novelties Fill Preparation LHCb State Control 8. Operational Procedure 9. Mode Handshakes 10. Injection State 11. HV Status 12. Luminosity Leveling 13. Background Monitoring 14. Protective Beam Dump 15. Timing Monitoring 16. VELO Closure 17. Farm Handling 18. Run Configuration 19. Error and Problems 20. Dead Time 21. Access Handling 22. Level 3 Alarm 23. Power Cut R. Jacobsson  The Run Chief/Run Coordinator authorizes access requests Informs you when the access may be done, and informs the requestor and the RPE in advance If the LHC starts an ACCESS you should inform the Run Chief  Procedure for starting access 1. Wait for the RPE piquet to come to the Control Room 2. The RPE piquet will go into the cavern 3. Wait until the RPE piquet lifts the RP-VETO and tells you to call the CCC 4. Call the CCC and ask them to put UX85 in Restricted Mode and to delegate access control to you (“Access Delegation”) 5. Give access with key to RPE piquet 6. The RPE piquet will inform you when the RP survey is completed and access to UX85B is allowed  See for instructions on how to get delegation and to give accesshttps://edms.cern.ch/document/

6. Shift Leader 1. Duties 2. Golden Rules 3. Cold Start 4. FSM Concepts 5. Novelties Fill Preparation LHCb State Control 8. Operational Procedure 9. Mode Handshakes 10. Injection State 11. HV Status 12. Luminosity Leveling 13. Background Monitoring 14. Protective Beam Dump 15. Timing Monitoring 16. VELO Closure 17. Farm Handling 18. Run Configuration 19. Error and Problems 20. Dead Time 21. Access Handling 22. Level 3 Alarm 23. Power Cut R. Jacobsson  When access is cleared by RP Grant access to the people listed for the intervention(s) in the authorized access request  see  Make sure people leave the cavern before the end of the access period  When accesses are completed: Call the CCC and asks to take back the Delegation and close UX85 Inform the RPE piquet 96

6. Shift Leader 1. Duties 2. Golden Rules 3. Cold Start 4. FSM Concepts 5. Novelties Fill Preparation LHCb State Control 8. Operational Procedure 9. Mode Handshakes 10. Injection State 11. HV Status 12. Luminosity Leveling 13. Background Monitoring 14. Protective Beam Dump 15. Timing Monitoring 16. VELO Closure 17. Farm Handling 18. Run Configuration 19. Error and Problems 20. Dead Time 21. Access Handling 22. Level 3 Alarm 23. Power Cut R. Jacobsson  The Fire Brigade or CCC will contact you  Make sure the CSAM alarm screen is in “Vue Alarms” not “Historique” Check the alarm and call the GLIMOS (ph , ) Procedures for Level 3 Alarms are in the folder by the PC.  Make sure PZ85 is OPEN  If the alarm is on the UX85A side the Fire Brigade can go down immediately  If alarm is on detector side Inform the Fire Brigade if the magnet is on and ask if they want it off The Fire Brigade can intervene when the RP VETO in ON If they ask to lift the VETO call the RPE piquet (ph , )  In case you cannot reach it or the CERN RP service (ph during working hours/ph outside working hours) Make sure the RAMSES panel is working  Inform Run Chief and Run Coordinator 97

6. Shift Leader 1. Duties 2. Golden Rules 3. Cold Start 4. FSM Concepts 5. Novelties Fill Preparation LHCb State Control 8. Operational Procedure 9. Mode Handshakes 10. Injection State 11. HV Status 12. Luminosity Leveling 13. Background Monitoring 14. Protective Beam Dump 15. Timing Monitoring 16. VELO Closure 17. Farm Handling 18. Run Configuration 19. Error and Problems 20. Dead Time 21. Access Handling 22. Level 3 Alarm 23. Power Cut R. Jacobsson  Don’t panick – Game Over  As Shift Leader, you should stay at the helm!  Find out extent of power cut – Site wide, LHCb only, LHC, surface/cavern?  If it is only local and LHC has beam, find out position of VELO Move out if needed in consultation with VELO piquet  BCM and LHC interfaces will normally be on power unless LHC is cut  Try to find out state of cavern ventilation and cooling Phone CCC-Technical Infrastructure (72201) to see about alarms  Contact DSS, all sub-systems and Run Chief and Run Coordinator  Follow-up on reestablishing power 98