6. Shift Leader 1. Introduction 2. SL Duties 3. Golden Rules 4. Operational Procedure 5. Mode Handshakes 6. Cold Start 7. LHCb State Control 8. Clock Switching 9. End of Fill 10. Machine Development 11. Run Control 12. System Allocation 13. System Configuration 14. Run/File Status 15. Farm Node Status 16. Dead Time 17. Error 18. Slow Control 19. Access R. Jacobsson 6. Shift Leader 1

1. Introduction 2. SL Duties 3. Golden Rules 4. Operational Procedure 5. Mode Handshakes 6. Cold Start 7. LHCb State Control 8. Clock Switching 9. End of Fill 10. Machine Development 11. Run Control 12. System Allocation 13. System Configuration 14. Run/File Status 15. Farm Node Status 16. Dead Time 17. Error 18. Slow Control 19. Access R. Jacobsson  The LHCb Shift Crew training consists of several courses 1.Introduction 2.Safety 3.Basic Concepts 4.Running LHCb 5.Data Manager 6.Shift Leader  All of these course are subjects to changes  This course is intended to describe the work of the Shift Leader Tasks Procedures Tools  This is a conceptual course not a tutorial, hands-on training is the only way to learn how to run the system  Shift Leaders should also look through the Data Manager slides  Please comment on missing or unclear topics, errors. Don’t hesitate to ask questions 2

6. Shift Leader 1. Introduction 2. SL Duties 3. Golden Rules 4. Operational Procedure 5. Mode Handshakes 6. Cold Start 7. LHCb State Control 8. Clock Switching 9. End of Fill 10. Machine Development 11. Run Control 12. System Allocation 13. System Configuration 14. Run/File Status 15. Farm Node Status 16. Dead Time 17. Error 18. Slow Control 19. Access R. Jacobsson  Assist Data Manager Watch his displays and help judging data quality based on your knowledge about the running  As Shift Leader you will mainly use the following tools LHCb State Control Run Control Farm Monitoring Trigger Rate Monitoring Alarm and Log screens Run Database LHC Operational Monitoring Detector Safety System Access Control System Elog  Attend Run Meeting (send DM if you are busy) 3

6. Shift Leader 1. Introduction 2. SL Duties 3. Golden Rules 4. Operational Procedure 5. Mode Handshakes 6. Cold Start 7. LHCb State Control 8. Clock Switching 9. End of Fill 10. Machine Development 11. Run Control 12. System Allocation 13. System Configuration 14. Run/File Status 15. Farm Node Status 16. Dead Time 17. Error 18. Slow Control 19. Access R. Jacobsson  Always keep acquisition together running Take data even without collisions (Destination ‘Local’)  In case a detector is excluded to fix a problem, run the system with the rest to detect other problems  Before stopping the run while data taking to fix a problem, think twice Is it necessary? Can another pending problem be fixed in parallel? Is the plan clear, who does what and in which order?  If you discover a problem or an anomaly in data, restart run (“fast run change”)  Recovery 3 attempts/10 minutes before calling expert  Don’t panic!  Click calmly – doubts about whether you did the action or not is a potential source of disaster..! Mouse at the centre of object Firm single click Wait until next action Observe all that is happening on the panel 4

6. Shift Leader 1. Introduction 2. SL Duties 3. Golden Rules 4. Operational Procedure 5. Mode Handshakes 6. Cold Start 7. LHCb State Control 8. Clock Switching 9. End of Fill 10. Machine Development 11. Run Control 12. System Allocation 13. System Configuration 14. Run/File Status 15. Farm Node Status 16. Dead Time 17. Error 18. Slow Control 19. Access R. Jacobsson Fill sequence for physics fill in LHCb language: NO_BEAM (Injection Permit = FALSE, Any state of LHCb, Internal clock) INJECTION (Injection Permit = TRUE, VELO out, External clock) RAMP (Injection Permit = FALSE) PHYS_ADJUST PHYSICS (VELO in) (ADJUST) (VELO out) DUMP (VELO out, to be changed) EOF (Internal Clock, Calibrations) 5 Handshake for Injection Handshake for Dump (Only when directly from PHYSICS) Handshake for Adjust Hectic phase!

6. Shift Leader 1. Introduction 2. SL Duties 3. Golden Rules 4. Operational Procedure 5. Mode Handshakes 6. Cold Start 7. LHCb State Control 8. Clock Switching 9. End of Fill 10. Machine Development 11. Run Control 12. System Allocation 13. System Configuration 14. Run/File Status 15. Farm Node Status 16. Dead Time 17. Error 18. Slow Control 19. Access R. Jacobsson 6 PROBLEM WARNING READY PREPARE STANDBY IMMINENT VETO READY OK LHC: NO_BEAM INJECTION LHCb: STANDBY Confirm VETO ECS GET READY READY  General rule: Get ready in ~5 minutes Confirm RAMP

6. Shift Leader 1. Introduction 2. SL Duties 3. Golden Rules 4. Operational Procedure 5. Mode Handshakes 6. Cold Start 7. LHCb State Control 8. Clock Switching 9. End of Fill 10. Machine Development 11. Run Control 12. System Allocation 13. System Configuration 14. Run/File Status 15. Farm Node Status 16. Dead Time 17. Error 18. Slow Control 19. Access R. Jacobsson  System control tree based on a Finite State Machine (‘FSM Tree’) Control from ‘top’, commands passed down and status up Sub-node status according to rules (“device unit”/”control unit”) Graphics User Interface associated with each sub-node 7

6. Shift Leader 1. Introduction 2. SL Duties 3. Golden Rules 4. Operational Procedure 5. Mode Handshakes 6. Cold Start 7. LHCb State Control 8. Clock Switching 9. End of Fill 10. Machine Development 11. Run Control 12. System Allocation 13. System Configuration 14. Run/File Status 15. Farm Node Status 16. Dead Time 17. Error 18. Slow Control 19. Access R. Jacobsson 8 Run Control LHCb State Control

6. Shift Leader 1. Introduction 2. SL Duties 3. Golden Rules 4. Operational Procedure 5. Mode Handshakes 6. Cold Start 7. LHCb State Control 8. Clock Switching 9. End of Fill 10. Machine Development 11. Run Control 12. System Allocation 13. System Configuration 14. Run/File Status 15. Farm Node Status 16. Dead Time 17. Error 18. Slow Control 19. Access R. Jacobsson 9 VELO Motion Control Coming to a screen near you! LHC State LHCb State Handshake HV/LV State

6. Shift Leader 1. Introduction 2. SL Duties 3. Golden Rules 4. Operational Procedure 5. Mode Handshakes 6. Cold Start 7. LHCb State Control 8. Clock Switching 9. End of Fill 10. Machine Development 11. Run Control 12. System Allocation 13. System Configuration 14. Run/File Status 15. Farm Node Status 16. Dead Time 17. Error 18. Slow Control 19. Access R. Jacobsson  Think of “MD” as “INJECTION” in this example! (Missing screen shot!) 10

6. Shift Leader 1. Introduction 2. SL Duties 3. Golden Rules 4. Operational Procedure 5. Mode Handshakes 6. Cold Start 7. LHCb State Control 8. Clock Switching 9. End of Fill 10. Machine Development 11. Run Control 12. System Allocation 13. System Configuration 14. Run/File Status 15. Farm Node Status 16. Dead Time 17. Error 18. Slow Control 19. Access R. Jacobsson 11

6. Shift Leader 1. Introduction 2. SL Duties 3. Golden Rules 4. Operational Procedure 5. Mode Handshakes 6. Cold Start 7. LHCb State Control 8. Clock Switching 9. End of Fill 10. Machine Development 11. Run Control 12. System Allocation 13. System Configuration 14. Run/File Status 15. Farm Node Status 16. Dead Time 17. Error 18. Slow Control 19. Access R. Jacobsson 12

6. Shift Leader 1. Introduction 2. SL Duties 3. Golden Rules 4. Operational Procedure 5. Mode Handshakes 6. Cold Start 7. LHCb State Control 8. Clock Switching 9. End of Fill 10. Machine Development 11. Run Control 12. System Allocation 13. System Configuration 14. Run/File Status 15. Farm Node Status 16. Dead Time 17. Error 18. Slow Control 19. Access R. Jacobsson 13

6. Shift Leader 1. Introduction 2. SL Duties 3. Golden Rules 4. Operational Procedure 5. Mode Handshakes 6. Cold Start 7. LHCb State Control 8. Clock Switching 9. End of Fill 10. Machine Development 11. Run Control 12. System Allocation 13. System Configuration 14. Run/File Status 15. Farm Node Status 16. Dead Time 17. Error 18. Slow Control 19. Access R. Jacobsson  The LHCb clock is switched automatically to the machine clocks when LHC Beam Mode changes to INJECTION. As a brutal change in the clock has consequences for electronics, it is then advisable NOT TO RUN at this time. Currently there is no automatic switch to internal clock  Manually SET_INTERNAL/SET_EXTERNAL may be done on LHC State Control Panel  Stop the run before 14

6. Shift Leader 1. Introduction 2. SL Duties 3. Golden Rules 4. Operational Procedure 5. Mode Handshakes 6. Cold Start 7. LHCb State Control 8. Clock Switching 9. End of Fill 10. Machine Development 11. Run Control 12. System Allocation 13. System Configuration 14. Run/File Status 15. Farm Node Status 16. Dead Time 17. Error 18. Slow Control 19. Access R. Jacobsson  The LHC clock may have jumps due to resynchronization between SPS and LHC. LHC Clocks only guaranteed to be stable INJECTION  DUMP 15

6. Shift Leader 1. Introduction 2. SL Duties 3. Golden Rules 4. Operational Procedure 5. Mode Handshakes 6. Cold Start 7. LHCb State Control 8. Clock Switching 9. End of Fill 10. Machine Development 11. Run Control 12. System Allocation 13. System Configuration 14. Run/File Status 15. Farm Node Status 16. Dead Time 17. Error 18. Slow Control 19. Access R. Jacobsson  Before dumping the beam by operator, the LHC performs the “Dump Handshake”  Move to a ‘safe state’ which means for us moving the VELO out of the beam to its garage position.  The run should be stopped confirming the handshake, so that all events of the run are taken in the same configuration.  But the beam may also be lost (dumped by a protection system) in which case one has to stop the run and move the VELO out but without beam in the machine…  Stop run before powering down voltages! 16

6. Shift Leader 1. Introduction 2. SL Duties 3. Golden Rules 4. Operational Procedure 5. Mode Handshakes 6. Cold Start 7. LHCb State Control 8. Clock Switching 9. End of Fill 10. Machine Development 11. Run Control 12. System Allocation 13. System Configuration 14. Run/File Status 15. Farm Node Status 16. Dead Time 17. Error 18. Slow Control 19. Access R. Jacobsson  Beam commissioning, beam studies etc  Exception to be handled according to planning  separate instructions INJECTION state safe for most activities Use of MD may change in future  In MD, only one single handshake for entire duration of MD (injection) Stay on Internal Clock General rule, if already in MD stay in MD at each injection handshake until next physics fill 17

6. Shift Leader 1. Introduction 2. SL Duties 3. Golden Rules 4. Operational Procedure 5. Mode Handshakes 6. Cold Start 7. LHCb State Control 8. Clock Switching 9. End of Fill 10. Machine Development 11. Run Control 12. System Allocation 13. System Configuration 14. Run/File Status 15. Farm Node Status 16. Dead Time 17. Error 18. Slow Control 19. Access R. Jacobsson 18 Top Node Click to get breakdown Important to check outcome of commands Concept of system ownership Subdetector may be included/excluded

6. Shift Leader 1. Introduction 2. SL Duties 3. Golden Rules 4. Operational Procedure 5. Mode Handshakes 6. Cold Start 7. LHCb State Control 8. Clock Switching 9. End of Fill 10. Machine Development 11. Run Control 12. System Allocation 13. System Configuration 14. Run/File Status 15. Farm Node Status 16. Dead Time 17. Error 18. Slow Control 19. Access R. Jacobsson Excluded & Locked Out Unlocked Locked Included By You By Someone else By Nobody 19 Incomplete Some CU excluded Some devices disabled Shared By You By Someone else Devices Enabled Disabled  Golden Rule: At beginning of shift, go through system to check what is excluded/disabled

6. Shift Leader 1. Introduction 2. SL Duties 3. Golden Rules 4. Operational Procedure 5. Mode Handshakes 6. Cold Start 7. LHCb State Control 8. Clock Switching 9. End of Fill 10. Machine Development 11. Run Control 12. System Allocation 13. System Configuration 14. Run/File Status 15. Farm Node Status 16. Dead Time 17. Error 18. Slow Control 19. Access R. Jacobsson  System resources are allocated dynamically for an activity 20

6. Shift Leader 1. Introduction 2. SL Duties 3. Golden Rules 4. Operational Procedure 5. Mode Handshakes 6. Cold Start 7. LHCb State Control 8. Clock Switching 9. End of Fill 10. Machine Development 11. Run Control 12. System Allocation 13. System Configuration 14. Run/File Status 15. Farm Node Status 16. Dead Time 17. Error 18. Slow Control 19. Access R. Jacobsson  You should not make changes in here unless instructed Currently shifter selects TCK 0x for INJECTION  PHYS_ADJUST and finally 0x for the rest of the fill  Will become simplified with a more “self-explanatory” button on RC 21

6. Shift Leader 1. Introduction 2. SL Duties 3. Golden Rules 4. Operational Procedure 5. Mode Handshakes 6. Cold Start 7. LHCb State Control 8. Clock Switching 9. End of Fill 10. Machine Development 11. Run Control 12. System Allocation 13. System Configuration 14. Run/File Status 15. Farm Node Status 16. Dead Time 17. Error 18. Slow Control 19. Access R. Jacobsson  Always send the Configure command from Top! Reset command may be sent from anywhere top or local subsystem 22

6. Shift Leader 1. Introduction 2. SL Duties 3. Golden Rules 4. Operational Procedure 5. Mode Handshakes 6. Cold Start 7. LHCb State Control 8. Clock Switching 9. End of Fill 10. Machine Development 11. Run Control 12. System Allocation 13. System Configuration 14. Run/File Status 15. Farm Node Status 16. Dead Time 17. Error 18. Slow Control 19. Access R. Jacobsson 23

6. Shift Leader 1. Introduction 2. SL Duties 3. Golden Rules 4. Operational Procedure 5. Mode Handshakes 6. Cold Start 7. LHCb State Control 8. Clock Switching 9. End of Fill 10. Machine Development 11. Run Control 12. System Allocation 13. System Configuration 14. Run/File Status 15. Farm Node Status 16. Dead Time 17. Error 18. Slow Control 19. Access R. Jacobsson 24

6. Shift Leader 1. Introduction 2. SL Duties 3. Golden Rules 4. Operational Procedure 5. Mode Handshakes 6. Cold Start 7. LHCb State Control 8. Clock Switching 9. End of Fill 10. Machine Development 11. Run Control 12. System Allocation 13. System Configuration 14. Run/File Status 15. Farm Node Status 16. Dead Time 17. Error 18. Slow Control 19. Access R. Jacobsson  Many sources may generate DAQ deadtime Technically meaning that trigger is ‘throttled’ to some level Normal sources  FE derandomizer occupance  Trigger gap due to RICH  Synchronous Front-End reset Technical  TELL1s problem in receiving data from FE or sending data to FARM  Desychronized or incomplete events detected in Event Filter Farm  Data congestion in Event Filter Farm or Storage 25 Click on Deadtime bar Click to find out which subdetector is throttling

6. Shift Leader 1. Introduction 2. SL Duties 3. Golden Rules 4. Operational Procedure 5. Mode Handshakes 6. Cold Start 7. LHCb State Control 8. Clock Switching 9. End of Fill 10. Machine Development 11. Run Control 12. System Allocation 13. System Configuration 14. Run/File Status 15. Farm Node Status 16. Dead Time 17. Error 18. Slow Control 19. Access R. Jacobsson  Most problems are recovered by a (well-chosen) sequence of RECOVER from top node RESET often on only system which is in error (sometimes more than one…) CONFIGURE from top (sometimes more than one…)  FARM (single subfarms, single nodes), MONITORING, RECONSTRUCTION, CALIBRATION FARM may be manipulated while run is taking data perfectly well!  Recovering a subdetector from error does not necessarily always require “STOP_RUN” only “STOP_TRIGGER” and then reset subdetector/configure/start_run again  Failing FARM nodes/SUBFARMs may be excluded(included) on the fly 26

6. Shift Leader 1. Introduction 2. SL Duties 3. Golden Rules 4. Operational Procedure 5. Mode Handshakes 6. Cold Start 7. LHCb State Control 8. Clock Switching 9. End of Fill 10. Machine Development 11. Run Control 12. System Allocation 13. System Configuration 14. Run/File Status 15. Farm Node Status 16. Dead Time 17. Error 18. Slow Control 19. Access R. Jacobsson 27

6. Shift Leader 1. Introduction 2. SL Duties 3. Golden Rules 4. Operational Procedure 5. Mode Handshakes 6. Cold Start 7. LHCb State Control 8. Clock Switching 9. End of Fill 10. Machine Development 11. Run Control 12. System Allocation 13. System Configuration 14. Run/File Status 15. Farm Node Status 16. Dead Time 17. Error 18. Slow Control 19. Access R. Jacobsson  The LHCb Slow Control System is a large ensemble of various pieces of hardware and software Controls "stable" parameters like voltages and crate status, and constantly monitors that all values are within tolerances  DCS (Detector Control System) Handles the powering of the front-end electronics, but also any possible gas system, cooling, temperature monitor, and more for some detectors like the VELO with vacuum and position systems. Its state has to be READY before a run can be started  The CONFIGURE command first tries to do that if needed  Problems in this area requires sub-detector experts  DAI (DAq Infrastructure) Handles the crates housing readout modules in D3, and in the cavern. Here also it has to be READY before to configure the system 28

6. Shift Leader 1. Introduction 2. SL Duties 3. Golden Rules 4. Operational Procedure 5. Mode Handshakes 6. Cold Start 7. LHCb State Control 8. Clock Switching 9. End of Fill 10. Machine Development 11. Run Control 12. System Allocation 13. System Configuration 14. Run/File Status 15. Farm Node Status 16. Dead Time 17. Error 18. Slow Control 19. Access R. Jacobsson  Most crates are controlled from a central INFDAI system Don’t power up/down without clear sub-detector instructions 29

6. Shift Leader 1. Introduction 2. SL Duties 3. Golden Rules 4. Operational Procedure 5. Mode Handshakes 6. Cold Start 7. LHCb State Control 8. Clock Switching 9. End of Fill 10. Machine Development 11. Run Control 12. System Allocation 13. System Configuration 14. Run/File Status 15. Farm Node Status 16. Dead Time 17. Error 18. Slow Control 19. Access R. Jacobsson 30

6. Shift Leader 1. Introduction 2. SL Duties 3. Golden Rules 4. Operational Procedure 5. Mode Handshakes 6. Cold Start 7. LHCb State Control 8. Clock Switching 9. End of Fill 10. Machine Development 11. Run Control 12. System Allocation 13. System Configuration 14. Run/File Status 15. Farm Node Status 16. Dead Time 17. Error 18. Slow Control 19. Access R. Jacobsson  The Run Chief authorizes access requests Informs you when the access may be done, and informs the requestor and the RPE in advance If the LHC starts an ACCESS you should inform the Run Chief  Procedure for starting access 1. Wait for the RPE piquet to come to the Control Room 2. The RPE piquet will go into the cavern 3. Wait until the RPE piquet lifts the RP-VETO and tells you to call the CCC 4. Call the CCC and ask them to put UX85 in Restricted Mode and to delegate access control to you (“Access Delegation”) 5. Give access with key to RPE piquet 6. The RPE piquet will inform you when the RP survey is completed and access to UX85B is allowed  See for instructions on how to get delegation and to give accesshttps://edms.cern.ch/document/

6. Shift Leader 1. Introduction 2. SL Duties 3. Golden Rules 4. Operational Procedure 5. Mode Handshakes 6. Cold Start 7. LHCb State Control 8. Clock Switching 9. End of Fill 10. Machine Development 11. Run Control 12. System Allocation 13. System Configuration 14. Run/File Status 15. Farm Node Status 16. Dead Time 17. Error 18. Slow Control 19. Access R. Jacobsson  When access is cleared by RP Grant access to the people listed for the intervention(s) in the authorized access request  see  Make sure people leave the cavern before the end of the access period  When accesses are completed: Call the CCC and asks to take back the Delegation and close UX85 Inform the RPE piquet 32