R. Jacobsson Real-time monitoring of ambient dose equivalent rates in the LHCb cavern. Three monitors show the radiation in the service cavern UX85-A which is accessible during beam operation. The other monitors are located on the other side of the radiation wall where the detector is (UX85-B). The latter are mainly used by RP before opening detector area for access. The monitors are associated with alarms and interlocks. Diagnostics and Actions: Regularly check that the display is operational Check monitors in UX85-A when people are accessing during beam operation In case of alarms from the monitors in the accessible area (yellow or red highlights) 1. Make sure no one is in UX85-A 2. Put PZ85 in close access mode from the access console 3. In case somebody is in UX85-A, contact them to come to the surface 4. Inform immediately the RP service (CERN) of the alarm ph during working hours and ph outside working hours 5. Inform the GLIMOS and the RSO Eric Thomas (ph ), Gloria Corti (ph ) Restart and Problem: Switch OFF/ON console box Start anywhere with PVSS shortcut: “/group/online/ecs/Shortcuts38/INF/INFMAG/INFMAG_UI_lbRamses_FS” Problem: Send mail to Gloria Corti and Run Chief 1

R. Jacobsson The BLSs are six scintillators located in front of the VELO to monitor fast beam losses, transmitted to LHC as BKG 2. The BCM consists of two stations (upstream/downstream) of diamond detectors. They monitor large beam losses over 80µs and 1280µs divided by dump thresholds (per mille). BCM is transmitted to LHC as BKG 1 and 3, and is connected to the Beam Interlock System for protective beam dump. RMS is based on metal-foils covering the IT for integrated dose and background at scales of seconds. BLS and RMS are mainly for confirmation and detailed background analysis. Diagnostics and Actions: BCM: Watch levels especially at injection and start of physics, and later at regular intervals Any activity above 300 per mille should be noted, specifying station/integration time Injection: Beam losses may reach 300 per mille of dump threshold May happen a few times but should not be systematically above  inform LHC Circulating beam: Beam losses of 100 per mille is an anomaly and affects data taking Up to 40 per mille in 2011 purely due to collisions Single spike OK, repeatedly or constant, inform LHC Dump: If dump threshold is reached (around 100 per mille on display) “DUMP” will flash on display CCC should be called to inform them that we dumped and that we will investigate Run Coordinator should be called (163730) alternatively or Run Chief before rearming Note: On reception of PostMortemTrigger (PMT) due to a beam dump by any system, BCM reads history buffers and is unavailable for 2 min (no data update). It is rearmed automatically afterwards. Restart and Problem: Start with PVSS shortcut: “/group/online/ecs/Shortcuts38/LHC/LHCCOM/LHCCOM_UI_background.pnl” Problem: Call or

R. Jacobsson Shows the curent, magnetic field and temperature of the LHCb magnet. The magnet is under the control of the CCC but LHCb authorizes powering by a hardware key. Diagnostics and Actions: Magnetic current/field: Should normally be ON with a stable current of 5850A. Unless other instructions, call Run Chief if it is not ON or if there are large drifts. Temperature: While ON the temperatures should be stable with T Out ~50 degrees. Call Run Chief or if it is largely different or shows strong drift. Beam Interlock: Beam interlock in red indicates that the beam permit is false and that the LHC beams have been dumped or may not be injected. Call Run Chief or Magnet polarity: The LHC convention is that field UP = NEGATIVE (power converter) polarity, and field DOWN = POSITIVE, obviously… Restart and Problem: Switch OFF/ON console box Start anywhere with PVSS shortcut: “/group/online/ecs/Shortcuts38/INF/INFMAG/INFMAG_UI_lbMagnet_FS” Problem: Call or

R. Jacobsson The BPMs are two bidirectional couplers measuring the transversal position of the beams on both sides of LHCb about 20m away from the IP (LEFT=upstream of VELO, RIGHT=downstream of MUONs). The reference orbit (‘golden orbit’) is marked in the trends and as an empty ring in the graphical view. Please, be aware of the zoom feature in the graphical view. The beams are separated vertically during ramp and squeeze and should be at equal levels for collisions. They are always separated horizontally because of the crossing angle. The VELO beam monitoring allows checking the stability of the luminous region. Diagnostics and Actions: Check when luminosity (collisions) is expected during the Adjust phase. In case of a stop of the DAQ with VELO IN, check for proper updating and drifts VELO relies on receiving BPM information during DAQ stops In case of an offset of more than a mm or a strong drift during physics, please inform LHC and call or Watch the VELO luminous region, should have drift in Z<5mm, in X<20mm, and in Y<10mm  if more contact LHC Check that the System Status in right corner is “Running”, if not call or Restart and Problem: Switch OFF/ON console box Start anywhere with PVSS shortcut: “/group/online/ecs/Shortcuts38/LHC/LHCCOM/LHCCOM_UI_beamPos_v1r00.pnl” Problem: Call or

R. Jacobsson Displays the global timing of the experiment and the monitoring of the LHC bunch filling schemes. The information come from the Beam Phase and Intensity Monitor which is connected to two beam pickups on each side of LHCb. The phase of the global timing and the bunch positions are based on comparing the arrival time of the bunches with respect to the LHC RF clock. It also monitors longitudinal stability of the interaction point by comparing the arrival times of colliding bunches (deltaT). The beam phase and deltaT are greyed out until the ramp starts in physics fills. Diagnostics and Actions: Check that BPIM1 and 2 status are green from state RAMP and onwards Phase of Beam 1: Check in RAMP and PHYS_ADJUST. Should be well within the +/- 0.5ns after the ramp. Note, the acceleration changes phase by ~+100ps. If on or outside +/-0.5ns boundary or strong drift, call or deltaT: Check in PHYS_ADJUST, should be within +/-100ps, otherwise call or Filling Scheme: Check that scheme makes sense after injection If strange or “Wrong Bunch Warning” ON, call or Check that #colliding bunches agree with expected number on “LHC/LHCb Operation Info” Calibration Pulses: Check that calibration pulses (marked in yellow) are located in gaps If wrong, alarm will appear on alarm screen  Call or Restart and Problem: Switch OFF/ON console box Start anywhere with PVSS shortcut: “/group/online/ecs/Shortcuts38/LHC.LHCCOM/LHCCOM_UI_Timing.pnl” Problem: Call or

R. Jacobsson Shows the state of the vacuum valves and pressure in the beam vacuum chamber at different locations close to LHCb. The information is used to correlate with changes in background, triggers rates and occupancies. The lower two panels show the rates of interactions in beam-gas crossings as measured by the BLS scintillators and by the ODIN. Diagnostics and Actions: If drastic changes in background, trigger rates or occupancies, check for correlation with pressure changes or beam-gas rates. If true, inform LHC and Run Chief In case of drastic changes of beam-gas rates (>50%) in the BLS or ODIN during physics, call or Check that System Status is “Running”, if not call or Restart and Problem: Switch OFF/ON console box Start anywhere with PVSS shortcut: “/group/online/ecs/Shortcuts38/LHC/LHCCOM/LHCCOM_UI_BeamGas.pnl” Problem: or

R. Jacobsson Shows the intensity of beam 1 and beam 2, the individual bunch intensities and the beam lifetimes. The information come from the LHCb Beam Phase and Intensity Monitor which is connected to two beam pickups on each side of LHCb, and from the LHC Fast Bunch Current Transformer as well as the Direct Current Transformer. Diagnostics and Actions: If drastic changes in background, trigger rates or occupancies, check for correlation with beam intensity loss and reduced lifetimes. If true, check with LHC and inform Run Chief Allows a safety cross-check of beam presence against information received from LHC. If intensity is displayed, but no other monitors do, it indicates communication problem with LHC. Call or Check that System Status is “Running”, if not call or Restart and Problem: Start anywhere with PVSS shortcut: “/group/online/ecs/Shortcuts38/ Problem: or

R. Jacobsson Shows the rates of luminosity counters and luminosity information as monitored with the BLS scintillator system and with the LHCb detector by ODIN. The L0CALO(bb) rate will only work when the CALO, TCALO and TFC are configured. The status of the luminosity leveling is shown at the bottom. The leveling is performed as a function of a target luminosity and various system parameters. Diagnostics and Actions: Check that System Status is “Running” and that numbers are updating, if not call or “LHC Leveling” should be enabled (green) immediately at Stable Beams, if not call LHC Luminosity ramp is performed in steps 150,250,275,300,325,350 with few minutes at each step The “Leveling Requested” will go green when luminosity differs from target luminosity by >3% “Leveling Active” becomes blue until the request is acknowledged by LHC leveling application “Leveling Active” becomes green when the leveling is being performed, call LHC if this does not happen or luminosity is not changing towards target within 3min of the request. Always make sure we are within 3% of target Physics deadtime should be < ~5-6% Restart and Problem: Start anywhere with PVSS shortcut: “/group/online/ecs/Shortcuts38/LHC/LHCCOM/LHCCOM_UI_Luminosity.pnl Problem: Call or

R. Jacobsson Shows the data taking inefficiencies in terms of time lost and luminosity lost due to HV, VELO closure, DAQ, and throttling. Very useful to get overview of problems during a fill. Diagnostics and Actions: Make sure numbers are updating during data taking, if not call or Keep total inefficiency below 8%! Restart and Problem: Switch OFF/ON console box Start anywhere with PVSS shortcut: “/group/online/ecs/Shortcuts38/LHC/LHCCOM/LHCCOM_UI_Efficiencies.pnl” Problem: Call or daytime 9

R. Jacobsson Web browser displaying the status of LHCb runs and files, mainly oriented towards file transfer to Tier0. The Type column indicates the data type (COLLISION10) and the destination: Local (only on Online disks), Castor (Copied to Tier0 but not processed, typically for calibration data) or OFFLINE (copied and processed, normal type for physics data). Diagnostics and Actions: During data taking, check that files are opened for the most recent run, and check that files are closed and migrated towards CASTOR, and bookkeeping for previous runs. If not call Online Piquet. Restart and Problem: Switch OFF/ON console box Problem: Send mail to LHCb helpdesk 10

R. Jacobsson Shows the trigger flow through the entire system up to the HLT and the calibration farm, the global breakdown of different triggers, and the current TCK. Diagnostics and Actions: Check that all trigger rates make sense as soon as beams are colliding and at start of physics data taking. Check that the total output rate of the L0DU does not differ from the input physics rate in ODIN by more than kHz, if yes call or Check that lumi triggers, calibration triggers, and NZS monitoring triggers are produced and received by the HLT and the calibration farm. Restart and Problem: Switch OFF/ON console box Start anywhere with PVSS shortcut: “/group/online/ecs/Shortcuts38/LHCb/LBECSINFO/LBECSINFO_UI_lbTriggersOverview.pnl” Problem: Call Online Piquet 11

R. Jacobsson Shows the status and activities in the storage and monitoring systems for the LHCb partition. The top left part displays the storage view, with the various streams and the number of events written in each. The bottom left part displays the monitoring view, with all monitoring tasks and their consumed events. The fraction of consumed events indicates if a task follows the rate, or is too slow or stuck. The right part indicates which slices of the two systems are used, and by which partition. In normal running mode, only LHCb should be there. TDET is no harm Diagnostics and Actions: Check that data is flowing through the storage system Check that all monitoring tasks sees a sufficient rate of events If not, the number in front of the task allows finding the task on the monitoring nodes in the Monitoring FSM tree from the Run Control in order to perform a reset Restart and Problem: Switch OFF/ON console box May also be started from ‘plus’ node logged in as ‘lhcb_shift’ >source /group/online/dataflow/scripts/run_terminal05.sh –partition=LHCb Problem: Call Online Piquet 12

R. Jacobsson Shows the status and activities in the HLT farm, the calibration farm, and the monitoring farm. Top line shows the global HLT input rate and output rate, and the number of lost MEPs in the last 5 seconds. The top left box displays the events seen by the HLT and calibration farms. Normally this box should contain only one global line for the HLT. If the rate of a subfarm is different from the average it expands to shows the details automatically. Bottom left box lists the tasks sorted by FSM state. Only the number is given for those tasks in the most common states. Other tasks, and tasks in error, are listed individually. The normal mode is a single line with RUNNING. The right box displays the list of tasks in the monitoring and reconstruction farms, with event rates. Diagnostics and Actions: Irregular repartition between farms indicates a problem in processing capacity. Check if many nodes have been excluded in the HLT FSM view Check the processing status in the “LHCb Farm Monitor” Check if many nodes are not sending MEP Requests by opening the ODIN “Node Status” panel from the “Trigger Throttle Status” panel, try resetting the farm Call Online Piquet if it starts affecting the deadtime High rate of lost MEPs >100, reset subdetector and or subfarm if localized, otherwise call Online Piquet When resetting the complete farms, check that all tasks are gone in the task list. = Restart and Problem: Switch OFF/ON console box Also start with PVSS shortcut: “/group/online/ecs/Shortcuts38/LHCb/EcsMonitor/EcsMonitor_UI_NodeStatus.pnl” Problem: Call Online Piquet 13

R. Jacobsson Monitors the data flow in the LHCb partition. It displays one box per sub-farm, including calibration, reconstruction, monitoring and storage. Hitting return on a sub-farm pops up a display showing the list of nodes in the farm, and counters on event flow in each node. Hitting “.” on the node name gives the buffer manager view of the node: The various buffers with their occupancy, and the list of tasks with their status: Consumer or producer and their state. Press H to get more options to check processes and CPU. Diagnostics and Actions: For farms which seem to be saturated or not processing Check that the tasks on the nodes are in ‘ac’ (active), meaning that the task is consuming events. If the number of events is not increasing, this means the task is looping. Buffers should not be in permanence full, and the counters should be increasing. Else, something is blocked either in the node (all Moore stuck?) or downstream (storage blocked?). Reset node or entire subfarm, call Online Piquet if it doesn’t improve and it starts affecting the deadtime Restart and Problem: Switch OFF/ON console box May also be started from ‘plus’ node logged in as ‘lhcb_shift’: >/group/online/dataflow/scripts/rfarmMon –partition=LHCb Problem: Call Online Piquet 14

R. Jacobsson Allows inspecting the physics triggers visually provided Monitoring Farm is working and there are real L0 triggers. Start Login as lhcb_shift on this machine (password on the white board) Start PuTTY and select ‘plus’ – Login is automatic as online, no password Type ‘evdisp’ Or use faster less sophisticated version on test: SetupProject Panoramix v17r9 Source /home/ghez/TEST/evdisp.sh Problem: Call Online Piquet daytime 15

R. Jacobsson LHCb Operation Info This is a PLASMA screen. Every hour, for one minute, a white strip runs over the screen to erase any permanent memory. This is not a fault, don’t call the online support!