CERN R. Jacobsson Between LHC and the Grid - Aspects of Operating the LHC Experiments – T. Camporesi, C.Clement, C. Garabatos Cuadrado, L. Malgeri, T. Pauly, R. Jacobsson 1 *Already opened by Stefano Radaelli:

CERN R. Jacobsson Experiments have never in the past been so tightly connected to the accelerator What’s so special about LHC? 1.Stored energy 2 x 360 MJ and fragile detectors Experiment protection Operational communication and procedures  Reliability requires direct communication interfaces 2.High interaction rate and large events size Fast and reliable readout, storage and transfer to offline processing Fast feedback from Data Quality checking 3.High intensity proton collider and sensitive detectors Monitoring and understanding/analyzing/optimizing experimental conditions Luminosity determination 4.Long-term detector stability and aging due to radiation Automatic Calibrations 5.Many years of 24h operation with few people and non-experts Operating the whole detector from one console  Understandable high-level tools for diagnostics, alarms and data monitoring  Homogeneity in the system  Shifter training These lectures will describe largely how we have addressed and solved these 2

CERN R. Jacobsson l At all levels we need tools to predict, prevent, and limit “non-bb” exposure  Framework to combine information and understand correlations between background, beam characteristics and machine settings  Dedicated archiving and analysis tools  We need to look at the entire LHC machine 3 Instantaneous damage Beam Interlock Background ………….…………Trigger rates…………………… ………...………..Poor data quality………………… ……………………..….Single event upsets………. …………..……Accelerated aging……….………… …………………Long-term damage……………..... Online monitoring Accumulated dose and Luminosity Background Beam characteristics Machine settings Halo/beam-gas/………………....…….scraping……………....Beam incident

CERN R. Jacobsson 4

CERN R. Jacobsson 5

CERN R. Jacobsson 6 Readout Supervisor LHC accelerator Beam Phase and Intensity Monitor Subdetectors High Level Trigger Farm L0 trigger RS Event Bank Multi Event Requests Bunch currents Filling scheme Clock/orbit, UTC, LHC Info Filling Scheme Lumi Scan HW and run parameters Run statistics Online Luminosity Detector status L0 Decision RO Electronics Trigger Throttle Timing/Synch.ctrl FE Electronics Timing/Synch.ctrl  Readout Supervisor is the intelligent readout controller The “Orchestra Director” Completely based on Field Programmable Gate Logic devices Pool of RSs for autonomous stand-alone running of any subdetector

CERN R. Jacobsson 7  Detector Performance, Readout Performance and Data Quality Histogram collected from all systems Monitoring Farm spying on event streams at best effort  Also produces histograms from an online reconstruction at best effort Histogram analysis  Automatic checks and alarms Histogram inspected by Data Manager Shifter Calib Histogram Handling (ECS) Automatic Histogram Analysis Interactive Presenter

CERN R. Jacobsson 8 Online 35 = 70 MB/s or 2 GB file(~60 kevts) / 30s Offline Tier-0 / 1 Tier-2 Storage (CASTOR) Reconstruction Simulation Stripping Analysis HLT Bulk Stream Express Stream 5 Hz Storage (CASTOR) Reconstruction Calibration Alignment QC Offline Control Room 20h/file, 20 kB/evt Bookkeping Data and Production Management Data Quality Checking Test Jobs QC Run Info Data Quality

CERN R. Jacobsson  Typical Physics fill cycle  “Machine Development” (MD) periods Some scheduled days/weeks the LHC will be operated not to produce physics data (Stable Beam) but to study machine physics and improve performance  LHCb will not take data, but safety should be ensured. A shift crew will be needed still, but with lower activities. 9

CERN R. Jacobsson Calib  Readout control has two aspects: Control of data transfer  MEP Packing  Destination assignment for event building and HLT  Load balancing  Partitioning for parallel activities Management of event types and associated destinations/processing  Physics triggers  Calibration triggers  Luminosity triggers  Non-zero suppressed data  Luminosity scans (Vernier scan)  Driven and managed by the LHCb Timing and Fast Control System Responsible for distributing timing, trigger and synchronous and asynchronous information to entire readout system FPGA based master: Readout Supervisor  Also performs rate control and generates all types of auto-triggers and calibration sequences 10

CERN R. Jacobsson 11

CERN R. Jacobsson  In LHCb we synthesize Machine/Beam modes into 9 “Internal LHC States” INJECTION, RAMP, PHYS_ADJUST, PHYSICS, ADJUST, DUMP EOF (End-Of-Fill), NO_BEAM, MD (Machine Development) 12 VELO allowed IN Software “handshakes” between LHC and experiments INJECTION, ADJUST and BEAM DUMP

CERN R. Jacobsson Fill sequence for physics fill in LHCb language: NO_BEAM (Injection Permit = FALSE, Any state of LHCb, Internal clock) INJECTION (Injection Permit = TRUE, VELO out, External clock) RAMP (Injection Permit = FALSE) PHYS_ADJUST PHYSICS (VELO in) (ADJUST) (VELO out) DUMP (VELO out, to be changed) EOF (Internal Clock, Calibrations) 13 Handshake for Injection Handshake for Dump (Only when directly from PHYSICS) Handshake for Adjust

CERN R. Jacobsson 14 PROBLEM WARNING READY PREPARE STANDBY IMMINENT VETO READY OK LHC: NO_BEAM INJECTION LHCb: STANDBY Confirm VETO Experiment Control System GET READY READY  General rule: Get ready in ~5 minutes Confirm RAMP

CERN R. Jacobsson 15  Real-Time scheme to validate High Level Trigger, data flow and offline processing =  Be ready to receive, process and analyze 7 million events in the first hour of collisions MEP Requests Injector Simulated events  Replacing detector with injection of 10 8 “accepted” simulated events real-time in Online system at HLT rate (2 kHz)  Also allows testing new HLT versions with minimum bias events

CERN R. Jacobsson 16 Two years of intense work 2006 – 2008 with the aim to:  Operate the detector AND people as a unit with common tools  Bring all components (sub-detectors and service systems) to operational state.  Define, implement and validate the tools and procedures needed to run the detector as a whole  Organise the activities to reach the ready state in time  Understand and calibrate the detector  Test pulses, radioactive sources  Cosmics  LHC injection tests  First days with beam  Operate with two shifters  Operating the whole detector from one console  Understandable high-level tools for diagnostics, alarms and data monitoring  Homogeneity in the system  Shifter training  On-call Experts for all sub-systems and sub-detectors  Reach operational efficiency  Starting (<10min) and restarting (<1 min) rapidly and smoothly Actually all achieved for pilot run 2009! Crucial tool: Readout and processing of sets of consecutive 25ns clock cycles around “detector activity” trigger Time and space alignment Leakage in preceding and subsequent clock cycles Optimize signal over spill-over

CERN R. Jacobsson 17 LHCb did use cosmics but for obvious geometrical reasons not sufficient…  Beam 2 dumps on injection line beam stopper (TED) ideal.. But backwards…! TED TI8 LHC Vertex Locator Scintillator Pad Detector Muon

CERN R. Jacobsson  Ideal for monitoring bunch profiles, opimization and absolute luminosity 18 Beam 2 Beam 1 Beam-beam predicted from beam-gas Beam-beam

CERN R. Jacobsson 19 +/-5 mm 30 mm Beam 30 mm  VELO is a Movable Device: Out position: 35mm from beam line In nominal data taking position: 5mm from beam line Data taking position determined during Open Tracking by determining luminous region before moving in for every fill May only leave its “garage position” during STABLE BEAM

CERN R. Jacobsson 20  LHCb Main Background Monitor (BCM) Diamond based Beam Condition Monitor Running sums 80  s and 1280  s Directly connected to LHC Beam Dump System (LBDS)  Also hardware interface for Injection Permit  High-sensitivity and high time resolution monitor Scintillator based Beam Loss Monitor 25ns integration and fast readout Used mainly to understand background and discover problems early (collimator settings, aperture, beam-gas, injection etc) 20 BCM