CERN R. Jacobsson CERN 16 th IEEE NPSS Real Time Conference, Beijing, China, May 10–15, LHCb Readout System and Real-Time Event Management - Emphasis on the architectural and functional aspects not the particular technologies - Commissioning and status “An Integrated Control System for the LHCb Experiment”, CMS1-1, Thursday “Handling Online Information in the LHCb Experiment”, CMSP-8, Tuesday Federico Alessio, CERN, in place of Richard Jacobsson, CERN, on behalf of the LHCb Online Team

CERN R. Jacobsson CERN 16 th IEEE NPSS Real Time Conference, Beijing, China, May 10–15, 2009 Place in the World 2 Underground cavern at ~100m depth LHC in 50 institutes in 16 countries 700 people ~10m ~20m

CERN R. Jacobsson CERN 16 th IEEE NPSS Real Time Conference, Beijing, China, May 10–15, 2009  Challenge of an hadronic precision experiment  Particle multiplicity  Very large background  Small ratio of interesting B-meson decays O (10 −3 –10 −9 )  High statistics  Design luminosity L ~2x10 32 cm -2 s -1 (1/50 Atlas&CMS)  10 MHz visible interactions  100 kHz bb-event rate  ~ bb-pair / year at LHCb  2 kHz event storage rate LHCb – Single-arm forward spectrometer 3 p-beam Efficient trigger for many B decay topologies Muon system, ECAL+Preshower, HCAL, Vertex Locator Efficient particle identification RICH Good decay time resolution Vertex Locator 5 mm from beam Good mass resolution Tracker and Magnet B 1 cm

CERN R. Jacobsson CERN 16 th IEEE NPSS Real Time Conference, Beijing, China, May 10–15, 2009 Trigger Architecture 4 High-Level Trigger 2 kHz Level -0 L0 e,  40 MHz 1 MHz L0 had L0  ECAL Alley Had. Alley Global reconstruction 30 kHz HLT1 HLT2 Muon Alley Inclusive selections ,  +track,  Exclusive selections Storage: Event size ~35kB  Level-0 Hardware Trigger 40 MHz  1 MHz  Search for high-pT , e, , hadron candidates  Latency 4  s, i.e. pipelining 160 events  High Level Trigger Farm with O (1000) quadcores  HLT1: Confirm L0 candidate with more complete info, and add impact parameter and lifetime cuts  HLT2: global event reconstruction + selections  Processing time available O (milliseconds)  Output rate 2 kHz  HLT needs to know how L0 is configured  How to distribute to 1000 nodes simultaneously in seconds when optimizing parameters during LHC fill?

CERN R. Jacobsson CERN 16 th IEEE NPSS Real Time Conference, Beijing, China, May 10–15, 2009  Evolution: Eliminating trigger levels Readout Architecture RT2003 5

CERN R. Jacobsson CERN 16 th IEEE NPSS Real Time Conference, Beijing, China, May 10–15, LHCb Readout System RT2009 SWITCH High-Level Trigger farm Detector Timing & Fast Control SWITCH READOUT NETWORK LHC clock Event Requests Event building Front-End CPUCPU CPUCPU CPUCPU CPUCPU CPUCPU CPUCPU CPUCPU CPUCPU CPUCPU CPUCPU CPUCPU CPUCPU CPUCPU CPUCPU CPUCPU CPUCPU CPUCPU CPUCPU CPUCPU CPUCPU CPUCPU CPUCPU CPUCPU CPUCPU Readout Board VELO STOT RICH ECalHCalMuon SWITCH Mon. farm CPUCPU CPUCPU CPUCPU CPUCPU Readout Board FE Electronics L0 trigger L0 Trigger 320 ROBs 24 Gb/s 4 Gb/s 50 TB with 70 MB/s 3000 GbE ports 35 GB/s 50 subfarms of ~40 nodes Shielding wall 5000 optical/analog links O (4 Tb/s) Offline See “Controlling a Large Trigger Farm Using Industrial Tools”, OPF2, Wednesday and “Management of the LHCb Readout Network”, OPF-3, Wednesday 11.00

CERN R. Jacobsson CERN 16 th IEEE NPSS Real Time Conference, Beijing, China, May 10–15, 2009  Design principle: A limited number of protocols and technologies  Readout network: GbE IPv4  Farm-to-storage: GbE TCP/IP  Overhead reduction  ~ 120 Bytes  Overhead on IP/Eth is 58 Bytes   Pack fragments of several consecutive events = MultiEvent Packet (MEP) Protocol Data Transfer Protocols 7

CERN R. Jacobsson CERN 16 th IEEE NPSS Real Time Conference, Beijing, China, May 10–15, 2009 Calib  Readout control has two aspects:  Control of data transfer MEP Packing Destination assignment for event building and HLT Load balancing Partitioning for parallel activities  Management of event types and associated destinations/processing Physics triggers Calibration triggers Luminosity triggers Non-zero suppressed data Luminosity scans (Vernier scan)  Driven and managed by the LHCb Timing and Fast Control System  Responsible for distributing timing, trigger and synchronous and asynchronous information to entire readout system  FPGA based master: Readout Supervisor Also performs rate control and generates all types of auto-triggers and calibration sequences Centralized Readout Control 8

CERN R. Jacobsson CERN 16 th IEEE NPSS Real Time Conference, Beijing, China, May 10–15, 2009 TFC Master Info Flow 9 Readout Supervisor LHC accelerator Beam Phase and Intensity Monitor Subdetectors HLT farm L0 trigger RS event bank Multi Event Requests Bunch currents Clock/orbit, UTC, LHC Info HW and run parameters Run statistics Detector status L0 Decision RO Electronics Trigger Throttle TFC FE Electronics TFC

CERN R. Jacobsson CERN 16 th IEEE NPSS Real Time Conference, Beijing, China, May 10–15, 2009 Data Transfer Control Readout network: Push protocol with passive pull mechanism  MEP Packing Control by Readout Supervisor: 1. Trigger Type transmitted synchronously to all Readout Boards  Determines processing in ROB and synch check 2. IP Destination transmitted synchronously to all Readout Boards for each MEP  Reception triggers closure and sending of last MEP  Interleaving Trigger Type and Destination determines MEP packing  Dynamic packing factor depending on event types  Trigger type determines destination type (HLT, calibration, etc)  Farm Destination for next MEP chosen based on Credit scheme 1.Farm nodes transmits Event Requests with a Credit(“declare as ready to receive”) 2.Readout Supervisor round-robin in Destination Table 3.Select destinations with positive Credit and decrement  Effectively, load balancing of readout network and HLT Farm  Static load balancing of network by organization of Destination Tables in Readout Supervisor  Dynamic load balancing of HLT farm nodes  Nodes request events when ready  In all cases, event loss is minimized in case of failing/blocked/slow links or nodes  Ultimately, credit scheme regulate readout rate when low on credits Calib 10

CERN R. Jacobsson CERN 16 th IEEE NPSS Real Time Conference, Beijing, China, May 10–15, 2009 Event Management  Readout Supervisor Data Bank appended to each event  Book-keeping  Run Number regroups events taken with same configuration  UTC Time to correlate event with Conditions DB  Coarse quality bits from each sub-detector which may be used by HLT  Trigger Type and Calibration Type determine type of processing  Window of consecutive 25ns clock cycles which should be processed together (see later)  HLT needs to know how L0 is configured  How to distribute to 1000 nodes simultaneously?  Trigger Configuration Key distributed in RS data bank allows optimizing the trigger parameters in real-time during LHC fill 11

CERN R. Jacobsson CERN 16 th IEEE NPSS Real Time Conference, Beijing, China, May 10–15, 2009  Detector Performance, Readout Performance and Data Quality  Histogram collected from all systems  Monitoring Farm spying on event streams at best effort Also produces histograms from an online reconstruction at best effort  Histogram analysis  Automatic checks and alarms  Histogram inspected by Data Manager Shifter Online Monitoring 12 Calib Histogram Handling (ECS) Automatic Histogram Analysis Interactive Presenter

CERN R. Jacobsson CERN 16 th IEEE NPSS Real Time Conference, Beijing, China, May 10–15, 2009 LHCb Data and Process Flow 13 Online 35 = 70 MB/s or 2 GB file(~60 kevts) / 30s Offline Tier-0 / 1 Tier-2 Storage (CASTOR) Reconstruction Simulation Stripping Analysis HLT Bulk Stream Express Stream 5 Hz Storage (CASTOR) Reconstruction Calibration Alignment QC Offline Control Room 20h/file, 20 kB/evt Bookkeping Data and Production Management Data Quality Checking Test Jobs QC Run Info Data Quality

CERN R. Jacobsson CERN 16 th IEEE NPSS Real Time Conference, Beijing, China, May 10–15, 2009  Real-Time scheme to validate High Level Trigger, data flow and offline processing =  Be ready to receive, process and analyze 7 million events in the first hour of collisions No-beam System Tests 14 MEP Requests Injector Simulated events  Replacing detector with injection of 10 8 “accepted” simulated events real-time in Online system at HLT rate (2 kHz) See poster “High-speed Data Injection for Data Flow Verification in LHCb, CMSP-28, Tuesday 16.40

CERN R. Jacobsson CERN 16 th IEEE NPSS Real Time Conference, Beijing, China, May 10–15, 2009 Commissioning LHCb 15 Two years of intense work 2006 – 2008 with the aim to:  Operate the detector AND people as a unit with common tools  Bring all components (sub-detectors and service systems) to operational state.  Define, implement and validate the tools and procedures needed to run the detector as a whole  Organise the activities to reach the ready state in time  Understand and calibrate the detector  Test pulses, radioactive sources  Cosmics  LHC injection tests  First days with beam  Operate with two shifters  Operating the whole detector from one console  Understandable high-level tools for diagnostics, alarms and data monitoring  Homogeneity in the system  Shifter training  On-call Experts for all sub-systems and sub-detectors  Reach operational efficiency  Starting (<10min) and restarting (<1 min) rapidly and smoothly Crucial tool: Readout and processing of sets of consecutive 25ns clock cycles around “detector activity” trigger Time and space alignment Leakage in preceding and subsequent clock cycles Optimize signal over spill-over

CERN R. Jacobsson CERN 16 th IEEE NPSS Real Time Conference, Beijing, China, May 10–15, 2009 Commissioning with Cosmics 16  Challenge: LHCb geometry is NOT well suited for cosmics…  “Horizontal” cosmics well below a Hz  Still 1.6x10 6 good events (July – September 2008 ) recorded for the large sub-detectors

CERN R. Jacobsson CERN 16 th IEEE NPSS Real Time Conference, Beijing, China, May 10–15, 2009 First Glimpse of LHC Protons 17 Sector Tests Aug-Sep 2008  Beam 2 dumped on injection line beam stopper (TED) TED TI8 LHC Vertex Locator Scintillator Pad Detector Muon

CERN R. Jacobsson CERN 16 th IEEE NPSS Real Time Conference, Beijing, China, May 10–15, 2009 Experience with 150 Joules 18  RICH2 “photon blast” LHC Turn-On - September 10, 2008  An all too short honeymoon with LHC…  Contrary to what we wish for the future, the splashes were Highly Desired Events!

CERN R. Jacobsson CERN 16 th IEEE NPSS Real Time Conference, Beijing, China, May 10–15, 2009 Readout Architecture RT See talk “A 40 MHz Trigger-free Readout Architecture for LHCb”, RTSA2, Tuesday  Upgrade plans already well underway: Another level of Natural Selection

CERN R. Jacobsson CERN 16 th IEEE NPSS Real Time Conference, Beijing, China, May 10–15, 2009  LHCb has become an operational experiment “waiting” for beam  Readout system mature with advanced readout control and event management  Reached a good compromise between use of COTS and custom electronics  Commissioning  Still (too…) many experts in the control room…necessary ones or not…  Injection tests end of August 2008 gave the first ever LHC-induced tracks  Beam-collimator shots were obviously the high-light of 2008, “unfortunately”…  LHCb very ready for the long run with LHC COLLISIONS  Upgrade path is towards full 40 MHz trigger-free readout  Becoming popular concept, cmp future acelerators  Good topic for next years Real Time Conference Conclusions 20