TELL1 A common data acquisition board for LHCb Guido Haefeli, University of Lausanne Guido Haefeli

Outline LHCb readout scheme LHCb data acquisition Optical links Event building network Common readout requirements Trigger rates Buffers Bandwidth Data flow on the board Synchronization Level 1 trigger pre-processing and zero-suppression Higher level trigger processing Gigabit Ethernet interface Board implementation FPGAs Level 1 buffer Higher level trigger multi event packet buffer Summary

LHCb trigger system L0: fully synchronous and pipelined fixed latency Pile-Up Calorimeter Muon L1: software trigger with maximal latency VELO TT (Outer Tracker) HLT: software trigger Access to all sub-detectors

LHCb data acquisition Front End of detectors in cavern 60-100m TELL1 in counting room

Optical link implementation

Event building network TELL1 HLT Traffic 40KHz MEP  /16 Mux  x8 Level-1 Traffic 1.11MHz MEP /32 Mux x2 SFC /94 Front-end Electronics FE FE FE FE FE FE FE FE FE FE FE FE TRM Multiplexing Layer Commercial network equipment Switch Switch Switch Switch Switch Readout Network L1-Decision Sorter 94 Links 7.1 GB/s … 94 SFCs SFC Switch CPU SFC Switch CPU SFC Switch CPU SFC Switch CPU CPU Farm Gb Ethernet Level-1 Traffic Mixed Traffic HLT Traffic ~1800 CPUs

Trigger rates and buffering Max. L0 Accept rate = 1.11 MHz Max. L1 Accept rate = 40 KHz L0 buffer is implemented on the Front End fixed to be 160 clock cycles ! L1 buffer 58254 events which equals to 52.4us !

Bandwidth requirements Input data bandwidth for a 24 optical link motherboard Optical receiver 24 fibres x 1.28 Gbit/s  30.7Gbit/s Analog receiver 64 channels x 10-bit @ 40 MHz  25.6 Gbit/s L1 Buffer Write data bandwidth 30.7 Gbit/s Read data bandwidth 4 Gbit/s DAQ links 4 Gigabit Ethernet links ECS 10/100 Ethernet for remote control

A bit of history L1 trigger scheme changed During the last year the maximal L1T latency has increased from 1.8ms to 52ms (x32). This forces the change SRAM FIFO  SDRAM Detectors added to L1T (TT, OT) and potentially others Decreasing cost for the optical links  data processing is done in the counting room More and more functionalities on the read out board because no Readout Unit and no Network Processors! The event fragments are packed in so called “Multi Event Packets” MEP to optimize ethernet packet size and packet rate. The acquisition board adds IP destination, does Ethernet framing and transmit data buffering …)

How can we use a common read out ? FE FE FE FE Adaptation to two link system is possible with receiver mezzanine cards. A-RxCard A-RxCard O-RxCard FPGAs allow the adaptation for different data processing. PP-FPGA PP-FPGA PP-FPGA PP-FPGA L1B L1B L1B L1B SyncLink-FPGA Sufficient bandwidth for the entire acquisition path ECS TTCrx RO-Tx FEM Mezzanine card for detector specific needs ECS TTC L1T HLT L0 and L1 Throttle

Advantages being common ! FE FE FE FE Solution for new system requirements can be found regarding the architecture of one readout board only. A-RxCard A-RxCard O-RxCard PP-FPGA PP-FPGA PP-FPGA PP-FPGA Cost reduction due bigger quantity serial production (300 board for LHCb). L1B L1B L1B L1B SyncLink-FPGA Reduce maintenance cost with a single system. ECS TTCrx RO-Tx FEM ECS TTC L1T HLT L0 and L1 Throttle A lot of common software interfaces.

O-RxCard (Mezzanine card) Cluster encapsulation L1T dataflow X12 PP-FPGA SyncLink-FPGA O-RxCard 32 PP-FPGA 32 PP-FPGA 32 broadcast TTC PP-FPGA ECS FIFO 32 L1T IP RAM L1T DEST FIFO L1T Link FIFO 32 FIFO O-RxCard (Mezzanine card) X12 8 L1PP FIFO L1T Framer RO-Interface POS-Level 3 Data Link L1T MEP Buffer 32 FIFO 16 FIFO 32 64 64 64 32 32 8 L1PP FIFO FIFO 32 64 Kbyte 1 Kbyte ID Check Shared data path for 2 channel RO-TxCard @ 100 MHz 16 Sync FIFO Cluster encapsulation 8 L1PP FIFO 1 Kbyte 32 L1B 8 L1PP FIFO 64 KByte internal SRAM @ 100 MHz Link DDR @80MHz @80MHz

HLT ZeroSupp Event Encaps. O-RxCard (Mezzanine card) HLT dataflow 0.9us/event 20us/event 320us/MEP PP-FPGA, Altera Stratix 1S20, 18K LE SyncLink Stratix 1S25, 25K LE X12 16 FIFO 32 O-RxCard PP-FPGA broadcast TTC 16 FIFO 32 ECS PP-FPGA HLT IP RAM HLT DEST FIFO HLT ZeroSupp Event Encaps. 16 FIFO 32 Sync FIFO ID Check 32 16 FIFO 32 FIFO 32 HLT Framer RO-Interface POS-Level 3 X12 HLT MEP Buffer O-RxCard (Mezzanine card) 32 32 32 32 Sync FIFO ID Check 32 16 FIFO 32 L1B Data link 32 FIFO 16 FIFO 32 4 Kbyte Shared data path for 2 channel RO-TxCard @ 100 MHz 1 Kbyte 1 Kbyte FIFO 32 16 Sync FIFO ID Check FIFO 32 16 Sync FIFO Link DDR @80MHz 1 Mbyte external QDR SRAM @ 100 MHz @80MHz @120MHz @120MHz 64 KEvent DDR SDRAM

LOW V POWER HLT MEP TTCrx FEM PP FPGA O-RxCard L1B Glue Card CCPC Sync Link CCPC A-RxCard RO-TxCard O-RxCard L1B TTCrx HLT MEP FEM LOW V POWER Glue Card

DDR bank data signal layout Equal length signal traces are required for DDR SDRAM Implementation (4 x 48-bit wide bus @ 240MHz data transfer rate ! (46 Gbit/s) Guido Haefeli

14cm

Summary After evaluating different concepts for data processing and acquisition a common read out board for LHCb has been specified and designed. It serves for 24-optical with a data transfer rate of 1.28 Gbit/s each. The board implements data identification, L1 buffering an zero suppression. It is made for the use with standard Gigabit ethernet equipment as switches and routers.