Latest ideas in DAQ development for LHC B. Gorini - CERN 1

Guiding principles in latest studies Replace obsolete modular electronics technologies (e.g. VME) Whenever possible opt for COTS components Favor S/W over F/W when performance requirements allows it 2

Modular electronics replacement A clear choice for VMS replacement: ATCA ATCA is rather a packaging standard – High availability – Protocol agnostic backplane – … It offers less than VMS in terms of common development environment A DAQ specific common platform is useful – E.g. the SLAC RCE for ATCA system deployments in high energy physics 3

The RCE Modular DAQ based on generic ubiquitous building blocks: – Reconfigurable Cluster Elements (RCE) Mezzanines based on large FPGAs and System on Chip (now ARM processor) very high data bandwidth at input and multi-Gbps Ethernet output. – Cluster Interconnects (CI) On a generic ATCA carrier, the Cluster On Board (COB) 4

ATLAS, ALICE and LHCb are considering to move away from modular electronics for their back-end – The plan is to receive front-end links on PCIe cards housed on PCs Latest ideas: less modular electronics? 5 Detector F/E Boards F/E Links

LHCb original plan 6

LHCb current plan 7

The PCIe40 Board 8

9 FE Today Run 4 Present ATLAS DAQ: Custom electronic components ROD Custom point-to-point links ReadOut Driver HLTPU Ethernet PCs High-Level Trigger Farm ~1500 nodes ~100 machines ROS Point-to-point S-links ReadOut System (Data Buffer) 100 kHz / ~200 GB/s ~1800 links 40 MHz

Run 4 10 FE Readout HLTPU HPC Network Versatile Link, GBT, LpGBT PCs 2023 High-Level Trigger Farm HPC Network COTS network technology Proposed ATLAS DAQ: 40 MHz ~10,000 links ~200 systems less than 10 TB/s

11 Dynamic routing rules, automatic failover and load balancing Support multiple GBT modes, virtual e-links Bi-directional connection with F/E Trigger signal and LHC clock synchronous distribution Routing of multiple traffic types: physics events, detector control, configuration, calibration, monitoring FELIX functionality Multi-cast, cloning, QoS GBT: Rad-hard F/E link technology developed at CERN E-link: variable-width logical link on top GBT. Can be used to logically separate different streams on a single link.

Advantages of PCIe solutions Less custom electronics, more COTS components – Possibility of separate upgrade of FPGA, NICs, CPUs – Can follow processor computing power evolution Fully programmable F/E-B/E connectivity – Not limited to crate/shelf backplane capability Scalable architecture – Can add readout nodes as needed Dynamic load balancing – Programmable data routing algorithms Increased fail safety – Automatic recovery of readout node crashes Main questions: – Probably increased latency – Most data processing in processors rather than FPGAs 12

FELIX Demonstrator System 13 FELIX FPGA Card Memory Large Buffers per group of elinks CPU Custom Device Driver FELIX Application PCIe Gen- 3 8 lanes DMA Config GBT Optical Links MSI-X DMA TTC FMC 64 Gb/s NIC Optical Links 64 Gb/s 24 – 48 links 2 – 4 40-Gb/s ports Elink router “TTC” is the Timing, Trigger and Control System PCIe Gen- 3 8 lanes

FELIX development Platform 14 HiTech Global PCIe development board with TTCfx Xilinx Virtex-7 PCIe Gen-2/3 x8 24 bi-directional links SuperMicro X10DRG-Q 2x Haswell CPU, up to 10 cores 6x PCIe Gen-3 slots 64 GB DDR4 Memory Mellanox ConnectX-3 VPI FDR/QDR Infiniband 2x10/40 GbE onnectx_3_vpi

Perspectives LHCb is advancing in the design of their custom PCIe board (PCIe40) – Were expecting first prototype by last December ALICE plan is to use LHCb PCIe H/W ATLAS will complete the implementation of a demonstrator based on a commercial board – Initial design review planned after completion – Decision whether to develop custom board or stay with a commercial one will follow Commercial market is becoming latency aware – Hedge funds are driving (at least) a big computer manufacturer to develop similar FPGA/PCIe card to support high frequency trading requirements There is room for considering a common solution – Ideally purely based on COTS (including PCIe boards) – Ideally only software development (F/W will be needed only for readout link interface) – Following development (Gen 4) or replacement (?) of PCIe for higher throughput reach 15