DAQ Overview + selected Topics Beat Jost Cern EP

Beat Jost, Cern 2 FE/Controls/DAQ Workshop 2000 Overall Architecture Read-out Network (RN) RU Control & Monitoring RU 2-4 GB/s 4 GB/s 20 MB/s Variable latency L2 ~10 ms L3 ~200 ms LAN Read-out units (RU) Timing & Fast Control Front-End Electronics VDET TRACK ECAL HCAL MUON RICH LHC-B Detector L0 L1 Level 0 Trigger Level 1 Trigger 40 MHz 1 MHz 40 kHz Fixed latency 4.0  s Variable latency <1 ms Data rates 40 TB/s 1 TB/s Front-End Multiplexers (FEM) 1 MHz Front End Links Trigger Level 2 & 3 Event Filter SFC CPU Sub-Farm Controllers (SFC) Storage Throttle

Beat Jost, Cern 3 FE/Controls/DAQ Workshop 2000 Event Builder Architecture

Beat Jost, Cern 4 FE/Controls/DAQ Workshop 2000 Event Building Protocol qPure “push” protocol ãEach source (RU) of the event-builder sends data autonomously to a destination as soon as they are available ãThe destination is assigned through a static algorithm based on the event number. The identical algorithm runs in all sources (belonging to the same partition) -> The RUs belong to only one partition ãThe destinations (SFCs) receive the the fragments of the RUs and assemble them to form complete events. The SFCs will be configured to treat only fragments from the sources of the same partition -> The SFCs belong to only one partition ãNo communication (other than event fragments) between RUs and SFCs

Beat Jost, Cern 5 FE/Controls/DAQ Workshop 2000 Event Fragment Assembly qThe current idea is that the SFCs will be equipped with intelligent NICs (Network Interface Cards) ãThe NICs will handle all the fragment assembly to form complete events. ãThe host computer will only receive fully assembled events (or events flagged as erroneous) -> load on host reduced by ~100 (number of SFCs). SFC CPU MEM Bridge Local Bus PCI Bus SNIC Readout Network Controls Network Subfarm Network NIC

Beat Jost, Cern 6 FE/Controls/DAQ Workshop 2000 SFC Functionality qAssemble complete events qMonitor the state of the CPUs connected to it, implement dynamic load balancing among them and send raw data to ‘free’ CPUs qReceive the accepted and reconstructed events from the CPUs and send them to the storage controller qMonitor the buffer occupancy in the sub-farm and issue L1-Throttle in case buffers start getting full qProvide monitoring information to ECS

Beat Jost, Cern 7 FE/Controls/DAQ Workshop 2000 Data Flow

Beat Jost, Cern 8 FE/Controls/DAQ Workshop 2000 Readout Network Technologies Limit speed of end-node connections to ~1 Gb/s Current Candidate Technologies åATM –‘no’ smart NICs –enough performance –too expensive åMyrinet –smart NICs exist –works in principle –cheap switches, ~expensive interfaces –No buffering in switches –have to cascade switches -> have to build FIFOs to put between levels of switches (scalability) åGb Ethernet –under study –has smart NICs –prob. works –cheap –Prob. have to cascade switches, should improve with 10GbE

Beat Jost, Cern 9 FE/Controls/DAQ Workshop 2000 Other Networks qData Network ãCheapest possible compatible with performance requirements åInput rate into CPU: ~5 MB/s åOutput rate from CPU: ~25 kB/s ÜFast Ethernet should be sufficient qControls Network ãCheapest possible (prob. Fast Ethernet)

Beat Jost, Cern 10 FE/Controls/DAQ Workshop 2000 Open Issues qNon Zero-suppressed readout ãMemory on smart NICs is limited (currently 1 MB, including code) ãPossible solution could be to do fragment assembly in the SFC itself? (Not too nice…)