1. LHCb readout infrastructure NA62 TDAQ WG Meeting April 1 st, 2009 Niko Neufeld, PH/LBC

2. General features Two trigger levels: L0 (hardware), HLT (software) L0 fixed latency 4 us, no ZS, deterministic buffer status, buffers protected by central trigger distribution system (TFC) using emulation Frontend links (G- link/GOL @ ~ 160 MB/s) Uniform Readout-board TELL1 (up to 24 input links) 4 output links (Gigabit Ethernet 1000 BaseT) Gigabit Ethernet network Farm (up to ~1500 servers) 2 LHCb Readout Infrastructure - NA62 TDAQ

3. LHCb Frontend Architecture 3 LHCb Readout Infrastructure - NA62 TDAQ

4. LHCb DAQ 40 GB/s 80 MB/s Average event size 40 kB Average rate into farm 1 MHz Average rate to tape 2 kHz 4 LHCb Readout Infrastructure - NA62 TDAQ

5. DAQ features Push-protocol Message coalescing: data for a single trigger in LHCb are small: (ca 100 bytes)  put several together into one message on the network Farmnodes announce availability to TFC TFC decides where to send and how many events to put into one packet After TELL1 all hardware is commercial off the shelf (COTS) albeit high-end Switch (main router): Force10 E1200 (up to 1260 Gigabit Ethernet ports) Separated networks for DAQ and controls important for robustness & monitoring Farmnodes (currently 2.5 GHz, 1 to 2 GB/core, 8 cores, Intel Harpertown) 5 LHCb Readout Infrastructure - NA62 TDAQ

6. What’s nice? Almost only COTS components always profit from industry progress (Moore’s law) and price decline Very few different components few experts needed Scalable: farm is only limited by cooling and power, network is limited by size of F10 switch, but could buy a second chassis 6 LHCb Readout Infrastructure - NA62 TDAQ

7. Hardware / technology choices Ethernet over copper: longest distance in the LHCb Online system is 36 m (standard allows 90 m) – tremendous cost advantage! Force10. At the time of our evaluation (~ 2006), this switch was the only one which could withstand our traffic (because of its superior buffering) PCs: normally go for best price/performance, new servers can easily handle the traffic (up to 6 Gbit/s), some tuning of kernel parameters is necessary 7 LHCb Readout Infrastructure - NA62 TDAQ

8. TELL1 from the DAQ point of view We had very few real problems with the hardware except: Ethernet problem of the CCPC (wrong connector): fixed by bringing active switches close to the crates Spurious VME resets (now fixed) Software: CCPC software complete and stable (most files have not been touched in years): I2C, JTAG, local bus, allows for interrupt handling from FPGAs (not used but working), the libraries are protected against concurrent access to hardware resources No software upgrades planned, except following OS trends (will move to SLC5 eventually) and some performance improvements in the main DIM-server TELL1 problems in the readout: Misconfiguration (now rare) desynchronisation: dirty fibres, glitches, front-end, etc… LHCb Readout Infrastructure - NA62 TDAQ 8

9. Controls In LHCb both slow-controls (DCS) and run-control are based on PVSS and the JCOP framework TELL1 is fully integrated into PVSS (as framework component): software layering is quite modular Custom hard- and software is usually integrated into PVSS using DIM LHCb Readout Infrastructure - NA62 TDAQ 9

10. Problems Commercial hardware: harddisks, power-supplies, ethernet switch ports, memory barettes Software: funny problems with BMCs / booting, BIOS oddities, All these are related to the fact that we have a lot and that the hardware is cheap: maintenance Large system with many components requires quite a few people to run 10 LHCb Readout Infrastructure - NA62 TDAQ

11. Hardware inventory 290 TELL1s in production 1 x Force10 E1200 switch 50 x HP3500 aggregation switch (farm/DAQ) 70 x HP2650 aggregation switch (control) 100 SuperMicro Twinserver (E4) 350 DELL blade servers (M605) ~ 120 DELL 1425 SC servers (controls) LHCb Readout Infrastructure - NA62 TDAQ 11