LECC2003 AmsterdamMatthias Müller A RobIn Prototype for a PCI-Bus based Atlas Readout-System B. Gorini, M. Joos, J. Petersen (CERN, Geneva) A. Kugel, R. Männer, M. Müller, M. Yu (University of Mannheim) B. Green (Royal Holloway University London) G. Kieft (NIKHEF, Amsterdam)

LECC2003 AmsterdamMatthias Müller 2 Outline Overview The Atlas Readout Sub-System (ROS) PCI based Atlas ROS The RobIn Prototype Measurements Conclusions

LECC2003 AmsterdamMatthias Müller 3 Overview PCI-based-ROS is one of the two implementation option of the Atlas ROS.  Uses custom PCI board for receiving / buffering data  RobIn  Host is a PC with multiple PCI-Buses  Gigabit Ethernet connection to LVL2 and EF  PC running multithreaded Software and Master-DMA based PCI messaging scheme Data request rates of measured Full scale system achieves LVL1 Rate of (with GE Net I/O)

LECC2003 AmsterdamMatthias Müller 4 Atlas Readout Subsystem Overview Buffers detector data while LVL2 computes trigger decision 1600 links from detector up to 160 MB/s input bandwidth, 100kHz input rate. 2 kHz output to LVL2 on request via Gigabit Ethernet Output to Event Filter on event accept (~3kHz) LVL2 EF ROS

LECC2003 AmsterdamMatthias Müller 5 VME bus RCPRCP RODROD RODROD RODROD RODROD Config & Control Event sampling & Calibration data … PCI bus ROBIN NIC Gigabit Ethernet links LVL2 & Event Builder Networks Alternative data paths ROD Crate Processor ROLs Data 90 crates (~40 racks) 144 4U PCs (~15 racks) 1600 links (HOLA S-link, 160 MByte/s per link) In USA15 (underground) In SDX15 (at surface) Atlas Readout Subsystem

LECC2003 AmsterdamMatthias Müller 6 PCI based Atlas ROS: Hardware Available: 2 GHz, 2.4 GHz and 3 GHz Xeon PC OS: Linux CERN RedHat 7.3, kernel (patched) 532MB/s CPU (2.4GHz) Mem DDR RAM PCI 64bit/66MHz SCSI 2xFE/GE Slot 1 Slot 2 Slot 3 Slot 4 Slot 5Slot 6 ~2GB/s 532MB/s PCI 64bit/66MHz RobIn GEth

LECC2003 AmsterdamMatthias Müller 7 PCI based Atlas ROS: Software ROS software multi-threaded Fragment Manager interface for RobIn hardware abstraction

LECC2003 AmsterdamMatthias Müller 8 The RobIn Prototype

LECC2003 AmsterdamMatthias Müller 9 The RobIn Prototype (2) Requests to RobIn sent  by PCI single cycles (data requests)  by PLX Bus Master DMA (clear requests) Event data from RobIn:  FPGA sends fragment without first word  First word transmitted finally to signal end-of-transfer

LECC2003 AmsterdamMatthias Müller 10 Measurements Initially RobIn Prototype not available All presented measurements (except one) with alternative RobIn hardware  MPRACE1 MPRACE1:  Common purpose PCI based FPGA Co-Processor  FPGA and PCI bridge identical to RobIn Prototype  FPGA only board  no PowerPC processor available.  Implementing the same PCI messaging as the RobIn Prototype Measurements on three different PCs: a 2GHz Xeon, a 2.4GHz Xeon and a 3GHz Xeon

LECC2003 AmsterdamMatthias Müller 11 Measurements - Multi-threading - Bare data request performance with 1 RobIn, no I/O to Gigabit Ethernet Variation of Request Handler threads shows maximum at 14

LECC2003 AmsterdamMatthias Müller 12 Measurements - Fragment Size Dependency - MPRACE: Up to 512 bytes: fix request overheads overlap the returning fragment data transmissions from the RobIn.  very small fragment size dependency RobIn Prototype: comparison with MPRACE seems to be valid, up to 1kB no fragment size dependency

LECC2003 AmsterdamMatthias Müller 13 Measurements - Influence of DC I/O - 4 ROLs per RobIn (MPRACE) emulated Network I/O to LVL2 and EF reduce performance by a factor of 3.

LECC2003 AmsterdamMatthias Müller 14 Measurements - DC I/O and CPU scalability - 4 ROLs per RobIn (MPRACE) emulated Moving towards a 3 GHz PC improves performance by ~25%.

LECC2003 AmsterdamMatthias Müller 15 Conclusions Max. request performance per RobIn is 170 kHz (1kB fragment size). “Standalone” ROS can handle 12 ROLs on 3 RobIns with 300 kHz LVL1 input rate. Full scale ROS System (3GHz Xeon PC) handles 130 kHz LVL1 input rate (> Atlas requirements) First measurements with RobIn Prototype confirm the results obtained with an earlier prototype (MPRACE).

LECC2003 AmsterdamMatthias Müller 16 RobIn (MPRACE1) PLX9656 (PCI Connection) Xilinx VirtexII FPGA ControlPLD Expansion Connector ZBT SRAM 2MB SDRAM Socket Local Bus 32bit/66MHz PCI Bus 64bit/66MHz Parts common to the RobIn Prototype: PLX Pci Bridge, Local Bus, FPGA Firmware implements RobIn Prototype Message Passing protocol On-board “local” bus limited to 266MB/s (half of max. PCI throughput)

LECC2003 AmsterdamMatthias Müller 17 Measurements - Influence of DC I/O - 4 ROLs per RobIn (MPRACE) emulated Network I/O to LVL2 and EF reduce performance to 1/3 Large EB fractions: performance limited by GE line speed Small EB fractions: performance limited by PC’s computing power 100 kHz Gigabit Ethernet Line Speed 3 kHz Atlas Baseline

LECC2003 AmsterdamMatthias Müller 18 Measurements - Multiple PCI Buses - Request rate decreases, even though PCI – Bus is not saturated. Low parallelism in software?