The ALICE Data-Acquisition Read-out Receiver Card C. Soós et al. (for the ALICE collaboration) LECC September 2004, Boston

13-17 September 2004LECC 2004, Boston2 Outline Introduction Introduction Hardware Hardware Firmware Firmware Software Software Performance Performance Applications Applications Summary Summary

13-17 September 2004LECC 2004, Boston3 Introduction: ALICE DAQ ALICE requirements with Pb-Pb beams ALICE requirements with Pb-Pb beams – 25 GB/s aggregate BW from detectors – 2.5 GB/s aggregate event building BW – 1.25 GB/s aggregate BW to tape About 400 Detector Data Link (DDL) About 400 Detector Data Link (DDL) – Full-duplex optical link – Data rate up to 200 MB/s The Readout Receiver Card (D-RORC) The Readout Receiver Card (D-RORC) – PCI compatible adapter – Integrates two DDL interfaces ALICE data-acquisition software: DATE ALICE data-acquisition software: DATE Local Data Concentrator DDL SIU Detector Readout Electronics DDL DIU D-RORC Event Building Network Global Data Collector Storage Network Transient Data Storage DDL ports 50 25

13-17 September 2004LECC 2004, Boston4 Introduction: DAQ and HLT Some of the detectors require filtering or data reduction Some of the detectors require filtering or data reduction DAQ and HLT interface DAQ and HLT interface – Standard DDL links – Data splitter in the D-RORC Each D-RORC hosts two DIUs Each D-RORC hosts two DIUs – First DIU receives detector data – Second DIU transfers the copy of the raw data HLT data is transferred back using DDL and D-RORC HLT data is transferred back using DDL and D-RORC HLT Farm H-RORC FEP D-RORC LDC D-RORC LDC DIU SIU Event Building Network Readout Electronics Detector

13-17 September 2004LECC 2004, Boston5 Hardware Altera FPGA PCI 64-bit/66 MHz CMC interface Optical transceivers JTAG interface Conf. Flash LVDS interface 850 nm VCSEL laser Gbit/s Pluggable modules High-speed serial I/F 2 inputs + 2 outputs Purpose: detector busy +3.3V compatible signals Bus master enabled Standard extension I/F About 180 user I/O APEX-E device family EP20K400E EPC4 Programmable via internal or external JTAG chain FPGA programming and debugging Flash memory programming Electrical transceivers Multi-rate transceivers Serial-Parallel-Serial converters Integrated 8B/10B endec Clock recovery

13-17 September 2004LECC 2004, Boston6 Firmware PCI interface core PCI interface core – Handles PCI transactions – Performs PCI mastering Receiver and Transmitter Receiver and Transmitter – Buffer data and initiate the DMA Control registers Control registers – Mapped to PCI memory – Provide control interface – Provide status information DDL interface DDL interface – Performs DDL transactions – Provides DDL status PCI core (64-bit master, memory mapped) Transmit data path DDL interface Control registers Receive data path 64-bit PCI or PCI-X bus, 3.3V signaling (!) High-speed I/F to the transceivers

13-17 September 2004LECC 2004, Boston7 PCI bus PC memory, managed by PHYSMEM Receiver DMA controller Receive Address FIFO DMA report #1 DMA report #2 DMA report #3 DMA report #4 DMA report #5 Receiver DMA Buffer Descriptor #1 Buffer Descriptor #2 Buffer Descriptor #3 Buffer Descriptor #4 Buffer Descriptor #5 Receive Report FIFO Firmware Software D-RORC DDL Push

13-17 September 2004LECC 2004, Boston8 D-RORC driver: Linux device driver, runtime loadable module D-RORC driver: Linux device driver, runtime loadable module – Finds the D-RORC cards on the PCI buses – Maps the registers into the user memory space D-RORC API layer: collection of library routines written in C D-RORC API layer: collection of library routines written in C – Ensures exclusive access to the hardware using device locking – Provides simple programming I/F for higher level applications Command line executables Command line executables – Hardware identification – Reset components (DIU, SIU etc.) – Send commands, reads status – Send data blocks – Receives and checks data blocks Software Linux kernel D-RORC driverPhysmem D-RORC API layer Application (e.g. DATE)

13-17 September 2004LECC 2004, Boston9 Performance: Test bed Supermicro server motherboard with dual Xeon 2.4 GHz Supermicro server motherboard with dual Xeon 2.4 GHz Six PCI-X slots, 4 bus segments ( ) Six PCI-X slots, 4 bus segments ( ) Linux OS Linux OS ALICE Data-Acquisition software (DATE) ALICE Data-Acquisition software (DATE)

13-17 September 2004LECC 2004, Boston10 Performance: Single channel Bandwidth vs. block size measurements with internal and external (DDL) data source using one D-RORC channel Bandwidth vs. block size measurements with internal and external (DDL) data source using one D-RORC channel Steady increase until the maximum bandwidth is reached Steady increase until the maximum bandwidth is reached – Internal: BWmax = Fpci [MHz] x 4 [Bytes] = 264 MB/s – External: BWmax = BWddl = 206 MB/s

13-17 September 2004LECC 2004, Boston11 Performance: Dual channel Bandwidth vs. block size measurements with internal and external (DDL) data source using two D-RORC channel Bandwidth vs. block size measurements with internal and external (DDL) data source using two D-RORC channel Steady increase until the maximum bandwidth is reached Steady increase until the maximum bandwidth is reached – Internal: BWmax = Fpci [MHz] x 4 [Bytes] x 2 – Loss = 484 MB/s – External: BWmax = BWddl x 2 = 412 MB/s

13-17 September 2004LECC 2004, Boston12 Performance: Dual vs. 2 Single Bandwidth vs. block size measurement with internal data source Bandwidth vs. block size measurement with internal data source Different maximum (different arbitration) Different maximum (different arbitration) – Dual-channel D-RORC:BWmax = 484 MB/s – Two single-channel D-RORC:BWmax = 464 MB/s

13-17 September 2004LECC 2004, Boston13 Performance: Six D-RORCs Testing the fully populated PC using internal data source Testing the fully populated PC using internal data source – Interoperability test – Measure the maximal input bandwidth PCI #6 PCI #5 PCI #4 PCI #3 PCI #2 PCI #1 Segment #1 #2 #3 #4 Controller #1 #2 1 Ch 1Ch Bandwidth [MB/s] Normalized Bandwidth [MB/s/Ch]

13-17 September 2004LECC 2004, Boston14 D-RORC Detector LDC D-RORC DIU HLT LDC Fast Ethernet GDC 3x 250 GB disk Detector LDC VME processor CAEN VME boards Si Telescope TOF DDL 3DDL 4 DDL 5 HLT (3 PCs) RCU 1RCU 2 DDL 1 DDL 2 IROC CASTOR 1.5 TB 10 MB/s /castor/cern.ch/alice/testbeam2004/T10 Applications: TPC sector test Test beam of one complete Inner Read-out Chamber (IROC) of the ALICE TPC detector (May 2004) Test beam of one complete Inner Read-out Chamber (IROC) of the ALICE TPC detector (May 2004)

13-17 September 2004LECC 2004, Boston15 Summary D-RORC card has been developed as the high-speed interface between the DDL and the PCI bus D-RORC card has been developed as the high-speed interface between the DDL and the PCI bus Using two integrated DDL channels Using two integrated DDL channels – reduces the number of PCI slots – offers data paths from the detectors to the DAQ and HLT systems Linux device driver and API library based on standard C, as well as Linux executables are available Linux device driver and API library based on standard C, as well as Linux executables are available The card has been tested thoroughly in the lab The card has been tested thoroughly in the lab – 1 CH bandwidth = 264 MB/s – 2 CH bandwidth = 484 MB/s – 4 D-RORC on different PCI segments = 1045 MB/s Real applications show the stability and reliability of the card Real applications show the stability and reliability of the card

Thank you!