A TCP/IP transport layer for the DAQ of the CMS Experiment Miklos Kozlovszky for the CMS TriDAS collaboration CERN European Organization for Nuclear Research ACAT03 - December 2003

CMS & Data Acquisition Collision rate 40 MHz Level-1 Maximum trigger rate 100 kHz Average event size ­ 1 Mbyte No. of In-Out units 1000 Readout network bandwidth ­ 1 Terabit/s Event filter computing power ­ MIPS Data production ­ Tbyte/day CMS Data

Event builder : Physical system interconnecting data sources with data destinations. It has to move each event data fragments into a same destination Event fragments : Event data fragments are stored in separated physical memory systems Full events : Full event data are stored into one physical memory system associated to a processing unit Data sources for 1 MByte events ~1000sHTL processing nodes NxM EVB Building the events

Distributed DAQ framework developed within CMS. Construct homogeneous applications for heterogeneous processing clusters. Multi-threaded (important to take advantage of SMP efficiently). Zero copy message passing for the event data. Peer to peer communication between the applications. I 2 O for data transport, and SOAP for configuration and control. Hardware and transport independency. OS and Device Drivers HTTP Ethernet Myrinet XDAQ Util/DDM Processing Sensor readout TCP PCI Subject of presentation XDAQ Framework

Reuse old, “cheap” Ethernet for DAQ Transport layer requirements –Reliable communication –Hide the complexity of TCP –Efficient implementation –Simplex communication via sockets –Configurable Support of blocking and non-blocking I/O TCP/IP Peer Transport Requirements

Pending Queues –Thread safe PQ management –One PQ for each destination –Independent sending through sockets Only one “Select” function call both to receive the packet and send the blocked data. Implementation of the non-blocking mode 12345n 12345n #2 Pending Queues XDAQ Application Framesen d 12345n #n Select

Receiver Object(s) OS XDAQ Executive Peer Transport Layer ptATCP Applications (XDAQ) ptATCPPort(s) XDAQ Framework Sender Object(s) Input SAP(s)Output SAP(s) Driver(s) NIC (10GE)NIC (FE)NIC (GE) = Creation of object = Sending = Receiving = other communication Communication via the transport layer

Throughput optimisation Single railMulti-rail App 1 App 2 App 1 Operating System tuning (kernel options+buffers) Jumbo Frames Transport protocol options Communication techniques –Blocking vs. Non-Blocking I/O –Single/Multi-rail –Single/Multi-thread –TCP options (e.g.:Nagle algorithm) –….

Test network Cluster size:8x8 CPU: 2x Intel Xeon (2.4 GHz), 512KB Cache I/O system:PCI-X: 4 buses (max 6). Memory: Two-way interleaved DDR: 3.2 GB/s (512 MB) NICs: 1 Intel 82540EM GE 1 Broadcom NeXtreme BCM 5703x GE 1 Intel Pro 2546EB GE (2port) OS: Linux RedHat (SMP) Switches: 1 BATM- T6 Multi Layer Gigabit Switch (medium range) 2 Dell Power Connect 5224 (medium range)

Conditions: XDAQ+Event Builder –No Readout Unit inputs –No Builder Unit outputs –No Event Manager PC: dual P4 Xeon Linux NIC: e-1000 Switch: Powerconnect 5224 Standard MTU (1500 Bytes) Each BU builds 128 events Fixed fragment sizes Result: For fragment size > 4 kB: Thru /node ~100 MB/s i.e. 80% utilisation Working point Event Building on the cluster

Two Rail Event Builder measurements Test case: Bare Event Builder (2x2) No RU inputs No BU outputs No Event Manager Options: Non blocking TCP Jumbo frames (mtu 8000) Two rail One thread RU working point (16 kB) Throughput/node = 240 MB/ s i.e. 95% bandwidth

Achieved 100 MB/s per node in 8x8 configuration (1rail). Improvements seen with the use of two rail, non-blocking I/O, with Jumbo frames. In 2x2 configuration over 230 MB/s obtained. High CPU load. We are also studying other networking and traffic shaping options. Conclusions