LNL CMS G. MaronCPT Week CERN, 23 April Legnaro Event Builder Prototypes Luciano Berti, Gaetano Maron Luciano Berti, Gaetano Maron INFN – Laboratori Nazionali di Legnaro

LNL CMS G. MaronCPT Week CERN, 23 April GE Event Builder Test conditions: No command or event aggregation (each packet transports a command or data frame relative to a single event) No command or event aggregation (each packet transports a command or data frame relative to a single event) full data transfer from/to PC memory full data transfer from/to PC memory recovery from packet loss recovery from packet loss fixed fragment sizes are varied bytes fixed fragment sizes are varied bytes Components:Hardware: switch: FoundryNet FastIron switch: FoundryNet FastIron NIC: SysKonnect SK9821 NIC: SysKonnect SK9821 PC: Supermicro PIII (i840) PC: Supermicro PIII (i840) Software: vxWorks based 15 x 15

LNL CMS G. MaronCPT Week CERN, 23 April Event builder layout EVM RUs BUs Slot 1 Slot 2 Slot 3 Slot 4 RUs and BUs distributed in all switch slots: – Part of the traffic localized within the slot – Reduces switch backplane utilization RU performance problem found with problem found with this configuration this configuration

LNL CMS G. MaronCPT Week CERN, 23 April Modified Event Builder layout EVM RUs BUs Slot 1 Slot 2 Slot 3 Slot 4 Request data commands Fast Ethernet Slot Request data commands - RU fast control message over FE (PCI 32/33) - RU data transfer on GE (PCI 64/66)

LNL CMS G. MaronCPT Week CERN, 23 April The GE Event Builder

LNL CMS G. MaronCPT Week CERN, 23 April EB protocol BU allocate EVM confirm 1 23n RUs send cache

LNL CMS G. MaronCPT Week CERN, 23 April Concurrent building threads in the same BU BU EVM 1 23n RUs BU thread 1 BU thread 2 BU thread 3

LNL CMS G. MaronCPT Week CERN, 23 April BU allocate EVM confirm Sequential reading 1 23n RUs BU allocate EVM confirm Random reading 1 23n RUs 45 send cache send cache Sequential vs Random reading

LNL CMS G. MaronCPT Week CERN, 23 April BU allocate EVM confirm 1 23n RUs 45 send cache “Sliding Window” multiple send to Rusmultiple send to Rus reduce the total rebuilding timereduce the total rebuilding time less events in the Busless events in the Bus not yet testednot yet tested

LNL CMS G. MaronCPT Week CERN, 23 April No difference on performanceNo difference on performance But more allocated event needed on BUs,But more allocated event needed on BUs, All the measurements with random readingAll the measurements with random reading Sequential reading Random reading Sequential - random reading comparison

LNL CMS G. MaronCPT Week CERN, 23 April Recovery from Packets loss BU Req. Data RU timer timer start EvtData Req. Data ( retry ) start cancel timeout EvtData BU Req. EvtId EVM timer timer start EvtId Req. EvtId (retry) start cancel timeout EvtId BU – EVM communication BU – RU communication Timeouts ms

LNL CMS G. MaronCPT Week CERN, 23 April EVB 15x15 performance - Throughput Throughput up to 116 MB/s, ie 93% link speed Throughput up to 116 MB/s, ie 93% link speed no packet loss observed (as expected) no packet loss observed (as expected) 15 x 15

LNL CMS G. MaronCPT Week CERN, 23 April EVB Scaling

LNL CMS G. MaronCPT Week CERN, 23 April EVB Performance – Event Rate Nominal fragment size 2kbytes: event rate = 52 kHz 15 x 15

LNL CMS G. MaronCPT Week CERN, 23 April Conic Event Builder Event Manager Manager Builder Network RU BU FU FU FU FU Event Manager Manager RU FUFUFUFUFUFUFUFU Builder Network FUFUFUFU symmetric EVB conic EVB faster ports at Rus faster ports at Rus slower ports at BUs slower ports at BUs

LNL CMS G. MaronCPT Week CERN, 23 April Conic Event Builder: Layout EVM RUs FUs GE Slot 1 Request Data Command FE Slot 1 FE Slot 2

LNL CMS G. MaronCPT Week CERN, 23 April EVB throughput – Conic vs Symmetric conic EVB: no performance degradation vs symmetric

LNL CMS G. MaronCPT Week CERN, 23 April EVB Conic – Scaling

LNL CMS G. MaronCPT Week CERN, 23 April Conic: RU/FU Throughput ratio

LNL CMS G. MaronCPT Week CERN, 23 April To be done and test variable size eventsvariable size events EB performances with the new implemented “ window” mechanismEB performances with the new implemented “ window” mechanism latency times measurementslatency times measurements Fault generation with the new implemented Random Error Generator to check the error recovery procedureFault generation with the new implemented Random Error Generator to check the error recovery procedure

LNL CMS G. MaronCPT Week CERN, 23 April Multistage Event Builder All our results have been obtained with a single switch event builder configurationAll our results have been obtained with a single switch event builder configuration We propose to extend our tests to a multistage ethernet switches topology and to study the behavior of this configuration.We propose to extend our tests to a multistage ethernet switches topology and to study the behavior of this configuration.

LNL CMS G. MaronCPT Week CERN, 23 April Plain Topology In the Event Builder application data flows in only one directionIn the Event Builder application data flows in only one direction The inter-switch Gigabit Ethernet links are full-duplexThe inter-switch Gigabit Ethernet links are full-duplex Result : half of the inter-switch bandwidth available is wastedResult : half of the inter-switch bandwidth available is wasted RUs BUs

LNL CMS G. MaronCPT Week CERN, 23 April Full Mesh Topology Full Mesh Topology RU and BU distributed in all the switchesRU and BU distributed in all the switches Inter-switch links are used in both directionInter-switch links are used in both direction Same number of ports of the plain topologySame number of ports of the plain topology Twice of the bandwidth of the plain topology in the inter-switch linksTwice of the bandwidth of the plain topology in the inter-switch links RUs BUs

LNL CMS G. MaronCPT Week CERN, 23 April Plain and mesh topology limits Plain and mesh topology limits Each couple of switches is connected by a single linkEach couple of switches is connected by a single link This is a bottleneck if the traffic is not uniformly randomThis is a bottleneck if the traffic is not uniformly random The network is blocking for certain traffic patternsThe network is blocking for certain traffic patterns

LNL CMS G. MaronCPT Week CERN, 23 April Traffic with patterns If traffic has patterns (for example this could happens in the case the event builder is performed in steps) it could make sense to introduce an artificial mechanism that randomise the traffic.If traffic has patterns (for example this could happens in the case the event builder is performed in steps) it could make sense to introduce an artificial mechanism that randomise the traffic. This mechanism exist and it is called Universal RoutingThis mechanism exist and it is called Universal Routing

LNL CMS G. MaronCPT Week CERN, 23 April Universal Routing Reference Discovered by L.G. Valiant in 1980Discovered by L.G. Valiant in 1980 See: M.D May, P.W. Thompson, P.H. Welch NETWORKS,ROUTERS & TRASPUTER available on : M.D May, P.W. Thompson, P.H. Welch NETWORKS,ROUTERS & TRASPUTER available on : Those papers describe the Universal Routing applied to Transputers Networks, a wormhole routing based networkThose papers describe the Universal Routing applied to Transputers Networks, a wormhole routing based network We adapted the same concept to a packet switched network like Gigabit EthernetWe adapted the same concept to a packet switched network like Gigabit Ethernet

LNL CMS G. MaronCPT Week CERN, 23 April Universal Routing with GigaEthernet Based on Clos topologyBased on Clos topology Multiple path available between each couple of switchesMultiple path available between each couple of switches Every packets is sent to a randomly chosen intermediate switchEvery packets is sent to a randomly chosen intermediate switch The intermediate switch send the packet to the final destinationThe intermediate switch send the packet to the final destination Full bandwidth between each couple of switches and uniform buffer utilizationFull bandwidth between each couple of switches and uniform buffer utilization

LNL CMS G. MaronCPT Week CERN, 23 April Universal Routing Transformation of the CLOS topology to a folded CLOSTransformation of the CLOS topology to a folded CLOS The resulting number of ports is the same of the plain topologyThe resulting number of ports is the same of the plain topology Half duplex links Full duplex links RUs BUs RUs BUs

LNL CMS G. MaronCPT Week CERN, 23 April Large (500x500) multistage GE network (1) BUs RUs 20 Ports 25 Ports 40 Ports 25 switches with 60 x 1Gb ports25 switches with 60 x 1Gb ports 20 switches with 25 x 1 Gb ports20 switches with 25 x 1 Gb ports

LNL CMS G. MaronCPT Week CERN, 23 April Large (500x500) multistage GE network (2) 25 switches with 40 x 1Gb ports + 2 x 10 Gb uplinks25 switches with 40 x 1Gb ports + 2 x 10 Gb uplinks 2 switches with 25 x 10 Gb ports2 switches with 25 x 10 Gb ports BUs 20 RUs 2 Ports 10G 25 Ports 10G 40 Ports

LNL CMS G. MaronCPT Week CERN, 23 April Proposal for a multistage event builder demonstrator Multistage event builders can be emulated using the much cheaper fast ethernet connections and switches. The GE speed is not needed in these topological investigationsMultistage event builders can be emulated using the much cheaper fast ethernet connections and switches. The GE speed is not needed in these topological investigations The proposal is to have prototypes for:The proposal is to have prototypes for: –Full Mesh Topology –Folded CLOS topology with (and without) Universal Routing mechanism

LNL CMS G. MaronCPT Week CERN, 23 April Full Mesh 64x64 Event Builder Prototype RUs 8 BUs - 1 host node 4 Rus / 4 BUs or a mix of them a mix of them - 32 hosts FE NICS ( ) FE ports switch Missing components

LNL CMS G. MaronCPT Week CERN, 23 April Folded CLOS 64x64 Event Builder Prototype BUs RUs 16 Ports 4 Ports 32 Ports 3 FastIron with 3 24 FE ports mods - 1 host node 4 Rus / 4 BUs or a mix of them a mix of them - 32 hosts FE NICS ( ) FE ports switch FE ports FastIron module FE ports FastIron module Missing components - 1 host node 3 Rus / 3 BUs or a mix of them a mix of them - 32 hosts - 96 FE NICS ( ) FE ports switch FE ports FastIron module 64x64 48x48

LNL CMS G. MaronCPT Week CERN, 23 April Folded CLOS 80x80 Event Builder Prototype BUs 20 RUs 2 GE Ports 4 Ports 40 FE Ports 3 FastIron with 8 GE ports module (1000 BaseT or 1000 BaseSX) - 1 host node 4 Rus / 4 BUs or a mix of them a mix of them - 40 hosts (32+8) FE NICS ( ) FE ports + 2 GE links switch GE (Base SX) ports FastIron module Missing components 80x80 2

LNL CMS G. MaronCPT Week CERN, 23 April x80 Conic Event Builder Prototype FUs 2 GE Ports 20 FE Ports 3 RU1 RU2 RU3 RU4 RU5 RU6 RU7 RU8 FastIron with 2 8 GE ports modules (1000 BaseT or 1000 BaseSX) - 1 host node 4 FUs - 20 hosts - 80 FE NICS ( ) (48) FE ports + 2 GE up links switch GE (Base SX) ports FastIron module Missing components 8x80

LNL CMS G. MaronCPT Week CERN, 23 April Material for the event builder multistage prototypes Mesh 64x FE NICs FE ports Folded CLOS 80x PCs FE NICS FE ports with 2 GE uplinks 1000 baseT if the 1000 baseT uplinks are not available: 1) Folded CLOS 64x64: 72 FE NICs 4 48 FE ports switch 1 24 FE ports FastIron module 2) Folded CLOS 48x48: 40 FE NICs 4 >36 FE ports switch