“ PC  PC Latency measurements” G.Lamanna, R.Fantechi & J.Kroon (CERN) TDAQ WG –

Introduction latency measurement Why the latency measurement is important for us?: PrimitivesL0TP Primitives transmission to the L0TP through ethernet (Low level protocol, possibility to have switches, small spread in latency…) Data collection L1 PCs Data collection in L1 PCs. (which protocol? Routers? Network congestions? Reliability? Fast check at application levels?...) L1 trigger distribution L1 trigger distribution. (Small latency? Latency stability? Broadcasting? …) computer and network HW Feedback to select computer and network HW transmission protocol Select the transmission protocol

Transmission protocol: UDP vs TCP? TCPGOOD: reliableordered provides reliable, ordered delivery of a stream of bytes. Flow Flow control.BAD: heavy heavy protocol to be implemented in hardware, problem each packet needs to be acknowledged and at high rate can become a problem. not stable The latency is intrinsically not stable. fractioned The packets can be fractioned for transmission optimization reasonsUDPGOOD: simple fast simple and fast protocol, easy easy to be implemented in hardware, does not require big resources.BAD: reliabilityorderingintegrity no reliability, ordering, or data integrity unreliable provides an unreliable service and datagrams may arrive out of order.

Test setup latency senderreceiver The software tools for measure the latency provide averages and isn’t clear the contribution to the latency due to the sender and the receiver HW approach packet transmission HW approach: use the oscilloscope to measure the time difference in packet transmission PC #1 PC #2 scope LPT ETH [see also R.Fantechi TDAQ ]

Hardware used PCATELKRPN0GPU1 ProcessorP4 2.4 GHzXeon GHz (2 processors) Xeon GHz (2 processors, 16 cores) Cache512k4096k12 MB NICIntel 82541PI (bus PCI) Intel 80003E52LAN (bus PCI) Intel 82574L (PCI express x4) Drivere1000 e1000e kernel generic (ubuntu) generic (kubuntu) kernel el5 (SL5) PCATE LKRPN0 GPU1

Results: PCATE & GPU1 GPU1 PCATE

Results: GPU1 & LKRPN0 GPU1 LKRPN0

Results: LKRPN0 & PCATE PCATE LKRPN0 PCATE

Cross check with sockperf GPU1->PCATE compatible with the HW measurements In GPU1->PCATE the result is compatible with the HW measurements PCATE->GPU1 big latency In the PCATE->GPU1 we have a very big latency GPU1 PCATE GPU1 PCATE

GPU1 mother board architecture Process or 1 Process or 2 QUAD PORT GBE DUAL PORT GBE IOH ICH10R PCI- E x4 QPI ESIx 4 PCI- E x1 Test 1 Test 2

“Schizophrenic” test Same PC, different NIC GPU1 Same PC, Same NIC, different port

What we learned from this first studies? senderreceiver latency Both the characteristics of the sender and the receiver are important for the latency sockperf In particular the sender seems to play an important role using sockperf latencypacket dimension The latency increases with the packet dimension frequencyisn’t relevant jump The frequency of the packet isn’t relevant until the “jump” jumphigh frequency The “jump” happens at relatively high frequency rate (> 20 kHz) jumpNIC/BUS/Chipset CPU/Memory The “jump” probably depends on the NIC/BUS/Chipset more than on CPU/Memory latencyPCs packet dimension The absolute value of the latency depends on the PCs setup and the packet dimension: ~50 usGPU1LKRPN0~60 us PCATE Minimum (~300 B packets): ~50 us (recv GPU1 or LKRPN0), ~60 us (recv PCATE) >50us~70 us GPU1 LKRPN0~90 us PCATE Maximum (~1500B, no jumps if delay is >50us): ~70 us (recv GPU1 or LKRPN0), ~90 us (recv PCATE)

To Do TALK board 10 MHz Use the TALK board to go “faster” (the 10 MHz can’t be reached using PCs) jumps Identify the responsible of the “jumps”: NICPCs Same NIC on different PCs NICsPC Different NICs on same PC latencyTELL1 Measure the latency with the TELL1 suggestionswelcome Any suggestions is more than welcome!!! GPU1 PCATE LKRPN0