1. ESLEA PMB, Manchester, 23 Nov 2006, R. Hughes-Jones Manchester 1 Protocols Progress with Current Work. Richard Hughes-Jones The University of Manchester www.hep.man.ac.uk/~rich/ then “Talks” www.hep.man.ac.uk/~rich/

2. ESLEA PMB, Manchester, 23 Nov 2006, R. Hughes-Jones Manchester 2 uiGrid2002 monolithic code uConvert to use pthreads control Data input Data output uCode branch for Simon’s file transfer tests uWork on vlbi_recv: Output thread polled for data in the ring buffer – burned CPU Input thread signals output thread when there is work to do – else wait on semaphore – packet loss at high rate, variable thoughput Output thread uses sched_yield() when no work to do – CPU used Add code for: MarkV card and PCEVN interface Measure: throughput, packet loss, re-ordering, 1-way delay uMulti-flow Network performance – being set up Nov/Dec06 vlbi_udp: UDP on the WAN

3. ESLEA PMB, Manchester, 23 Nov 2006, R. Hughes-Jones Manchester 3 uCpu0 : 0.0% us, 0.0% sy, 0.0% ni, 99.7% id, 0.3% wa, 0.0% hi, 0.0% si uCpu1 : 11.3% us, 88.7% sy, 0.0% ni, 0.0% id, 0.0% wa, 0.0% hi, 0.0% si uCpu2 : 0.3% us, 0.0% sy, 0.0% ni, 99.3% id, 0.3% wa, 0.0% hi, 0.0% si uCpu3 : 9.3% us, 15.6% sy, 0.0% ni, 37.5% id, 0.0% wa, 1.3% hi, 36.2% si vlbi_udp: B2B UDP Tests uKernel 2.6.9 uvlbi_recv sched_yield() uWait 12 us uStable throughput 999 Mbit/s variation less than 1 Mbit/s uNo packet loss uInter-packet time uProcessing time mean 0.1005 sigma 0.1438 uCPU load:

4. ESLEA PMB, Manchester, 23 Nov 2006, R. Hughes-Jones Manchester 4 Dedicated Gbit link Onsala Sweden Gbit link Jodrell Bank UK Dwingeloo DWDM link Medicina Italy Chalmers University of Technology, Gothenburg Torun Poland Gbit link Metsähovi vlbi_udp: Multi-site Streams

5. ESLEA PMB, Manchester, 23 Nov 2006, R. Hughes-Jones Manchester 5 TCP: tcpdelay How does TCP move CBR data? u Want to examine how TCP moves Constant Bit Rate Data u VLBI Application Protocol u tcpdelay a test program: instrumented TCP program emulates sending CBR Data. Records relative 1-way delay Web100 Record TCP Stack activity n bytes Number of packets Wait time time  Data1 ●● ● Timestamp1 Time TCP & Network Receiver Timestamp2 Sender Data2 Timestamp4 Timestamp5 Data4 Timestamp3 Data3 Packet loss And now with Packet Loss

6. ESLEA PMB, Manchester, 23 Nov 2006, R. Hughes-Jones Manchester 6 TCP: tcpdelay Visualising the Results uIf Throughput NOT limited by TCP buffer size / Cwnd maybe we can re-sync with CBR arrival times. uNeed to store CBR messages during the Cwind drop in the TCP buffer uThen transmit Faster than the CBR rate to catch up Message number / Time Packet loss Delay in stream Expected arrival time at CBR Arrival time Stephen Kershaw

7. ESLEA PMB, Manchester, 23 Nov 2006, R. Hughes-Jones Manchester 7 TCP: tcpdelay JB-Manc uMessage size: 1448 Bytes uWait time: 22 us uData Rate: 525 Mbit/s uRoute: JB-Man uRTT ~1 ms uTCP buffer 2MB uDrop 1 in 10,000 packets u~2.5-3 ms increase in time for about 2000 messages ie ~ 44 ms uClassic Cwnd behaviour uCwnd dip corresponds to ~ 1.2M bytes data “Delayed” (~810 packets) uPeak Throughput ~ 620 Mbit/s

8. ESLEA PMB, Manchester, 23 Nov 2006, R. Hughes-Jones Manchester 8 Arrival Times: UKLight JB-JIVE-Manc uMessage size: 1448 Bytes uWait time: 22 us uData Rate: 525 Mbit/s uRoute: JB-UKLight-JIVE- UKLight-Man uRTT ~27 ms uTCP buffer 32M bytes uBDP @512Mbit 1.8Mbyte uEstimate catchup possible if loss < 1 in 1.24M uData needed for JIVE-Manc ~27ms Chicago-Manc ~120 ms uHave ~30 GBytes!!! Stephen Kershaw Presented at the Haystack Workshop

9. ESLEA PMB, Manchester, 23 Nov 2006, R. Hughes-Jones Manchester 9 TCP: TCP Stacks, Sharing, Reverse Traffic uDelayed by Provision of UKLight link Manc - Starlight uPC installed in Starlight and Manchester Sep06 uUdpmon tests Good Plateau ~990 Mbit/s wire rate No packet Loss Same in both directions uTCP studies: Work Now in progress

10. ESLEA PMB, Manchester, 23 Nov 2006, R. Hughes-Jones Manchester 10 DCCP: The Application View uStephen & Richard with help from Andrea uHad problems with Fedora Core 6 using stable kernel 2.6.19-rc1 DCCP data packets never reached the receiving TSAP ! Verify with tcpdump Using 2.6.19-rc5-g73fd2531-dirty uPorted udpmon to dccpmon Some system calls don’t work udccpmon tests Plateau ~990 Mbit/s wire rate No packet Loss Receive system crashed! uIperf tests 940Mbps, back-to-back uNeed more instrumentation in DCCP Eg a line in /proc/sys/snmp

11. ESLEA PMB, Manchester, 23 Nov 2006, R. Hughes-Jones Manchester 11 10 Gigabit Ethernet Lab u10 Gigabit Test Lab now set up in Manchester Cisco 7600 Cross Campus λ <1ms Neterion NICs 4 Myricom 10 Gbit NICs – delivery this week Chelsio being purchased u Boston/Supermicro X7DBE PCs Two Dual Core Intel Xeon Woodcrest 5130 2 GHz PCI-e and PCI-X uB2B performance so far SuperMicro X6DHE-G2 Kernel (2.6.13) & Driver dependent! One iperf TCP data stream 4 Gbit/s Two bi-directional iperf TCP data streams 3.8 & 2.2 Gbit/s UDP Disappointing uInstalled Fedora Core5 Kernels 2.6.17 & 2.6.18 (+web100 + packet drop) & 2.6.19 on the Intel dual-core PCs

12. ESLEA PMB, Manchester, 23 Nov 2006, R. Hughes-Jones Manchester 12 ESLEA-FABRIC:4 Gbit flows over GÉANT uSet up 4 Gigabit Lightpath Between GÉANT PoPs Collaboration with Dante GÉANT Development Network London – Amsterdam and GÉANT Lightpath service CERN – Poznan PCs in their PoPs with 10 Gigabit NICs uVLBI Tests: UDP Performance Throughput, jitter, packet loss, 1-way delay, stability Continuous (days) Data Flows – VLBI_UDP and multi-Gigabit TCP performance with current kernels Experience for FPGA Ethernet packet systems uDante Interests: multi-Gigabit TCP performance The effect of (Alcatel) buffer size on bursty TCP using BW limited Lightpaths

13. ESLEA PMB, Manchester, 23 Nov 2006, R. Hughes-Jones Manchester 13 Options Using the GÉANT LightPaths uSet up 4 Gigabit Lightpath Between GÉANT PoPs Collaboration with Dante PCs in Dante PoPs u10 Gigabit SDH backbone uAlkatel 1678 MCC uNode location: Budapest Geneva Frankfurt Milan Paris Poznan Prague Vienna uCan do traffic routing so make long rtt paths uIdeal: London Copenhagen

14. ESLEA PMB, Manchester, 23 Nov 2006, R. Hughes-Jones Manchester 14 Network/PC Booking System uBased on Meeting Room Booking System uDivide into Links and End systems uHard work by Stephen Kershaw uTesing with VLBI

15. ESLEA PMB, Manchester, 23 Nov 2006, R. Hughes-Jones Manchester 15 Any Questions?

16. ESLEA PMB, Manchester, 23 Nov 2006, R. Hughes-Jones Manchester 16 Backup Slides