ESLEA PMB, Manchester, 23 Nov 2006, R. Hughes-Jones Manchester 1 Protocols Progress with Current Work. Richard Hughes-Jones The University of Manchester then “Talks”

ESLEA PMB, Manchester, 23 Nov 2006, R. Hughes-Jones Manchester 2 uiGrid2002 monolithic code uConvert to use pthreads control Data input Data output uCode branch for Simon’s file transfer tests uWork on vlbi_recv: Output thread polled for data in the ring buffer – burned CPU Input thread signals output thread when there is work to do – else wait on semaphore – packet loss at high rate, variable thoughput Output thread uses sched_yield() when no work to do – CPU used Add code for: MarkV card and PCEVN interface Measure: throughput, packet loss, re-ordering, 1-way delay uMulti-flow Network performance – being set up Nov/Dec06 vlbi_udp: UDP on the WAN

ESLEA PMB, Manchester, 23 Nov 2006, R. Hughes-Jones Manchester 3 uCpu0 : 0.0% us, 0.0% sy, 0.0% ni, 99.7% id, 0.3% wa, 0.0% hi, 0.0% si uCpu1 : 11.3% us, 88.7% sy, 0.0% ni, 0.0% id, 0.0% wa, 0.0% hi, 0.0% si uCpu2 : 0.3% us, 0.0% sy, 0.0% ni, 99.3% id, 0.3% wa, 0.0% hi, 0.0% si uCpu3 : 9.3% us, 15.6% sy, 0.0% ni, 37.5% id, 0.0% wa, 1.3% hi, 36.2% si vlbi_udp: B2B UDP Tests uKernel uvlbi_recv sched_yield() uWait 12 us uStable throughput 999 Mbit/s variation less than 1 Mbit/s uNo packet loss uInter-packet time uProcessing time mean sigma uCPU load:

ESLEA PMB, Manchester, 23 Nov 2006, R. Hughes-Jones Manchester 4 Dedicated Gbit link Onsala Sweden Gbit link Jodrell Bank UK Dwingeloo DWDM link Medicina Italy Chalmers University of Technology, Gothenburg Torun Poland Gbit link Metsähovi vlbi_udp: Multi-site Streams

ESLEA PMB, Manchester, 23 Nov 2006, R. Hughes-Jones Manchester 5 TCP: tcpdelay How does TCP move CBR data? u Want to examine how TCP moves Constant Bit Rate Data u VLBI Application Protocol u tcpdelay a test program: instrumented TCP program emulates sending CBR Data. Records relative 1-way delay Web100 Record TCP Stack activity n bytes Number of packets Wait time time  Data1 ●● ● Timestamp1 Time TCP & Network Receiver Timestamp2 Sender Data2 Timestamp4 Timestamp5 Data4 Timestamp3 Data3 Packet loss And now with Packet Loss

ESLEA PMB, Manchester, 23 Nov 2006, R. Hughes-Jones Manchester 6 TCP: tcpdelay Visualising the Results uIf Throughput NOT limited by TCP buffer size / Cwnd maybe we can re-sync with CBR arrival times. uNeed to store CBR messages during the Cwind drop in the TCP buffer uThen transmit Faster than the CBR rate to catch up Message number / Time Packet loss Delay in stream Expected arrival time at CBR Arrival time Stephen Kershaw

ESLEA PMB, Manchester, 23 Nov 2006, R. Hughes-Jones Manchester 7 TCP: tcpdelay JB-Manc uMessage size: 1448 Bytes uWait time: 22 us uData Rate: 525 Mbit/s uRoute: JB-Man uRTT ~1 ms uTCP buffer 2MB uDrop 1 in 10,000 packets u~2.5-3 ms increase in time for about 2000 messages ie ~ 44 ms uClassic Cwnd behaviour uCwnd dip corresponds to ~ 1.2M bytes data “Delayed” (~810 packets) uPeak Throughput ~ 620 Mbit/s

ESLEA PMB, Manchester, 23 Nov 2006, R. Hughes-Jones Manchester 8 Arrival Times: UKLight JB-JIVE-Manc uMessage size: 1448 Bytes uWait time: 22 us uData Rate: 525 Mbit/s uRoute: JB-UKLight-JIVE- UKLight-Man uRTT ~27 ms uTCP buffer 32M bytes 1.8Mbyte uEstimate catchup possible if loss < 1 in 1.24M uData needed for JIVE-Manc ~27ms Chicago-Manc ~120 ms uHave ~30 GBytes!!! Stephen Kershaw Presented at the Haystack Workshop

ESLEA PMB, Manchester, 23 Nov 2006, R. Hughes-Jones Manchester 9 TCP: TCP Stacks, Sharing, Reverse Traffic uDelayed by Provision of UKLight link Manc - Starlight uPC installed in Starlight and Manchester Sep06 uUdpmon tests Good Plateau ~990 Mbit/s wire rate No packet Loss Same in both directions uTCP studies: Work Now in progress

ESLEA PMB, Manchester, 23 Nov 2006, R. Hughes-Jones Manchester 10 DCCP: The Application View uStephen & Richard with help from Andrea uHad problems with Fedora Core 6 using stable kernel rc1 DCCP data packets never reached the receiving TSAP ! Verify with tcpdump Using rc5-g73fd2531-dirty uPorted udpmon to dccpmon Some system calls don’t work udccpmon tests Plateau ~990 Mbit/s wire rate No packet Loss Receive system crashed! uIperf tests 940Mbps, back-to-back uNeed more instrumentation in DCCP Eg a line in /proc/sys/snmp

ESLEA PMB, Manchester, 23 Nov 2006, R. Hughes-Jones Manchester Gigabit Ethernet Lab u10 Gigabit Test Lab now set up in Manchester Cisco 7600 Cross Campus λ <1ms Neterion NICs 4 Myricom 10 Gbit NICs – delivery this week Chelsio being purchased u Boston/Supermicro X7DBE PCs Two Dual Core Intel Xeon Woodcrest GHz PCI-e and PCI-X uB2B performance so far SuperMicro X6DHE-G2 Kernel (2.6.13) & Driver dependent! One iperf TCP data stream 4 Gbit/s Two bi-directional iperf TCP data streams 3.8 & 2.2 Gbit/s UDP Disappointing uInstalled Fedora Core5 Kernels & (+web100 + packet drop) & on the Intel dual-core PCs

ESLEA PMB, Manchester, 23 Nov 2006, R. Hughes-Jones Manchester 12 ESLEA-FABRIC:4 Gbit flows over GÉANT uSet up 4 Gigabit Lightpath Between GÉANT PoPs Collaboration with Dante GÉANT Development Network London – Amsterdam and GÉANT Lightpath service CERN – Poznan PCs in their PoPs with 10 Gigabit NICs uVLBI Tests: UDP Performance Throughput, jitter, packet loss, 1-way delay, stability Continuous (days) Data Flows – VLBI_UDP and multi-Gigabit TCP performance with current kernels Experience for FPGA Ethernet packet systems uDante Interests: multi-Gigabit TCP performance The effect of (Alcatel) buffer size on bursty TCP using BW limited Lightpaths

ESLEA PMB, Manchester, 23 Nov 2006, R. Hughes-Jones Manchester 13 Options Using the GÉANT LightPaths uSet up 4 Gigabit Lightpath Between GÉANT PoPs Collaboration with Dante PCs in Dante PoPs u10 Gigabit SDH backbone uAlkatel 1678 MCC uNode location: Budapest Geneva Frankfurt Milan Paris Poznan Prague Vienna uCan do traffic routing so make long rtt paths uIdeal: London Copenhagen

ESLEA PMB, Manchester, 23 Nov 2006, R. Hughes-Jones Manchester 14 Network/PC Booking System uBased on Meeting Room Booking System uDivide into Links and End systems uHard work by Stephen Kershaw uTesing with VLBI

ESLEA PMB, Manchester, 23 Nov 2006, R. Hughes-Jones Manchester 15 Any Questions?

ESLEA PMB, Manchester, 23 Nov 2006, R. Hughes-Jones Manchester 16 Backup Slides