5 Annual e-VLBI Workshop, September 2006, Haystack Observatory R. Hughes-Jones Manchester 1 The Network Transport layer and the Application or TCP/IP and VLBI Data Richard Hughes-Jones & Stephen Kershaw The University of Manchester then “Talks” and look for Haystack

5 Annual e-VLBI Workshop, September 2006, Haystack Observatory R. Hughes-Jones Manchester 2 Outline uThroughput Tests on Mark5s TCP Memory-2-memory tests CPU Load tests uData delay on a TCP link – How suitable is TCP? 4th Year MPhys Project Stephen Kershaw & James Keenan The effect of distance uThroughput on the 630Mbit JB-JIVE UKLight Link TCP Performance

5 Annual e-VLBI Workshop, September 2006, Haystack Observatory R. Hughes-Jones Manchester 3 VLBI Application Protocol u Want to examine the data wave front uVLBI signal wave front

5 Annual e-VLBI Workshop, September 2006, Haystack Observatory R. Hughes-Jones Manchester 4 TCP: Data moving - Filling the Pipe uTCP has to hold a copy of data in flight in case of re-transmission uOptimal (TCP buffer) window size depends on: Bandwidth end to end, i.e. min(BW links ) i.e. the bottleneck bandwidth Round Trip Time (RTT) uThe number of bytes in flight to fill the entire path: Bandwidth*Delay Product BDP = RTT*BW Can increase bandwidth by orders of magnitude RTT Time Sender Receiver ACK Segment time on wire = bits in segment/BW

5 Annual e-VLBI Workshop, September 2006, Haystack Observatory R. Hughes-Jones Manchester 5 VLBI Application Protocol u Want to examine how TCP moves Constant Bit Rate Data u tcpdelay a test program: instrumented TCP program emulates sending CBR Data. Records relative 1-way delay Record TCP Stack activity with web100 Data1 ●●● Timestamp1 Time TCP & Network Receiver Timestamp2 Sender Data2 Timestamp4 Timestamp5 Data4 Timestamp3 Data3 Packet loss uVLBI data is produced at Constant Bit Rate

5 Annual e-VLBI Workshop, September 2006, Haystack Observatory R. Hughes-Jones Manchester 6 Check the Message Send Times u10,000 Messages uMessage size: 1448 Bytes uWait time: 0 uTCP buffer 64k uRoute: Man-UKL-JIVE-prod-Man uRTT ~26 ms uSlope 0.44 ms/message uFrom TCP buffer size & RTT Expect: ~42 messages/RTT ~0.6ms/message Send time sec 1 sec Message number Manchester 4th Year MPhys Project Stephen Kershaw & James Keenan

5 Annual e-VLBI Workshop, September 2006, Haystack Observatory R. Hughes-Jones Manchester 7 Send Time Detail 100 ms Message 102 Message 76 About 25 us One rtt Send time sec 26 messages Message number uTCP Send Buffer limited uAfter SlowStart the TCP Buffer is full upackets sent out in bursts each RTT uProgram blocked on sendto()

5 Annual e-VLBI Workshop, September 2006, Haystack Observatory R. Hughes-Jones Manchester 8 1-Way Delay 1 way delay 100 ms Message number 100 ms u10,000 Messages uMessage size: 1448 Bytes uWait time: 0 uTCP buffer 64k uRoute: Man-UKL-JIVE-prod-Man uRTT ~26 ms

5 Annual e-VLBI Workshop, September 2006, Haystack Observatory R. Hughes-Jones Manchester 9 = 1.5 x RTT = 1 x RTT 26 ms Message number ≠ 0.5 x RTT 1 way delay 10 ms 10 ms uWhy not just 1 RTT? uAfter SlowStart TCP Buffer Full uMessages at front of TCP Send Buffer have to wait for next ACKs burst – 1 RTT later uMessages further back in the TCP Send Buffer wait for 2 RTT 1-Way Delay Detail

5 Annual e-VLBI Workshop, September 2006, Haystack Observatory R. Hughes-Jones Manchester 10 Problem #1 Packet Loss Is it important ?

5 Annual e-VLBI Workshop, September 2006, Haystack Observatory R. Hughes-Jones Manchester 11

5 Annual e-VLBI Workshop, September 2006, Haystack Observatory R. Hughes-Jones Manchester 12

5 Annual e-VLBI Workshop, September 2006, Haystack Observatory R. Hughes-Jones Manchester 13 Packet loss Is TCP is suitable for use in transferring real- time eVLBI data, on a lossy network? t

5 Annual e-VLBI Workshop, September 2006, Haystack Observatory R. Hughes-Jones Manchester 14 Packet loss

5 Annual e-VLBI Workshop, September 2006, Haystack Observatory R. Hughes-Jones Manchester 15 Packet loss

5 Annual e-VLBI Workshop, September 2006, Haystack Observatory R. Hughes-Jones Manchester 16 Problem #1 Packet Loss Is it important ?

5 Annual e-VLBI Workshop, September 2006, Haystack Observatory R. Hughes-Jones Manchester 17 5 ms Message number uRoute: B2B on LAN gig8-gig1 uPing 188 μs u10,000 Messages uMessage size: 1448 Bytes uWait times: 0 μs uDrop 1 in 1000 uManc-JIVE tests show times increasing with a “saw-tooth” around 10 s 1-Way Delay with packet drop 800 us 28 ms 1 way delay 10 ms

5 Annual e-VLBI Workshop, September 2006, Haystack Observatory R. Hughes-Jones Manchester 18 uStandard TCP not optimum for high throughput long distance links uPacket loss is a killer for TCP Check on campus links & equipment, and access links to backbones Users need to collaborate with the Campus Network Teams Dante Pert uNew stacks are stable and give better response & performance Still need to set the TCP buffer sizes ! Check other kernel settings e.g. window-scale maximum Watch for “TCP Stack implementation Enhancements” uTCP tries to be fair Large MTU has an advantage Short distances, small RTT, have an advantage uTCP does not share bandwidth well with other streams uThe End Hosts themselves Plenty of CPU power is required for the TCP/IP stack as well and the application Packets can be lost in the IP stack due to lack of processing power Interaction between HW, protocol processing, and disk sub-system complex uApplication architecture & implementation are also important The TCP protocol dynamics strongly influence the behaviour of the Application. uUsers are now able to perform sustained 1 Gbit/s transfers Summary & Conclusions

5 Annual e-VLBI Workshop, September 2006, Haystack Observatory R. Hughes-Jones Manchester 19 More Information Some URLs 1 uUKLight web site: uMB-NG project web site: uDataTAG project web site: uUDPmon / TCPmon kit + writeup: uMotherboard and NIC Tests: & “Performance of 1 and 10 Gigabit Ethernet Cards with Server Quality Motherboards” FGCS Special issue uTCP tuning information may be found at: & uTCP stack comparisons: “Evaluation of Advanced TCP Stacks on Fast Long-Distance Production Networks” Journal of Grid Computing 2004 uPFLDnet uDante PERT

5 Annual e-VLBI Workshop, September 2006, Haystack Observatory R. Hughes-Jones Manchester 20 uLectures, tutorials etc. on TCP/IP: uEncylopaedia uTCP/IP Resources uUnderstanding IP addresses uConfiguring TCP (RFC 1122) ftp://nic.merit.edu/internet/documents/rfc/rfc1122.txt uAssigned protocols, ports etc (RFC 1010) & /etc/protocols More Information Some URLs 2

5 Annual e-VLBI Workshop, September 2006, Haystack Observatory R. Hughes-Jones Manchester 21 Any Questions?

5 Annual e-VLBI Workshop, September 2006, Haystack Observatory R. Hughes-Jones Manchester 22 Backup Slides

5 Annual e-VLBI Workshop, September 2006, Haystack Observatory R. Hughes-Jones Manchester 23 TCP – providing reliability uPositive acknowledgement (ACK) of each received segment Sender keeps record of each segment sent Sender awaits an ACK – “I am ready to receive byte 2048 and beyond” Sender starts timer when it sends segment – so can re-transmit Segment n ACK of Segment n RTT Time Sender Receiver Sequence 1024 Length 1024 Ack 2048 Segment n+1 ACK of Segment n +1 RTT Sequence 2048 Length 1024 Ack 3072 uInefficient – sender has to wait

5 Annual e-VLBI Workshop, September 2006, Haystack Observatory R. Hughes-Jones Manchester 24 Message 102 Message ms Send time sec 26 messages Comparison of Send Time & 1-way delay Message number

5 Annual e-VLBI Workshop, September 2006, Haystack Observatory R. Hughes-Jones Manchester 25 1 way delay μs Packet number 1 way delay – packets Packet packets ~ 5.5 x RTT

5 Annual e-VLBI Workshop, September 2006, Haystack Observatory R. Hughes-Jones Manchester 26 u10,000 Messages uMessage size: 724 Bytes uWait times: 20, 25, 30, 35, 40, 45 μs uTCP buffer 64k 1 way delay 100 ms Message number 1-Way Delay 724 byte msg

5 Annual e-VLBI Workshop, September 2006, Haystack Observatory R. Hughes-Jones Manchester 27 Packet number 1-Way Delay 724 bytes Detail u10,000 Messages uMessage size: 724 Bytes uWait times: 20, 25, 30, 35, 40, 45 μs uTCP buffer 64k uRegular cycle of ~125 packets 1 way delay 100 ms

5 Annual e-VLBI Workshop, September 2006, Haystack Observatory R. Hughes-Jones Manchester 28 Throughput Measurements uUDP Throughput uSend a controlled stream of UDP frames spaced at regular intervals n bytes Number of packets Wait time time  Zero stats OK done ●●● Get remote statistics Send statistics: No. received No. lost + loss pattern No. out-of-order CPU load & no. int 1-way delay Send data frames at regular intervals ●●● Time to send Time to receive Inter-packet time (Histogram) Signal end of test OK done Time Sender Receiver

5 Annual e-VLBI Workshop, September 2006, Haystack Observatory R. Hughes-Jones Manchester 29 tcpdump / tcptrace utcpdump: dump all TCP header information for a specified source/destination ftp://ftp.ee.lbl.gov/ utcptrace: format tcpdump output for analysis using xplot NLANR TCP Testrig : Nice wrapper for tcpdump and tcptrace tools uSample use: tcpdump -s 100 -w /tmp/tcpdump.out host hostname tcptrace -Sl /tmp/tcpdump.out xplot /tmp/a2b_tsg.xpl

5 Annual e-VLBI Workshop, September 2006, Haystack Observatory R. Hughes-Jones Manchester 30 tcptrace and xplot uX axis is time uY axis is sequence number uthe slope of this curve gives the throughput over time. uxplot tool make it easy to zoom in

5 Annual e-VLBI Workshop, September 2006, Haystack Observatory R. Hughes-Jones Manchester 31 Zoomed In View uGreen Line: ACK values received from the receiver uYellow Line tracks the receive window advertised from the receiver uGreen Ticks track the duplicate ACKs received. uYellow Ticks track the window advertisements that were the same as the last advertisement. uWhite Arrows represent segments sent. uRed Arrows (R) represent retransmitted segments

5 Annual e-VLBI Workshop, September 2006, Haystack Observatory R. Hughes-Jones Manchester 32 TCP Slow Start