1. TCP performance Sven Ubik ubik@cesnet.cz

2. FTP throughput capacity load ftp.uninett.no 12.3 Mb/s 1.2 Gb/s 80 Mb/s (6.6%) ftp.stanford.edu 1.3 Mb/s 600 Mb/s 180 Mb/s (30%) Protocols: TCP 95%, UDP 3%, other 2% International traffic 30% (april 2002) Géant, Internet2 vs. 10 Mb/s Ethernet ?

3. TCP flow control & congestion control

4. BW * delay product From CESNET: ping max.throughput for 64kB owin ftp.uninett.no 38 ms 13.8 Mb/s ftp.cs.columbia.edu 90 ms 5.8 Mb/s ftp.tamu.edu 133 ms 3.9 Mb/s ftp.stanford.edu 166 ms 2.6 Mb/s

5. Window Scale TCP Option (RFC 1323) Advertised rwnd shifted internally 1-14 bits 1. OS must support, for example, Linux 2.4: sysctl -w net/ipv4/tcp_adv_win_scale=1 2. Application must use a) default for all TCP connections sysctl -w net/ipv4/tcp_rmem=„4096 1048576 1048576“ sysctl -w net/ipv4/tcp_wmem=„4096 1048576 1048576“ b) application sets its own setsockopt(sockfd, SOL_SOCKET, SO_RCVBUF, (char *)&size, sizeof(int)); setsockopt(sockfd, SOL_SOCKET, SO_SNDBUF, (char *)&size, sizeof(int)); before connect() or listen() (e.g., netperf, modified ncftp+wuftpd) c) OS tunes automatically (dynamic right-sizing) 3. Timestamps + PAWS (Protect Against Wrapped Sequence Numbers)

6. Initialization: cwnd<=2*MSS, ssthresh high (rwnd) Slow start (cwnd<ssthresh): received new ack => cwnd=cwnd+max.segment Congestion avoidance (cwnd>ssthresh): RTT => cwnd=cwnd+max.segment usually approximated Timeout: ssthresh=max(owin/2, 2*max.segment) cwnd=max.segment (implies slow start) TCP congestion control

7. TCP congestion control throughput limitation Cesnet  Uninett: MSS=1460 bytes, RTT=44ms, packet loss rate~5*10 -6, Timeout=250 ms Padhye [1] equation: BW ~ 102.8 Mb/s Mathis [2] equation (BW ~ MSS/RTT * C/sqrt(p)): 110.4 Mb/s Higher MSS to speed-up congestion avoidance? [1] J. Padhye, V. Firoiu, D. Towsley, J. Kurose. „Modeling TCP Throughput: A Simple Model and its Emprirical Validation“ [2] M. Mathis, J. Semke, J. Mahdavi. „The Macroscopic Behaviour of the TCP Congestion Avoidance Algorithm“.

8. CESNET  UNINETT UDP throughput  Géant  Teleglobe created by qosplot http://www.cesnet.cz/english/project/qosip

9. Path capacity estimation tools pathchar: pathchar to tcp4-ge.uninett.no (158.38.0.194) 0 localhost | 61 Mb/s, 48 us (293 us) 1 195.113.147.1 (195.113.147.1) | 348 Mb/s, 13 us (354 us) 2 r21-pos0-0-stm16.cesnet.cz (195.113.156.114) | 108 Mb/s, 64 us (594 us) 3 cesnet.cz1.cz.geant.net (62.40.103.29) | 614 Mb/s, 4.15 ms (8.91 ms) 4 cz.de1.de.geant.net (62.40.96.38) | 557 Mb/s, 7 us (8.95 ms) 5 de1-1.de2.de.geant.net (62.40.96.130) | 1229 Mb/s, 10.9 ms (30.7 ms) 6 de.se1.se.geant.net (62.40.96.66) | ?? b/s, -13 us (30.6 ms) 7 nordunet-gw.se1.se.geant.net (62.40.103.118) | 805 Mb/s, 3.67 ms (38.0 ms) 8 no-gw.nordu.net (193.10.68.30) | 884 Mb/s, 13 us (38.0 ms) 9 oslo-gw1.uninett.no (193.10.68.50) | 656 Mb/s, 3.39 ms (44.8 ms) 10 trd-gw.uninett.no (128.39.0.250) | 99 Mb/s, 29 us (45.0 ms), 13% dropped 11 tcp4-ge.uninett.no (158.38.0.194) pathrate: „Phase I was aborted“ Final capacity estimate: 757 Mbps to 792 Mbps

10. CESNET  UNINETT TCP throughput max. rwnd CESNET  UNINETT  CESNET  [bytes] UNINETT [Mb/s] CESNET [Mb/s]UNINETT [Mb/s] (Teleglobe) 655357.5 7.58.7 13107215.2 14.417.0 26214428.7 27.633.4 52428858.1 54.462.9 1048576105.1 100.4103.4 2097152205.7 177.4103.3 4194304451.6 65.998.9 8388608394.9 67.098.9 FTP 150 MB: standard FTP: 130 s rwnd increased to 4MB: 5s

11. CESNET  UNINETT TCP throughput, cont.

12. Measurement tools Simulation (ns/2, sim.cesnet.cz) Emulation (Nist Net) Path capacity estimation tools (pathchar, pathrate, …) Capture + follow-up analysis (tcpdump + tcptrace and others) On-the-fly monitoring of TCP state variables (web100)

13. Capture + follow-up analysis tcpdump -i eth1 -p -s 96 -w trace.log tcp and host tcp4-ge.uninett.no tcptrace -l -f ’s_port!=12865’ -T -A300 -G trace.log xplot  tcpplot 

14. On-the-fly monitoring of TCP state variables www.web100.org - “Tools for end hosts to automatically achieve high bandwidth” - kernel data structures (approx. 120 variables), library and userland tools

15. On-the-fly monitoring of TCP state variables readvars 0.01 CurrentCwnd CurrentRwndRcvd

16. Parallel TCP GridFTP, LFTP ~100 Mb Uninett -> Cesnet: pget -n 5 14.7s 6.5 Mb/s pget -n 10 7.8s 12.1 Mb/s pget -n 15 5.5s 17.4 Mb/s to 10s 9.5 Mb/s pget -n 20 9.9s 9.6 Mb/s to 12s 7.9 Mb/s

17. E2E performance: No changes to network. Is it best-effort? Is it fair? Ultimate goal of E2E performance: Fully automatic adjustment to network and receiver conditions inside the operating system to maximize utilization of available resources. Use SACKs to avoid slow start after timeout Are you far enough from us? You are welcome! Sven Ubik, ubik@cesnet.cz Further research