Sven Ubik ubik@cesnet.cz TCP performance Sven Ubik ubik@cesnet.cz

Géant, Internet2 vs. 10 Mb/s Ethernet ? FTP throughput capacity load ftp.uninett.no 12.3 Mb/s 1.2 Gb/s 80 Mb/s (6.6%) ftp.stanford.edu 1.3 Mb/s 600 Mb/s 180 Mb/s (30%) Protocols: TCP 95%, UDP 3%, other 2% International traffic 30% (april 2002)

TCP flow control & congestion control

BW * delay product From CESNET: ping max.throughput for 64kB owin ftp.uninett.no 38 ms 13.8 Mb/s ftp.cs.columbia.edu 90 ms 5.8 Mb/s ftp.tamu.edu 133 ms 3.9 Mb/s ftp.stanford.edu 166 ms 2.6 Mb/s

Window Scale TCP Option (RFC 1323) Advertised rwnd shifted internally 1-14 bits 1. OS must support, for example, Linux 2.4: sysctl -w net/ipv4/tcp_adv_win_scale=1 2. Application must use a) default for all TCP connections sysctl -w net/ipv4/tcp_rmem=„4096 1048576 1048576“ sysctl -w net/ipv4/tcp_wmem=„4096 1048576 1048576“ b) application sets its own setsockopt(sockfd, SOL_SOCKET, SO_RCVBUF, (char *)&size, sizeof(int)); setsockopt(sockfd, SOL_SOCKET, SO_SNDBUF, (char *)&size, sizeof(int)); before connect() or listen() (e.g., netperf, modified ncftp+wuftpd) c) OS tunes automatically (dynamic right-sizing) 3. Timestamps + PAWS (Protect Against Wrapped Sequence Numbers)

TCP congestion control Initialization: cwnd<=2*MSS, ssthresh high (rwnd) Slow start (cwnd<ssthresh): received new ack => cwnd=cwnd+max.segment Congestion avoidance (cwnd>ssthresh): RTT => cwnd=cwnd+max.segment usually approximated Timeout: ssthresh=max(owin/2, 2*max.segment) cwnd=max.segment (implies slow start)

TCP congestion control throughput limitation Cesnet  Uninett: MSS=1460 bytes, RTT=44ms, packet loss rate~5*10 -6, Timeout=250 ms Padhye [1] equation: BW ~ 102.8 Mb/s Mathis [2] equation (BW ~ MSS/RTT * C/sqrt(p)): 110.4 Mb/s Higher MSS to speed-up congestion avoidance? [1] J. Padhye, V. Firoiu, D. Towsley, J. Kurose. „Modeling TCP Throughput: A Simple Model and its Emprirical Validation“ [2] M. Mathis, J. Semke, J. Mahdavi. „The Macroscopic Behaviour of the TCP Congestion Avoidance Algorithm“.

CESNET  UNINETT UDP throughput  Géant  Teleglobe created by qosplot http://www.cesnet.cz/english/project/qosip

Path capacity estimation tools pathrate: „Phase I was aborted“ Final capacity estimate: 757 Mbps to 792 Mbps pathchar: pathchar to tcp4-ge.uninett.no (158.38.0.194) 0 localhost | 61 Mb/s, 48 us (293 us) 1 195.113.147.1 (195.113.147.1) | 348 Mb/s, 13 us (354 us) 2 r21-pos0-0-stm16.cesnet.cz (195.113.156.114) | 108 Mb/s, 64 us (594 us) 3 cesnet.cz1.cz.geant.net (62.40.103.29) | 614 Mb/s, 4.15 ms (8.91 ms) 4 cz.de1.de.geant.net (62.40.96.38) | 557 Mb/s, 7 us (8.95 ms) 5 de1-1.de2.de.geant.net (62.40.96.130) | 1229 Mb/s, 10.9 ms (30.7 ms) 6 de.se1.se.geant.net (62.40.96.66) | ?? b/s, -13 us (30.6 ms) 7 nordunet-gw.se1.se.geant.net (62.40.103.118) | 805 Mb/s, 3.67 ms (38.0 ms) 8 no-gw.nordu.net (193.10.68.30) | 884 Mb/s, 13 us (38.0 ms) 9 oslo-gw1.uninett.no (193.10.68.50) | 656 Mb/s, 3.39 ms (44.8 ms) 10 trd-gw.uninett.no (128.39.0.250) | 99 Mb/s, 29 us (45.0 ms), 13% dropped 11 tcp4-ge.uninett.no (158.38.0.194)

CESNET  UNINETT TCP throughput max. rwnd CESNET  UNINETT  CESNET  [bytes] UNINETT [Mb/s] CESNET [Mb/s] UNINETT [Mb/s] (Teleglobe) 65535 7.5 7.5 8.7 131072 15.2 14.4 17.0 262144 28.7 27.6 33.4 524288 58.1 54.4 62.9 1048576 105.1 100.4 103.4 2097152 205.7 177.4 103.3 4194304 451.6 65.9 98.9 8388608 394.9 67.0 98.9 FTP 150 MB: standard FTP: 130 s rwnd increased to 4MB: 5s

CESNET  UNINETT TCP throughput, cont.

Measurement tools Simulation (ns/2, sim.cesnet.cz) Emulation (Nist Net) Path capacity estimation tools (pathchar, pathrate, …) Capture + follow-up analysis (tcpdump + tcptrace and others) On-the-fly monitoring of TCP state variables (web100)

Capture + follow-up analysis tcpdump -i eth1 -p -s 96 -w trace.log tcp and host tcp4-ge.uninett.no tcptrace -l -f ’s_port!=12865’ -T -A300 -G trace.log xplot  tcpplot 

On-the-fly monitoring of TCP state variables www.web100.org - “Tools for end hosts to automatically achieve high bandwidth” - kernel data structures (approx. 120 variables), library and userland tools

On-the-fly monitoring of TCP state variables readvars 0.01 CurrentCwnd CurrentRwndRcvd

Parallel TCP GridFTP, LFTP ~100 Mb Uninett -> Cesnet: pget -n 5 14.7s 6.5 Mb/s pget -n 10 7.8s 12.1 Mb/s pget -n 15 5.5s 17.4 Mb/s to 10s 9.5 Mb/s pget -n 20 9.9s 9.6 Mb/s to 12s 7.9 Mb/s

Sven Ubik, ubik@cesnet.cz Further research E2E performance: No changes to network. Is it best-effort? Is it fair? Ultimate goal of E2E performance: Fully automatic adjustment to network and receiver conditions inside the operating system to maximize utilization of available resources. Use SACKs to avoid slow start after timeout Are you far enough from us? You are welcome! Sven Ubik, ubik@cesnet.cz