1. Sven Ubik ubik@cesnet.cz
TCP performance
Sven Ubik

2. Géant, Internet2 vs. 10 Mb/s Ethernet ?
FTP throughput capacity load ftp.uninett.no Mb/s Gb/s Mb/s (6.6%) ftp.stanford.edu Mb/s Mb/s Mb/s (30%) Protocols: TCP 95%, UDP 3%, other 2% International traffic 30% (april 2002)

3. TCP flow control & congestion control

4. BW * delay product From CESNET: ping max.throughput for 64kB owin
ftp.uninett.no ms Mb/s
ftp.cs.columbia.edu ms Mb/s
ftp.tamu.edu ms Mb/s
ftp.stanford.edu ms Mb/s

5. Window Scale TCP Option (RFC 1323)
Advertised rwnd shifted internally 1-14 bits
1. OS must support, for example, Linux 2.4:
sysctl -w net/ipv4/tcp_adv_win_scale=1
2. Application must use
a) default for all TCP connections
sysctl -w net/ipv4/tcp_rmem=„ “
sysctl -w net/ipv4/tcp_wmem=„ “
b) application sets its own
setsockopt(sockfd, SOL_SOCKET, SO_RCVBUF, (char *)&size, sizeof(int));
setsockopt(sockfd, SOL_SOCKET, SO_SNDBUF, (char *)&size, sizeof(int));
before connect() or listen() (e.g., netperf, modified ncftp+wuftpd)
c) OS tunes automatically (dynamic right-sizing)
3. Timestamps + PAWS (Protect Against Wrapped Sequence Numbers)

6. TCP congestion control
Initialization:
cwnd<=2*MSS, ssthresh high (rwnd)
Slow start (cwnd<ssthresh):
received new ack => cwnd=cwnd+max.segment
Congestion avoidance (cwnd>ssthresh):
RTT => cwnd=cwnd+max.segment
usually approximated
Timeout:
ssthresh=max(owin/2, 2*max.segment)
cwnd=max.segment (implies slow start)

7. TCP congestion control throughput limitation
Cesnet  Uninett:
MSS=1460 bytes, RTT=44ms, packet loss rate~5*10 -6, Timeout=250 ms
Padhye [1] equation: BW ~ Mb/s
Mathis [2] equation (BW ~ MSS/RTT * C/sqrt(p)): Mb/s
Higher MSS to speed-up congestion avoidance?
[1] J. Padhye, V. Firoiu, D. Towsley, J. Kurose. „Modeling TCP Throughput: A Simple Model and its Emprirical Validation“
[2] M. Mathis, J. Semke, J. Mahdavi. „The Macroscopic Behaviour of the TCP Congestion Avoidance Algorithm“.

8. CESNET  UNINETT UDP throughput
 Géant
 Teleglobe
created by qosplot

9. Path capacity estimation tools
pathrate:
„Phase I was aborted“
Final capacity estimate: 757 Mbps to 792 Mbps
pathchar:
pathchar to tcp4-ge.uninett.no ( )
0 localhost
| Mb/s, 48 us (293 us)
( )
| Mb/s, 13 us (354 us)
2 r21-pos0-0-stm16.cesnet.cz ( )
| Mb/s, 64 us (594 us)
3 cesnet.cz1.cz.geant.net ( )
| Mb/s, ms (8.91 ms)
4 cz.de1.de.geant.net ( )
| Mb/s, 7 us (8.95 ms)
5 de1-1.de2.de.geant.net ( )
| Mb/s, ms (30.7 ms)
6 de.se1.se.geant.net ( )
| ?? b/s, us (30.6 ms)
7 nordunet-gw.se1.se.geant.net ( )
| Mb/s, ms (38.0 ms)
8 no-gw.nordu.net ( )
| Mb/s, 13 us (38.0 ms)
9 oslo-gw1.uninett.no ( )
| Mb/s, ms (44.8 ms)
10 trd-gw.uninett.no ( )
| Mb/s, 29 us (45.0 ms), 13% dropped
11 tcp4-ge.uninett.no ( )

10. CESNET  UNINETT TCP throughput
max. rwnd CESNET  UNINETT  CESNET 
[bytes] UNINETT [Mb/s] CESNET [Mb/s] UNINETT [Mb/s]
(Teleglobe)
FTP 150 MB: standard FTP: s
rwnd increased to 4MB: s

11. CESNET  UNINETT TCP throughput, cont.

12. Measurement tools Simulation (ns/2, sim.cesnet.cz)
Emulation (Nist Net)
Path capacity estimation tools (pathchar, pathrate, …)
Capture + follow-up analysis (tcpdump + tcptrace and others)
On-the-fly monitoring of TCP state variables (web100)

13. Capture + follow-up analysis
tcpdump -i eth1 -p -s 96 -w trace.log tcp and host tcp4-ge.uninett.no
tcptrace -l -f ’s_port!=12865’ -T -A300 -G trace.log
xplot  tcpplot 

14. On-the-fly monitoring of TCP state variables
- “Tools for end hosts to automatically
achieve high bandwidth”
- kernel data structures (approx. 120 variables), library and userland tools

15. On-the-fly monitoring of TCP state variables
readvars 0.01 CurrentCwnd CurrentRwndRcvd

16. Parallel TCP GridFTP, LFTP ~100 Mb Uninett -> Cesnet:
pget -n s Mb/s pget -n s Mb/s pget -n s Mb/s to s Mb/s pget -n s Mb/s to s Mb/s

17. Sven Ubik, ubik@cesnet.cz
Further research
E2E performance:
No changes to network.
Is it best-effort? Is it fair?
Ultimate goal of E2E performance:
Fully automatic adjustment to network and receiver conditions inside the operating system to maximize utilization of available
resources.
Use SACKs to avoid slow start after timeout
Are you far enough from us? You are welcome!
Sven Ubik,