1. Achieving high performance throughput in production networks
Les Cottrell – SLAC
Presented at the Internet 2 HENP Networking Working Group kickoff meeting at Internet 2 Ann Arbor, Michigan, Oct 26 ‘01
Partially funded by DOE/MICS Field Work Proposal on Internet End-to-end Performance Monitoring (IEPM), also supported by IUPAP

2. High Speed Bulk Throughput
Driven by:
Data intensive science, e.g. data grids
HENP data rates, e.g. BaBar 300TB/year, collection doubling yearly, i.e. PBytes in couple of years
Data rate from experiment today ~ 20MBytes/s ~ 200GBytes/d
Multiple regional computer centers (e.g. Lyon-FR, RAL-UK, INFN-IT, LBNL-CA, LLNL-CA, Caltech-CA) need copies of data
Tier A gets 1/3 data in 1/3 year (full rate), SLAC does not keep copy
Boeing 747 high throughput, BUT poor latency (~ 2 weeks) & very people intensive
So need high-speed networks and ability to utilize
High speed today = few hundred GBytes/day (100GB/d ~ 10Mbits/s)
Data vol
Moore’s law

3. How to measure network throughput
Selected about 2 dozen major collaborator sites in US, CA, JP, FR, CH, IT, UK over last year
Of interest to SLAC
Can get logon accounts
Use iperf
Choose window size and # parallel streams
Run for 10 seconds together with ping (loaded)
Stop iperf, run ping (unloaded) for 10 seconds
Change window or number of streams & repeat
Record # streams, window, throughput (Mbits/s), loaded & unloaded ping responses, cpu utilization, real time
Verify window sizes are set properly by using tcpdump can’t believe what application tells you LM
Note cpu speeds, interface speeds, operating system, path characteristics

4. Typical results 64kB 100kB Broke 100Mbps Trans Atlantic 32kB Barrier
Today
Hi-thru usually
= big windows
& multiple streams
Typical results
Improves ~ linearly
with streams for
small windows
64kB
100kB
Broke 100Mbps
Trans Atlantic
Barrier
32kB
16kB
8kB
Solaris Default window size

5. Windows vs Streams
Often for fixed streams*window product, streams are more effective than window size, e.g. SLAC>CERN, Jul ‘01:
There is an optimum number of streams above which performance flattens out
Common for throughputs to be asymmetric
more congestion one way, different routes, host dependencies
Window
Streams
Mbps
W*S
64KB 25 68
1600KB
128KB 12 56
1536KB
256KB 6 66
512KB 3 62
1024KB 2 26
2048KB 1 17
4096KB

6. Windows vs Streams Multi-streams often more effective than windows
more agile in face of congestion
Often easier to set up
Need root to configure kernel to set max window
Network components may not support big windows
Some OS’ treat max windows strangely D
May be able to take advantage of multiple paths
But:
may be considered over-aggressive (RFC 2914) p
can take more cpu cycles
how to know how many streams?

7. Iperf client CPU utilization
As expected increases with throughput (mainly kernel)
d 0.7*MHz/Mbits/s
For fixed throughput
Fewer streams take less cpu J 6
E.g. 1-4 streams take 20% less cpu than 8-16 streams for same throughput (if can get it)
With 1MHz ~ 1 Mbps, cpus are still slightly ahead of being able to keep up with bandwidth
Need to keep testing, evaluating faster cpus.
What happens next year when 10Gbps Ethernet becomes standard
Multiple cpus are not as effective as a single cpu for how they can handle the network interface.
Multiple cpus with multiple interfaces may be able to make better use of multiple streams

8. Throughput quality improvements
TCPBW < MSS/(RTT*sqrt(loss))
80% annual improvement ~ factor 10/4yr
Note this is measured from ESnet Labs in US so biased towards US sites
Lines are medians for many measured pairs, so half pairs have better than shown
China
Note E. Europe keeping up
Macroscopic Behavior of the TCP Congestion Avoidance Algorithm, Matthis, Semke, Mahdavi, Ott, Computer Communication Review 27(3), July 1997

9. Bandwidth changes with time 1/2
Short term competing cross-traffic, other users, factors of 3-5 observed in 1 minute
Long term: link, route upgrades, factors 3-16 in 12 months
All hosts had 100Mbps NICs.
Recently have measured 105Mbps SLAC > IN2P3 and 340Mbps Caltech > SLAC with GE

10. Network Simulator (ns-2)
From UCB, simulates network
Choice of stack (Reno, Tahoe, Vegas, SACK…)
RTT, bandwidth, flows, windows, queue lengths …
Compare with measured results
Agrees well
Confirms observations (e.g. linear growth in throughput for small window sizes as increase number of flows)
The peaks measured for a single flow are not understood, and only occur on some links. As can be seen they are not predicted from the simulation.

11. Agreement of ns2 with observed

12. Ns-2 thruput & loss predict
90%
Indicates on unloaded link can get 70% of available bandwidth without causing noticeable packet loss
Can get over 80-90% of available bandwidth
Can overdrive: no extra throughput BUT extra loss

13. Simulator benefits No traffic on network (nb throughput can use 90%)
Can do what if experiments
No need to install iperf servers or have accounts
No need to configure host to allow large windows
BUT
Need to estimate simulator parameters, e.g.
RTT use ping or synack
Bandwidth, use pchar, pipechar etc., moderately accurate
AND its not the real thing
Need to validate vs. observed data
Need to simulate cross-traffic etc

14. Impact on Others Make ping measurements with & without iperf loading
Loss loaded(unloaded)
RTT
Looking at how to avoid impact: e.g. QBSS/LBE, application pacing, control loop on stdev(RTT) reducing streams, want to avoid scheduling

15. File Transfer Used bbcp (written by Andy Hanushevsky)
similar methodology to iperf, except ran for file length rather than time, provides incremental throughput reports, supports /dev/zero, adding duration
looked at /afs/, /tmp/, /dev/null
checked different file sizes
Behavior with windows & streams similar to iperf
Thrubbcp ~0.8*Thruiperf
For modest throughputs (< 50Mbits/s) rates are independent of whether destination is /afs/, /tmp/ or /dev/null.
Cpu utilization ~ 1MHz/Mbit/s is ~ 20% > than for iperf

16. Application rate-limiting
Bbcp has transfer rate limiting
Could use network information (e.g. from Web100 or independent pinging) to bbcp to reduce/increase its transfer rate, or change number of parallel streams
No rate limiting, 64KB window, 32 streams
15MB/s rate limiting, 64KB window, 32 streams

17. Using bbcp to make QBSS measurements
Run bbcp src data /dev/zero, dst=/dev/null, report throughput at 1 second intervals
with TOS=32 (QBSS)
After 20 s. run bbcp with no TOS bits specified (BE)
After 20 s. run bbcp with TOS=40 (priority)
After 20 more secs turn off Priority
After 20 more secs turn off BE

18. QBSS test bed with Cisco 7200s
Set up QBSS testbed
Configure router interfaces
3 traffic types:
QBSS, BE, Priority
Define policy, e.g.
QBSS > 1%, priority < 30%
Apply policy to router interface queues
10Mbps
100Mbps
100Mbps
1Gbps
100Mbps
class-map match-all qbss
match ip dscp 8
class-map match-all hi-priority
match ip dscp 10
policy-map qos-policy
class qbss
bandwidth percent 1
random-detect
class hi-priority
bandwidth percent 30
class class-default
bandwidth percent 69
interface ...
service-policy output qos-policy

19. Example of effects
Also tried: 1 stream for all, and priority at 30%

20. QBSS with Cisco 6500 6500s + Policy Feature Card (PFC) BE 100%
Routing by PFC2, policing on switch interfaces
2 queues, 2 thresholds each
QBSS assigned to own queue with 5% bandwidth – guarantees QBSS gets something
BE & Priority traffic in 2nd queue with 95% bandwidth
Apply ACL to switch port to police Priority traffic to < 30% BE
100%
Cisco 6500s +
MSFC/Sup2
QBSS
(~5%)
Priority
(30%)
100Mbps
1Gbps
1Gbps
1Gbps
1Gbps
Time

21. Impact on response time (RTT)
Run ping with Iperf loading with various QoS settings, iperf ~ 93Mbps
No iperf ping avg RTT ~ 300usec (regardless of QoS)
Iperf = QBSS, ping=BE or Priority: RTT~550usec
70% greater than unloaded
Iperf=Ping QoS (exc. Priority) then RTT~5msec
> factor of 10 larger RTT than unloaded
If both ping & iperf have QoS=Priority then ping RTT very variable since iperf limited to 30%
RTT quick when iperf limited, long when iperf transmits

22. Possible HEP usage
Apply priority to lower volume interactive voice/video-conferencing and real time control
Apply QBSS to high volume data replication
Leave the rest as Best Effort
Since 40-65% of bytes to/from SLAC come from a single application, we have modified to enable setting of TOS bits
Need to identify bottlenecks and implement QBSS there
Bottlenecks tend to be at edges so hope to try with a few HEP sites

23. Acknowledgements for SC2001
Many people assisted in getting accounts, setting up servers, providing advice, software etc.
Suresh Man Singh, Harvey Newman, Julian Bunn (Caltech), Andy Hanushevsky, Paola Grosso, Gary Buhrmaster, Connie Logg (SLAC), Olivier Martin (CERN), Loric Totay, Jerome Bernier (IN2P3), Dantong Yu (BNL), Robin Tasker, Paul Kummer (DL), John Gordon (RL), Brian Tierney, Bob Jacobsen, (LBL), Stanislav Shalunov (Internet 2), Joe Izen (UT Dallas), Linda Winkler, Bill Allcock (ANL), Ruth Pordes, Frank Nagy (FNAL), Emanuele Leonardi (INFN), Chip Watson (JLab), Yukio Karita (KEK), Tom Dunigan (ORNL), John Gordon (RL), Andrew Daviel (TRIUMF), Paul Avery, Greg Goddard (UFL), Paul Barford, Miron Livny (UWisc), Shane Canon (NERSC), Andy Germain (NASA), Andrew Daviel (TRIUMF), Richard baraniuk, Rold Reidi (Rice).

24. SC2001 demo
Send data from SLAC/FNAL booth computers (emulate a tier 0 or 1 HENP site) to over 20 other sites with good connections in about 6 countries
Throughputs from SLAC range from 3Mbps to > 300Mbps
Part of bandwidth challenge proposal
Saturate 2Gbps connection to floor network
Apply QBSS to some sites, priority to a few and rest Best Effort
See how QBSS works at high speeds
Competing bulk throughput streams
Interactive low throughput streams, look at RTT with ping

25. WAN thruput conclusions
High FTP performance across WAN links is possible
Even with 20-30Mbps bottleneck can do > 100Gbytes/day
Can easily saturate a fast Ethernet interface over WAN
Need GE NICs, > OC3 WANs & to improve performance
Performance is improving
OS must support big windows selectable by application
Need multiple parallel streams in some cases
Loss is important in particular interval between losses
Can get close to max thruput with small (<=32Mbyte) with sufficient (5-10) streams
Improvements of 5 to 60 in thruput by using multiple streams & larger windows
Impacts others users, QBSS looks hopeful

26. More Information IEPM/PingER home site: Bulk throughput site:
www-iepm.slac.stanford.edu/
Bulk throughput site:
www-iepm.slac.stanford.edu/monitoring/bulk/
Transfer tools:
TCP Tuning:
www-didc.lbl.gov/tcp-wan.html
QBSS measurements
www-iepm.slac.stanford.edu/monitoring/qbss/measure.html

27. Extra supporting slides

28. Streams usually share bandwidth
TCP fair share with best effort

29. Progress towards goal: 100 Mbytes/s Site-to-Site
Focus on SLAC – Caltech over NTON;
Using NTON wavelength division fibers up & down W. Coast US;
Replaced Exemplar with 8*OC3 & Suns with Pentium IIIs & OC12 (622Mbps)
SLAC Cisco with OC48 (2.4Gbps) and 2 × OC12;
Caltech Juniper M160 & OC48
~500 Mbits/s single stream achieved recently over single OC12.
2 OC12s, 1 machine at Caltech and 2 machines at SLAC gave >~ 600Mbits/

30. Optimizing streams Choose # streams to optimize throughput/impact
Measure RTT from Web100
App controls # streams

31. SC2000 WAN Challenge SC2000, Dallas to SLAC RTT ~ 48msec
SLAC/FNAL booth: Dell PowerEdge PIII 2 * 550MHz with 64bit PCI + Dell 850MHz both running Linux, each with GigE, connected to Cat 6009 with 2GigE bonded to Extreme SC2000 floor switch
NTON: OC48 to GSR to Cat 5500 Gig E to Sun E4500 4*460MHz and Sun E4500 6*336MHz
Internet 2: 300 Mbits/s
NTON 960Mbits/s
Details:
www-iepm.slac.stanford.edu/monitoring/bulk/sc2k.html

32. Optimum window & streams 1/2
Require RTT * BandWidth d a*window*streams
E.g. for SLAC CERN:
Window*streams=159msec*85Mbits/s~1.7Mbytes
Over 1000 pkts in pipe at a time
Use ping for RTT
Pipechar for BW
W KB S
W * S MB 64 30
1.92
128 11
1.41
256 7
0.93
512 3
1.5

33. LAN MTU measure @ Caltech
2 Dells dual 730MHz & dual 860MHz PCs
Connect with Syskonnect dual port GE interfaces,
Linux smp
Run iperf & ttcp, window
Got 475Mbps/port for 1.5KB MTU
Got 960MBytes/port for 9KB MTU
Windows 2000
Run Microsoft “speedy” application
Got 820Mbits/s with 9KB MTU, max IRQ 3000/s, 128KB buffers, 20MB window size for TCP, single stream
Used Microsoft “speedy” application, varied buffer lengths, the max IRQ rate on the cards, frame size used and number of threads (1 or 2). Set windows registry on both machines to 20MB and Tcp1323Opts to 3 (settings used by Microsoft to achieve Land speed record. Got ~ 820Mbits/s with 128KB buffers and 9K MTU and set a max of 3000 IRQs per second. Much poorer results if use defaults.

34. LAN windows & streams With today’s GE interfaces need big windows:
e.g. 2 hosts on same Ethernet switch, RTT~300usec, needs window size of 300KBytes
90-100MB/s for window >= 64KB & streams 5-8
windows*streams does not work
windows alone no good
streams alone no good
Between Sun E420R with 4 cpus at 450MHz and Netra 1405 with 4 cpus running at 440MHz, running Solaris 5.7

35. Bandwidth changes with time 2/2
CERN to Caltech for last year and a half
Improvements come in steps, when bottleneck upgraded

36. Other Improvements Improved TCP stacks Web100 experimental TCP stack
Faster recovery after losses, selective acknowledgement
Web100 experimental TCP stack
Allows read/write access to TCP internal variables
Can provide improved monitoring & diagnostics
So application can tune stack on the fly

37. Applications
Main network application focus today is on replication at multiple sites worldwide (mainly N. America, Europe and Japan)
Need fast, secure, easy to use, extendable way to copy data between sites
Need to interactive and real time at same time, e.g. experiment control, video & voice conferencing
HEP community has developed 2 major (freely available) applications to meet replication need: bbftp and bbcp

38. Bbcp
Peer-to-peer copy program with multiple (<=64) streams, large window support, secure password exchange (ssh control path, single use passwords (data path)), similar syntax to scp
bbcp
Data
Source
Sink
Agent
C++ component design allows testing new algorithms (relatively easy to extend)
Peer-to-peer
No server, if have program, have service (usually no need for admins), any node can act as source or sink, 3rd party copies
Provides sequential I/O (e.g. from /dev/zero, to pipe or tape or /dev/null) and progress reporting

39. Bbcp: algorithms Data pipelining Coordinated buffers
Multiple streams “simultaneously” pushed
Automatically adapts to router traffic shaping
Can control maximum rate
Can write to tape, read from /dev/zero, write to /dev/null, pipe
Check-pointing (resume failed transmission)
Coordinated buffers
All buffers same-sized end-to-end
Page aligned buffers
Allows direct I/O on many file-systems (e.g. Veritas)