1. High Speed File Replication
Les Cottrell & Davide Salomoni – SLAC
Presented at ESCC meeting, San Ramon, CA Nov. 2000
Partially funded by DOE/MICS Field Work Proposal on Internet End-to-end Performance Monitoring (IEPM), also supported by IUPAP

2. Overview PPDG & replication High thruput measurements Methodology LAN
WAN
SC2000
Packet reordering
Impact on others
How do we measure the QoS
Introduction to PingER and active end-to-end measurement methodology
Problem areas illustrated by results from PingER:
Generally, e.g. S. America, Spain, China, Germany to .edu & .ca
How do E. Europe & Russia look?
How does performance affect applications
Validating ping measurements and impact on FTP & Web performance
Overview of impact of performance on applications including , web, FTP, interactive apps
Detailed look at bulk data transfer expectations for HENP sites
Detailed look at critical performance metrics (RTT, loss, jitter, availability) and impact on VoIP
What can be done to improve QoS:
More bandwidth
Reserved bandwidth
Differentiated services

3. Participants in PPDG & GriPhyN
Projects
Nets
Caltech
SLAC
CalREN
NTON
ESNet
Abilene
SDSC
MREN
Fermilab
Wisconsin
Indiana
Boston
BNL
JLAB
ANL
Florida
LBNL/UCB
Sites

4. Site to site replication service
PRIMARY SITE
Data Acquisition,
CPU, Disk,
Tape Robot
SECONDARY SITE
CPU, Disk,
Tape Robot
Network Protocols Tuned for High Throughput
Use of DiffServ for (1) Predictable high priority delivery of high - bandwidth data streams (2) Reliable background transfers
Use of integrated instrumentation to detect/diagnose/correct problems in long-lived high speed transfers [NetLogger + DoE/NGI developments]
Coordinated reservaton/allocation techniques
for storage-to-storage performance

5. Measurement methodology
Iperf with multiple windows & streams
Selected 10 sites
Of critical interest to have high performance with SLAC
Can get iperf servers installed
Production links, do not control utilization

6. What does thruput depend on ?
Bandwidth end to end, i.e. min(BWlinks) AKA bottleneck bandwidth
Round Trip Time (RTT)
For TCP keep pipe full
window ~ RTT*BW
Thruput ~ 1/(sqrt(loss))
Competing utilization
Src
Rcv
RTT
ACK
Time
Pipe=RTT*BW

7. LAN thruput vs windows & streams
Hi-perf = big windows
& multiple streams
Default window size

8. LAN throughput measurements Sun/Solaris vs PIII/Linux

9. Progress towards goal: 100 Mbytes/s Site-to-Site
Focus on SLAC – Caltech over NTON;
Using NTON wavelength division fibers up & down W. Coast US;
Replaced Exemplar with 8*OC3 & Suns with Pentium IIIs & OC12 (622Mbps)
SLAC Cisco with OC48 (2.4Gbps) and 2 × OC12;
Caltech Juniper M160 & OC48
~500 Mbits/s single stream achieved recently over OC12.

10. Intercontinental high performance thruput on production networks SLAC (California US) to CERN (Switzerland)

11. SLAC to CERN thruput vs windows & streams
Hi-perf = big windows
& multiple streams
1MB
Improves ~ linearly
with streams for
small windows
100kB
64kB
16kB
32kB
8kB
Default window size

12. SLAC to CERN thruput vs windows & streams
>8Mbps
>10Mbps
Window size (kB)
<2Mbps
Parallel streams
Animate1
Animate2

13. E.g. thruput vs windows & streams
ANL
Caltech
Colorado
Window
IN2P3, FR
CERN, CH
Mbits/s
Daresbury, UK
Streams
I NFN, IT
Mbits/s
Mbits/s

14. Measured WAN thruput Poor agreement improvement w move to I2
Improve with time (upgrades)

15. Iperf throughput conclusions
Pathchar does a poor job of predicting thruput at these rates
Need > 1 stream
Can get close to max thruput with small (<=32Mbyte) with sufficient (5-10) streams
Improvements of 5 to 60 in thruput by using multiple streams & larger windows
Increasing streams often more effective than increasing windows
See www-iepm.slac.stanford.edu/monitoring/bulk/

16. File transfer + Compression
Bbftp tool from Gilles Farrache, IN2P3
10 streams & 256 kByte window SLAC > Lyon, FR got 25Mbps with NO compression
With compression cpu power is important
Sun E Mhz best could do was 13.6Mbps with 5 streams, more streams go slower
(e.g. 10 streams =>7.4Mbps)
Sun E MHz 26MHz with 10 streams
Compression factor 2-3 times
So data rate boosted to 27 – 41 Mbps (E4500) or Mbps (E450)

17. SC2000 WAN Challenge SC2000, Dallas to SLAC RTT ~ 48msec
SLAC/FNAL booth: Dell PowerEdge PIII 2 * 550MHz with 64bit PCI + Dell 850MHz both running Linux, each with GigE, connected to Cat 6009 with 2GigE bonded to SCInet Extreme switch
NTON/SLAC: OC48 to GSR to Cat 5500 Gig E to Sun E4500 4*460MHz and Sun E4500 6*336MHz
Internet 2: 300 Mbits/s
NTON: measured 990Mbits/s
==200TBytes in 20 days (BaBar yearly production
==250MBytes in 2 seconds (12K BaBar events or 1% error stats)
==copy 10 minute QuickTime movie in ~ 1 second
== 50K simultaneous VoIP calls (enough for 500 sites like SLAC)

18. Packet reordering Impacts TCP congestion avoidance algorithms
Is more common than had been thought
Took 256 PingER sites
Measured with pings
5 * 1 sec separations
50 back-to-back
Look for out of sequence

19. Impact on Others Make ping measurements with & without iperf loading
Loss loaded(unloaded)
RTT

20. Effect of load on other traffic
Measured ping RTT for 60 secs on normally unloaded link
with & without iperf load
Difference
~30-50ms

21. Possible alleviation Less than Best Effort QoS
Choose streams to optimize thruput/impact
Measure RTT and use to control number of streams

22. WAN thruput conclusions
High FTP performance across WAN links is possible
Even with 20-30Mbps bottleneck can do > 100Gbytes/day
OS must support big windows selectable by application
Need multiple parallel streams
Loss is important in particular interval between losses
Compression looks promising, but needs cpu power
Can get close to max thruput with small (<=32Mbyte) with sufficient (5-10) streams
Improvements of 5 to 60 in thruput by using multiple streams & larger windows
Impacts others users, need Less than Best Effort QoS service or friendlier applications

23. More Information This talk: IEPM/PingER home site Packet reordering
IEPM/PingER home site
www-iepm.slac.stanford.edu/
Bulk throughput measurements:
www-iepm.slac.stanford.edu/monitoring/bulk/
Effect of load on thruput & loss
www-iepm.slac.stanford.edu/monitoring/load/
Windows vs. streams
www-iepm.slac.stanford.edu/monitoring/bulk/window-vs-streams.html
SC2000 thruput to SLAC
www-iepm.slac.stanford.edu/monitoring/bulk/sc2k.html
Packet reordering
www-iepm.slac.stanford.edu/monitoring/reorder/