1 FAST TCP for Multi-Gbps WAN: Experiments and Applications Les Cottrell & Fabrizio Coccetti– SLAC Prepared for the Internet2, Washington, April 2003

Slides:



Advertisements
Similar presentations
Martin Suchara, Ryan Witt, Bartek Wydrowski California Institute of Technology Pasadena, U.S.A. TCP MaxNet Implementation and Experiments on the WAN in.
Advertisements

TCP transfers over high latency/bandwidth network & Grid TCP Sylvain Ravot
FAST TCP Anwis Das Ajay Gulati Slides adapted from : IETF presentation slides Link:
Cheng Jin David Wei Steven Low FAST TCP: design and experiments.
Restricted Slow-Start for TCP William Allcock 1,2, Sanjay Hegde 3 and Rajkumar Kettimuthu 1,2 1 Argonne National Laboratory 2 The University of Chicago.
Ahmed El-Hassany CISC856: CISC 856 TCP/IP and Upper Layer Protocols Slides adopted from: Injong Rhee, Lisong Xu.
CUBIC : A New TCP-Friendly High-Speed TCP Variant Injong Rhee, Lisong Xu Member, IEEE v 0.2.
Maximizing End-to-End Network Performance Thomas Hacker University of Michigan October 5, 2001.
1 High Performance Active End-to- end Network Monitoring Les Cottrell, Connie Logg, Warren Matthews, Jiri Navratil, Ajay Tirumala – SLAC Prepared for the.
TCP friendlyness: Progress report for task 3.1 Freek Dijkstra Antony Antony, Hans Blom, Cees de Laat University of Amsterdam CERN, Geneva 25 September.
Recent Research in Congestion Control The problem of high bandwidth-delay product connections By Guillaume Marceau Presented for WPI CS577, Advanced Computer.
High-Performance Throughput Tuning/Measurements Davide Salomoni & Steffen Luitz Presented at the PPDG Collaboration Meeting, Argonne National Lab, July.
Cheng Jin David Wei Steven Low FAST TCP: Motivation, Architecture, Algorithms, Performance.
TCP on High-Speed Networks Sangtae Ha and Injong Rhee North Carolina State University.
1 Characterization and Evaluation of TCP and UDP-based Transport on Real Networks Les Cottrell, Saad Ansari, Parakram Khandpur, Ruchi Gupta, Richard Hughes-Jones,
FAST TCP in Linux Cheng Jin David Wei
PFLDNet Argonne Feb 2004 R. Hughes-Jones Manchester 1 UDP Performance and PCI-X Activity of the Intel 10 Gigabit Ethernet Adapter on: HP rx2600 Dual Itanium.
Presented by Anshul Kantawala 1 Anshul Kantawala FAST TCP: From Theory to Experiments C. Jin, D. Wei, S. H. Low, G. Buhrmaster, J. Bunn, D. H. Choe, R.
1 High Performance WAN Testbed Experiences & Results Les Cottrell – SLAC Prepared for the CHEP03, San Diego, March 2003
The Effects of Systemic Packets Loss on Aggregate TCP Flows Thomas J. Hacker May 8, 2002 Internet 2 Member Meeting.
Experiences in Design and Implementation of a High Performance Transport Protocol Yunhong Gu, Xinwei Hong, and Robert L. Grossman National Center for Data.
Large File Transfer on 20,000 km - Between Korea and Switzerland Yusung Kim, Daewon Kim, Joonbok Lee, Kilnam Chon
J. Bunn, D. Nae, H. Newman, S. Ravot, X. Su, Y. Xia California Institute of Technology High speed WAN data transfers for science Session Recent Results.
Maximizing End-to-End Network Performance Thomas Hacker University of Michigan October 26, 2001.
Technology for Using High Performance Networks or How to Make Your Network Go Faster…. Robin Tasker UK Light Town Meeting 9 September.
Network Tests at CHEP K. Kwon, D. Han, K. Cho, J.S. Suh, D. Son Center for High Energy Physics, KNU, Korea H. Park Supercomputing Center, KISTI, Korea.
1 Using Netflow data for forecasting Les Cottrell SLAC and Fawad Nazir NIIT, Presented at the CHEP06 Meeting, Mumbai India, February
FAST TCP in Linux Cheng Jin David Wei Steven Low California Institute of Technology.
High TCP performance over wide area networks Arlington, VA May 8, 2002 Sylvain Ravot CalTech HENP Working Group.
High-speed TCP  FAST TCP: motivation, architecture, algorithms, performance (by Cheng Jin, David X. Wei and Steven H. Low)  Modifying TCP's Congestion.
1 Overview of IEPM-BW - Bandwidth Testing of Bulk Data Transfer Tools Connie Logg & Les Cottrell – SLAC/Stanford University Presented at the Internet 2.
1 High performance Throughput Les Cottrell – SLAC Lecture # 5a presented at the 26 th International Nathiagali Summer College on Physics and Contemporary.
BIC Control for Fast Long-Distance Networks paper written by Injong Rhee, Lisong Xu & Khaled Harfoush (2004) Presented by Jonathan di Costanzo (2009/02/18)
Internet data transfer record between CERN and California Sylvain Ravot (Caltech) Paolo Moroni (CERN)
Iperf Quick Mode Ajay Tirumala & Les Cottrell. Sep 12, 2002 Iperf Quick Mode at LBL – Les Cottrell & Ajay Tirumala Iperf QUICK Mode Problem – Current.
NET100 Development of network-aware operating systems Tom Dunigan
Masaki Hirabaru NICT Koganei 3rd e-VLBI Workshop October 6, 2004 Makuhari, Japan Performance Measurement on Large Bandwidth-Delay Product.
IEPM. Warren Matthews (SLAC) Presented at the ESCC Meeting Miami, FL, February 2003.
1 Characterization and Evaluation of TCP and UDP-based Transport on Real Networks Les Cottrell, Saad Ansari, Parakram Khandpur, Ruchi Gupta, Richard Hughes-Jones,
Project Results Thanks to the exceptional cooperation spirit between the European and North American teams involved in the DataTAG project,
TCP transfers over high latency/bandwidth networks Internet2 Member Meeting HENP working group session April 9-11, 2003, Arlington T. Kelly, University.
Performance Engineering E2EpiPEs and FastTCP Internet2 member meeting - Indianapolis World Telecom Geneva October 15, 2003
GNEW2004 CERN March 2004 R. Hughes-Jones Manchester 1 Lessons Learned in Grid Networking or How do we get end-2-end performance to Real Users ? Richard.
TCP transfers over high latency/bandwidth networks & Grid DT Measurements session PFLDnet February 3- 4, 2003 CERN, Geneva, Switzerland Sylvain Ravot
FAST Protocols for High Speed Network David netlab, Caltech For HENP WG, Feb 1st 2003.
Final EU Review - 24/03/2004 DataTAG is a project funded by the European Commission under contract IST Richard Hughes-Jones The University of.
1 Evaluation of Advanced TCP stacks on Fast Long-Distance production Networks Prepared by Les Cottrell & Hadrien Bullot, Richard Hughes-Jones EPFL, SLAC.
INDIANAUNIVERSITYINDIANAUNIVERSITY Status of FAST TCP and other TCP alternatives John Hicks TransPAC HPCC Engineer Indiana University APAN Meeting – Hawaii.
FAST TCP Cheng Jin David Wei Steven Low netlab.CALTECH.edu GNEW, CERN, March 2004.
Masaki Hirabaru (NICT) and Jin Tanaka (KDDI) Impact of Bottleneck Queue on Long Distant TCP Transfer August 25, 2005 NOC-Network Engineering Session Advanced.
Network-aware OS DOE/MICS ORNL site visit January 8, 2004 ORNL team: Tom Dunigan, Nagi Rao, Florence Fowler, Steven Carter Matt Mathis Brian.
1 Achieving Record Speed Trans-Atlantic End-to-end TCP Throughput Les Cottrell – SLAC Prepared for the NANOG meeting, Salt Lake City, June 2003
R. Hughes-Jones Manchester
Prepared by Les Cottrell & Hadrien Bullot, SLAC & EPFL, for the
CENIC Road to Ten Gigabit: Biggest Fastest in the West
TransPAC HPCC Engineer
High Speed File Replication
TCP Performance over a 2.5 Gbit/s Transatlantic Circuit
Prepared by Les Cottrell & Hadrien Bullot, SLAC & EPFL, for the
MB-NG Review High Performance Network Demonstration 21 April 2004
Wide Area Networking at SLAC, Feb ‘03
High Performance Active End-to-end Network Monitoring
Characterization and Evaluation of TCP and UDP-based Transport on Real Networks Les Cottrell, Saad Ansari, Parakram Khandpur, Ruchi Gupta, Richard Hughes-Jones,
Breaking the Internet2 Land Speed Record: Twice
Wide-Area Networking at SLAC
Prepared by Les Cottrell & Hadrien Bullot, SLAC & EPFL, for the
High-Performance Data Transport for Grid Applications
Summer 2002 at SLAC Ajay Tirumala.
Presentation transcript:

1 FAST TCP for Multi-Gbps WAN: Experiments and Applications Les Cottrell & Fabrizio Coccetti– SLAC Prepared for the Internet2, Washington, April Partially funded by DOE/MICS Field Work Proposal on Internet End-to-end Performance Monitoring (IEPM), by the SciDAC base program.

2 Outline High throughput challenges New TCP stacks Tests on Unloaded (testbed) links –Performance of multi-streams –Performance of various stacks Tests on Production networks –Stack comparisons with single streams –Stack comparisons with multiple streams –Fairness Where do I find out more?

3 High Speed Challenges After a loss it can take over an hour for stock TCP (Reno) to recover to maximum throughput at 1Gbits/s –i.e. loss rate of 1 in ~ 2 Gpkts (3Tbits), or BER of 1 in 3.6*10 12 PCI bus limitations (66MHz * 64 bit = 4.2Gbits/s at best) At 2.5Gbits/s and 180msec RTT requires 120MByte window Some tools (e.g. bbcp) will not allow a large enough window – (bbcp limited to 2MBytes) Slow start problem at 1Gbits/s takes about 5-6 secs for 180msec link, –i.e. if want 90% of measurement in stable (non slow start), need to measure for 60 secs –need to ship >700MBytes at 1Gbits/s Sunnyvale-Geneva, 1500Byte MTU, stock TCP

4 New TCP Stacks Reno (AIMD) based, loss indicates congestion –Back off less when see congestion –Recover more quickly after backing off Scalable TCP: exponential recovery –Tom Kelly, Scalable TCP: Improving Performance in Highspeed Wide Area Networks Submitted for publication, December High Speed TCP: same as Reno for low performance, then increase window more & more aggressively as window increases using a table Vegas based, RTT indicates congestion –Caltech FAST TCP, quicker response to congestion, but … Standard Scalable High Speed cwnd=38pkts~0.5Mbits

5 Typical testbed 12*2cpu servers 4 disk servers GSRGSR 6*2cpu servers Sunnyvale 6*2cpu servers 4 disk servers OC192/POS (10Gbits/s) 2.5Gbits/s T640T640 Sunnyvale section deployed for SC2002 (Nov 02) (EU+US) Geneva Chicago SNV CHIAMS GVA > 10,000 km

6 Testbed Collaborators and sponsors Caltech: Harvey Newman, Steven Low, Sylvain Ravot, Cheng Jin, Xiaoling Wei, Suresh Singh, Julian Bunn SLAC: Les Cottrell, Gary Buhrmaster, Fabrizio Coccetti LANL: Wu-chun Feng, Eric Weigle, Gus Hurwitz, Adam Englehart NIKHEF/UvA: Cees DeLaat, Antony Antony CERN: Olivier Martin, Paolo Moroni ANL: Linda Winkler DataTAG, StarLight, TeraGrid, SURFnet, NetherLight, Deutsche Telecom, Information Society Technologies Cisco, Level(3), Intel DoE, European Commission, NSF

7 Windows and Streams Well accepted that multiple streams (n) and/or big windows are important to achieve optimal throughput Effectively reduces impact of a loss by 1/n, and improves recovery time by 1/n Optimum windows & streams changes with changes (e.g. utilization) in path, hard to optimize n Can be unfriendly to others

8 Even with big windows (1MB) still need multiple streams with Standard TCP Above knee performance still improves slowly, maybe due to squeezing out others and taking more than fair share due to large number of streams ANL, Caltech & RAL reach a knee (between 2 and 24 streams) above this gain in throughput slow

9 Stock vs FAST TCP MTU=1500B Need to measure all parameters to understand effects of parameters, configurations: –Windows, streams, txqueuelen, TCP stack, MTU, NIC card –Lot of variables Examples of 2 TCP stacks –FAST TCP no longer needs multiple streams, this is a major simplification (reduces # variables to tune by 1) Stock TCP, 1500B MTU 65ms RTT FAST TCP, 1500B MTU 65ms RTT FAST TCP, 1500B MTU 65ms RTT

10 TCP stacks with 1500B txqueuelen

11 Jumbo frames, new TCP stacks at 1 Gbits/s SNV-GVA But: Jumbos not part of GE or 10GE standard Not widely deployed in end networks

12 Production network tests SLACCERN SURFnet APAN Abilene CalREN Stanford ESnet CHICAGO SEATTL E CHI GVA Caltech RTT = 25 ms AMS APAN RTT = 147 ms NIKHEF RTT = 158 ms CERN RTT = 202 ms OC 48 OC 192 OC 12 OC 48 OC 12 SNV OC 12 All 6 hosts have 1GE interfaces (2 SLAC hosts send simultaneously) Competing flows, no jumbos Host running “New” TCP Host running Reno TCP Remote host

13 High Speed TCP vs Reno – 1 Stream Checked Reno vs Reno 2 hosts and very similar as expected 2 separate SLAC sending simultaneously to 1 receiver (2 iperf processes), 8MB window, pre-flush TCP config, 1500B MTU RTT bursty = congestion?

14 Nb large RTT=congestion?

15 Large RTTs => poor FAST

16 Scalable vs multi-streams SLAC to CERN, duration 60s, RTT 207ms, 8MB window

17 FAST & Scalable vs. Multi-stream Reno (SLAC>CERN ~230ms) Reno 1 streams 87 Mbits/s average FAST 1 stream 244 Mbits/s average Bottleneck capacity 622Mbits/s For short duration, very noisy, hard to distinguish Reno 8 streams 150 Mbits/s average FAST 1 stream 200 Mbits/s average Congestion events often sync

18 Scalable & FAST TCP with 1 stream vs Reno with n streams

19 Fairness FAST vs Reno Reno alone 221Mbps Fast alone 240Mbps Reno (45Mbps) & FAST (285Mbps) competing 1 Stream, 16MB window, SLAC to CERN

20 Summary (very preliminary) With single flow & empty network: –Can saturate 2.5 Gbps with standard TCP & jumbos –Can saturate 1Gbps with new stacks & 1500B frame or with standard & jumbos With production network, –FAST can take a while to get going –Once going, FAST TCP with one stream looks good compared to multi-stream RENO –FAST can back down early compared to RENO –More work needed on fairness Scalable –Does not look as good vs. multi-stream Reno

21 What’s next? Go beyond 2.5Gbits/s Disk-to-disk throughput & useful applications –Need faster cpus (extra 60% MHz/Mbits/s over TCP for disk to disk), understand how to use multi-processors Further evaluate new stacks with real-world links, and other equipment –Other NICs –Response to congestion, pathologies –Fairness –Deploy for some major (e.g. HENP/Grid) customer applications Understand how to make 10GE NICs work well with 1500B MTUs Move from “hero” demonstrations to commonplace

22 More Information 10GE tests –www-iepm.slac.stanford.edu/monitoring/bulk/10ge/www-iepm.slac.stanford.edu/monitoring/bulk/10ge/ –sravot.home.cern.ch/sravot/Networking/10GbE/10GbE_test.htmlsravot.home.cern.ch/sravot/Networking/10GbE/10GbE_test.html TCP stacks –netlab.caltech.edu/FAST/netlab.caltech.edu/FAST/ –datatag.web.cern.ch/datatag/pfldnet2003/papers/kelly.pdfdatatag.web.cern.ch/datatag/pfldnet2003/papers/kelly.pdf – Stack comparisons –www-iepm.slac.stanford.edu/monitoring/bulk/fast/www-iepm.slac.stanford.edu/monitoring/bulk/fast/ – –www-iepm.slac.stanford.edu/monitoring/bulk/tcpstacks/www-iepm.slac.stanford.edu/monitoring/bulk/tcpstacks/

23 Extras

24 FAST TCP vs. Reno – 1 stream N.b. RTT curve for Caltech shows why FAST performs poorly against Reno (too polite?)

25 Scalable vs. Reno - 1 stream 8MB windows, 2 hosts, competing

26 Other high speed gotchas Large windows and large number of streams can cause last stream to take a long time to close. Linux memory leak Linux TCP configuration caching What is the window size actually used/reported 32 bit counters in iperf and routers wrap, need latest releases with 64bit counters Effects of txqueuelen (number of packets queued for NIC) Routers do not pass jumbos Performance differs between drivers and NICs from different manufacturers –May require tuning a lot of parameters