FAST TCP in Linux Cheng Jin David Wei

Slides:



Advertisements
Similar presentations
Helping TCP Work at Gbps Cheng Jin the FAST project at Caltech
Advertisements

Michele Pagano – A Survey on TCP Performance Evaluation and Modeling 1 Department of Information Engineering University of Pisa Network Telecomunication.
TCP transfers over high latency/bandwidth network & Grid TCP Sylvain Ravot
Ningning HuCarnegie Mellon University1 Improving TCP Startup Performance using Active Measurements Ningning Hu, Peter Steenkiste Carnegie Mellon University.
FAST TCP Anwis Das Ajay Gulati Slides adapted from : IETF presentation slides Link:
Cheng Jin David Wei Steven Low FAST TCP: design and experiments.
Congestion Control Created by M Bateman, A Ruddle & C Allison As part of the TCP View project.
Ahmed El-Hassany CISC856: CISC 856 TCP/IP and Upper Layer Protocols Slides adopted from: Injong Rhee, Lisong Xu.
Presentation by Joe Szymanski For Upper Layer Protocols May 18, 2015.
CUBIC : A New TCP-Friendly High-Speed TCP Variant Injong Rhee, Lisong Xu Member, IEEE v 0.2.
TDC365 Spring 2001John Kristoff - DePaul University1 Internetworking Technologies Transmission Control Protocol (TCP)
AQM for Congestion Control1 A Study of Active Queue Management for Congestion Control Victor Firoiu Marty Borden.
High speed TCP’s. Why high-speed TCP? Suppose that the bottleneck bandwidth is 10Gbps and RTT = 200ms. Bandwidth delay product is packets (1500.
1 Spring Semester 2007, Dept. of Computer Science, Technion Internet Networking recitation #7 TCP New Reno Vs. Reno.
Cheng Jin David Wei Steven Low FAST TCP: Motivation, Architecture, Algorithms, Performance.
TCP on High-Speed Networks Sangtae Ha and Injong Rhee North Carolina State University.
FAST TCP Speaker: Ray Veune: Room 1026 Date: 25 th October, 2003 Time:10:00am.
TCP in Heterogeneous Network Md. Ehtesamul Haque # P.
Presented by Anshul Kantawala 1 Anshul Kantawala FAST TCP: From Theory to Experiments C. Jin, D. Wei, S. H. Low, G. Buhrmaster, J. Bunn, D. H. Choe, R.
FAST Protocols for Ultrascale Networks netlab.caltech.edu/FAST Internet: distributed feedback control system  TCP: adapts sending rate to congestion 
1 High Performance WAN Testbed Experiences & Results Les Cottrell – SLAC Prepared for the CHEP03, San Diego, March 2003
The Effects of Systemic Packets Loss on Aggregate TCP Flows Thomas J. Hacker May 8, 2002 Internet 2 Member Meeting.
Courtesy: Nick McKeown, Stanford 1 TCP Congestion Control Tahir Azim.
Experiences in Design and Implementation of a High Performance Transport Protocol Yunhong Gu, Xinwei Hong, and Robert L. Grossman National Center for Data.
Utility, Fairness, TCP/IP Steven Low CS/EE netlab.CALTECH.edu Feb 2004.
Large File Transfer on 20,000 km - Between Korea and Switzerland Yusung Kim, Daewon Kim, Joonbok Lee, Kilnam Chon
These materials are licensed under the Creative Commons Attribution-Noncommercial 3.0 Unported license (
TCP Throughput Collapse in Cluster-based Storage Systems
CS/EE 145A Congestion Control Netlab.caltech.edu/course.
UDT: UDP based Data Transfer Yunhong Gu & Robert Grossman Laboratory for Advanced Computing University of Illinois at Chicago.
FAST TCP Cheng Jin David Wei Steven Low netlab.CALTECH.edu.
MaxNet NetLab Presentation Hailey Lam Outline MaxNet as an alternative to TCP Linux implementation of MaxNet Demonstration of fairness, quick.
FAST TCP in Linux Cheng Jin David Wei Steven Low California Institute of Technology.
High TCP performance over wide area networks Arlington, VA May 8, 2002 Sylvain Ravot CalTech HENP Working Group.
High-speed TCP  FAST TCP: motivation, architecture, algorithms, performance (by Cheng Jin, David X. Wei and Steven H. Low)  Modifying TCP's Congestion.
Paper Review: Latency Evaluation of Networking Mechanisms for Game Traffic Jin, Da-Jhong.
Internet data transfer record between CERN and California Sylvain Ravot (Caltech) Paolo Moroni (CERN)
Self-generated Self-similar Traffic Péter Hága Péter Pollner Gábor Simon István Csabai Gábor Vattay.
TERENA Networking Conference, Zagreb, Croatia, 21 May 2003 High-Performance Data Transport for Grid Applications T. Kelly, University of Cambridge, UK.
Murari Sridharan and Kun Tan (Collaborators: Jingmin Song, MSRA & Qian Zhang, HKUST.
CS640: Introduction to Computer Networks Aditya Akella Lecture 15 TCP – III Reliability and Implementation Issues.
Computer Networking Lecture 18 – More TCP & Congestion Control.
1 CS 4396 Computer Networks Lab TCP – Part II. 2 Flow Control Congestion Control Retransmission Timeout TCP:
CS640: Introduction to Computer Networks Aditya Akella Lecture 15 TCP – III Reliability and Implementation Issues.
SCinet Caltech-SLAC experiments netlab.caltech.edu/FAST SC2002 Baltimore, Nov 2002  Prototype C. Jin, D. Wei  Theory D. Choe (Postech/Caltech), J. Doyle,
TCP transfers over high latency/bandwidth networks Internet2 Member Meeting HENP working group session April 9-11, 2003, Arlington T. Kelly, University.
Performance Engineering E2EpiPEs and FastTCP Internet2 member meeting - Indianapolis World Telecom Geneva October 15, 2003
1 Computer Networks Congestion Avoidance. 2 Recall TCP Sliding Window Operation.
Murari Sridharan Windows TCP/IP Networking, Microsoft Corp. (Collaborators: Kun Tan, Jingmin Song, MSRA & Qian Zhang, HKUST)
TCP continued. Discussion – TCP Throughput TCP will most likely generate the saw tooth type of traffic. – A rough estimate is that the congestion window.
TCP transfers over high latency/bandwidth networks & Grid DT Measurements session PFLDnet February 3- 4, 2003 CERN, Geneva, Switzerland Sylvain Ravot
FAST Protocols for High Speed Network David netlab, Caltech For HENP WG, Feb 1st 2003.
Final EU Review - 24/03/2004 DataTAG is a project funded by the European Commission under contract IST Richard Hughes-Jones The University of.
INDIANAUNIVERSITYINDIANAUNIVERSITY Status of FAST TCP and other TCP alternatives John Hicks TransPAC HPCC Engineer Indiana University APAN Meeting – Hawaii.
1 Testing TCP Westwood+ over Transatlantic Links at 10 Gigabit/Second rate Saverio Mascolo Dipartimento di Elettrotecnica ed Elettronica Politecnico di.
FAST TCP Cheng Jin David Wei Steven Low netlab.CALTECH.edu GNEW, CERN, March 2004.
Masaki Hirabaru (NICT) and Jin Tanaka (KDDI) Impact of Bottleneck Queue on Long Distant TCP Transfer August 25, 2005 NOC-Network Engineering Session Advanced.
1 FAST TCP for Multi-Gbps WAN: Experiments and Applications Les Cottrell & Fabrizio Coccetti– SLAC Prepared for the Internet2, Washington, April 2003
1 Achieving Record Speed Trans-Atlantic End-to-end TCP Throughput Les Cottrell – SLAC Prepared for the NANOG meeting, Salt Lake City, June 2003
© 2006 Andreas Haeberlen, MPI-SWS 1 Monarch: A Tool to Emulate Transport Protocol Flows over the Internet at Large Andreas Haeberlen MPI-SWS / Rice University.
DART SI-8: Pilot long-distance high speed and secure data transfer between the Repositories DART Workshop on Infrastructure Chief Investigator: Dr. Asad.
Transport Protocols over Circuits/VCs
Lecture 19 – TCP Performance
Prepared by Les Cottrell & Hadrien Bullot, SLAC & EPFL, for the
CS640: Introduction to Computer Networks
Breaking the Internet2 Land Speed Record: Twice
Review of Internet Protocols Transport Layer
Presentation transcript:

FAST TCP in Linux Cheng Jin David Wei

netlab.caltech.edu Outline  Overview of FAST TCP.  Implementation Details.  SC2002 Experiment Results.  FAST Evaluation and WAN-in-Lab.

netlab.caltech.edu FAST vs. Linux TCP Distance = 10,037 km; Delay = 180 ms; MTU = 1500 B; Duration: 3600 s Linux TCP Experiments: Jan 28-29, Linux TCP txqlen= Linux TCP txqlen= Linux TCP txqlen= Transfer (GB) 1, FAST FAST Linux TCP txqlen=10000 Throughput (Mbps) Bmps Peta Flows

netlab.caltech.edu Aggregate Throughput Linux TCP Linux TCP FAST Average utilization 19% 27% 92% FAST  Standard MTU  Utilization averaged over 1hr txq=100txq= % 16% 48% Linux TCP Linux TCP FAST 2G 1G

netlab.caltech.edu Summary of Changes  RTT estimation: fine-grain timer.  Fast convergence to equilibrium.  Delay monitoring in equilibrium.  Pacing: reducing burstiness.

netlab.caltech.edu FAST TCP Flow Chart Slow Start Fast Convergence Equilibrium Loss Recovery Normal Recovery Time-out

netlab.caltech.edu RTT Estimation  Measure queueing delay.  Kernel timestamp with s resolution.  Use SACK to increase the number of RTT samples during recovery.  Exponential averaging of RTT samples to increase robustness.

netlab.caltech.edu Fast Convergence  Rapidly increase or decrease cwnd toward equilibrium.  Monitor the per-ack queueing delay to avoid overshoot.

netlab.caltech.edu Equilibrium  Vegas-like cwnd adjustment in large time-scale -- per RTT.  Small step-size to maintain stability in equilibrium.  Per-ack delay monitoring to enable timely detection of changes in equilibrium.

netlab.caltech.edu Pacing  What do we pace? Increment to cwnd.  Time-Driven vs. event-driven. Trade-off between complexity and performance. Timer resolution is important.

netlab.caltech.edu Time-Based Pacing cwnd increments are scheduled at fixed intervals. dataackdata

netlab.caltech.edu Event-Based Pacing Detect sufficiently large gap between consecutive bursts and delay cwnd increment until the end of each such burst.

SCinet Caltech-SLAC experiments netlab.caltech.edu/FAST SC2002 Baltimore, Nov 2002 Experiment SunnyvaleBaltimore Chicago Geneva 3000km 1000km 7000km C. Jin, D. Wei, S. Low FAST Team and Partners Internet: distributed feedback system R f (s) R b ’ (s) x p TCP AQM Theory FAST TCP  Standard MTU  Peak window = 14,255 pkts  Throughput averaged over > 1hr  925 Mbps single flow/GE card 9.28 petabit-meter/sec 1.89 times LSR  8.6 Gbps with 10 flows 34.0 petabit-meter/sec 6.32 times LSR  21TB in 6 hours with 10 flows Implementation  Sender-side modification  Delay based Highlights Geneva-Sunnyvale Baltimore-Sunnyvale FAST I2 LSR #flows

netlab.caltech.edu Network (Sylvain Ravot, caltech/CERN)

netlab.caltech.edu FAST BMPS Internet2 Land Speed Record FAST Geneva-Sunnyvale Baltimore-Sunnyvale #flows FAST  Standard MTU  Throughput averaged over > 1hr

netlab.caltech.edu Aggregate Throughput 1 flow 2 flows 7 flows 9 flows 10 flows Average utilization 95% 92% 90% 88% FAST  Standard MTU  Utilization averaged over > 1hr 1hr 6hr 1.1hr6hr

netlab.caltech.edu Caltech-SLAC Entry Rapid recovery after possible hardware glitch Power glitch Reboot Mbps ACK traffic

SCinet Caltech-SLAC experiments netlab.caltech.edu/FAST SC2002 Baltimore, Nov 2002  Prototype C. Jin, D. Wei  Theory D. Choe (Postech/Caltech), J. Doyle, S. Low, F. Paganini (UCLA), J. Wang, Z. Wang (UCLA)  Experiment/facilities Caltech: J. Bunn, C. Chapman, C. Hu (Williams/Caltech), H. Newman, J. Pool, S. Ravot (Caltech/CERN), S. Singh CERN: O. Martin, P. Moroni Cisco: B. Aiken, V. Doraiswami, R. Sepulveda, M. Turzanski, D. Walsten, S. Yip DataTAG: E. Martelli, J. P. Martin-Flatin Internet2: G. Almes, S. Corbato Level(3): P. Fernes, R. Struble SCinet: G. Goddard, J. Patton SLAC: G. Buhrmaster, R. Les Cottrell, C. Logg, I. Mei, W. Matthews, R. Mount, J. Navratil, J. Williams StarLight: T. deFanti, L. Winkler TeraGrid: L. Winkler  Major sponsors ARO, CACR, Cisco, DataTAG, DoE, Lee Center, NSF Acknowledgments

netlab.caltech.edu Evaluating FAST  End-to-End monitoring doesn’t tell the whole story.  Existing network emulation (dummynet) is not always enough.  Better optimization if we can look inside and understand the real network.

netlab.caltech.edu Dummynet and Real Testbed

netlab.caltech.edu Dummynet Issues  Not running on a real-time OS -- imprecise timing.  Lack of priority scheduling of dummynet events.  Bandwidth fluctuates significantly with workload.  Much work needed to customize dummynet for protocol testing.

netlab.caltech.edu 10 GbE Experiment  Long-distance testing of Intel 10GbE cards.  Sylvain Ravot (Caltech) achieved 2.3 Gbps using single stream with jumbo frame and stock Linux TCP.  Tested HSTCP, Scalable TCP, FAST, and stock TCP under Linux. 1500B MTU: 1.3 Gbps SNV -> CHI; 9000B MTU: 2.3 Gbps SNV -> GVA

netlab.caltech.edu TCP Loss Mystery  Frequent packet loss with 1500-byte MTU. None with larger MTUs.  Packet loss even when cwnd is capped at packets.  Routers have large queue size of 4000 packets.  Packets captured at both sender and receiver using tcpdump.

netlab.caltech.edu How Did the Loss Happen? loss detected

netlab.caltech.edu How Can WAN-in-Lab Help?  We will know exactly where packets are lost.  We will also know the sequence of events (packet arrivals) that lead to loss.  We can either fix the problem in the network if any, or improve the protocol.

netlab.caltech.edu Conclusion  FAST improves the end-to-end performance of TCP.  Many issues are still to be understood and resolved.  WAN-in-Lab can help make FAST a better protocol.