1. FAST Protocols for High Speed Network David Wei @ netlab, Caltech For HENP WG, Feb 1st 2003

2. FAST Protocols for Ultrascale Networks netlab.caltech.edu/FAST Internet: distributed feedback control system TCP: adapts sending rate to congestion AQM: feeds back congestion information R f (s) R b ’ (s) xy pq TCPAQM Theory Calren2/Abilene Chicago Amsterdam CERN Geneva SURFNet StarLight WAN in Lab Caltech research & production networks Multi-Gbps 50-200ms delay Experiment 155Mb/s slow start equilibrium FAST recovery FAST retransmit time out 10Gb/s Implementation Students Choe (Postech/CIT) Hu (Williams) J. Wang (CDS) Z.Wang (UCLA) Wei (CS) Industry Doraiswami (Cisco) Yip (Cisco) Faculty Doyle (CDS,EE,BE) Low (CS,EE) Newman (Physics) Paganini (UCLA) Staff/Postdoc Bunn (CACR) Jin (CS) Ravot (Physics) Singh (CACR) Partners CERN, Internet2, CENIC, StarLight/UI, SLAC, AMPATH, Cisco People

3. FAST project Goal: Protocols (TCP/AQM) for ultrascale networks  Bandwidth: 10Mbps ~ > 100 Gbps  Delay: 50-200ms delay  Research: Theory, algorithms, design, implement, demo, deployment Urgent Need: –Large amount of Data to share (500TB in SLAC) –Typical file in SLAC transfer ~1 TB (15 mins with 10Gbps)

4. HEP Network (DataTAG) NL SURFnet GENEVA UK SuperJANET4 ABILEN E ESNET CALRE N It GARR-B GEANT NewYork Fr Renater STAR-TAP STARLIGHT Wave Triangle 2.5 Gbps Wavelength Triangle 2002 10 Gbps Triangle in 2003 Newman (Caltech)

5. Projected performance Ns-2: capacity = 155Mbps, 622Mbps, 2.5Gbps, 5Gbps, 10Gbps 100 sources, 100 ms round trip propagation delay ’01 155 ’02 622 ’03 2.5 ’04 5 ’05 10 J. Wang (Caltech)

6. Throughput as function of the time Chicago -> CERN Linux kernel 2.4.19 Traffic generated by iperf (I measure the throughput over the last 5 sec) TCP single stream RTT = 119msMTU = 1500 Duration of the test : 2 hours By Sylvain Ravot (Caltech) Current TCP (Linux Reno)

7. As MTU increase… 1.5K, 4K, 9K … By Sylvain Ravot (Caltech) Current TCP (Linux Reno)

8. Better? ???? By Some Dreamers (Somewhere)

9. FAST Network CERN (Geneva)  SLAC (Sunnyvale), GE, Standard MTU Sunnyval -> CERN Linux kernel 2.4.18-FAST enabled RTT = 180 ms MTU = 1500 By C. Jin & D. Wei (Caltech)

10. Theoretical Background

11. Congestion control x i (t) R

12. Congestion control x i (t) Example congestion measure p l (t) –Loss (Reno) –Queueing delay (Vegas) p l (t) x i (t) → p l (t) AQM: y l (t) TCP

13. TCP/AQM Congestion control is a distributed asynchronous algorithm to share bandwidth It has two components –TCP: adapts sending rate (window) to congestion –AQM: adjusts & feeds back congestion information They form a distributed feedback control system –Equilibrium & stability depends on both TCP and AQM –And on delay, capacity, routing, #connections p l (t) x i (t) TCP: Reno Vegas AQM: DropTail RED REM/PI AVQ

14. Methodology Protocol (Reno, Vegas, RED, REM/PI…) Equilibrium Performance Throughput, loss, delay Fairness Utility Dynamics Local stability Cost of stabilization

15. Goal: Fast AQM Scalable TCP Equilibrium properties –Uses end-to-end delay (and loss) as congestion measure –Achieves any desired fairness, expressed by utility function –Very high bandwidth utilization (99% in theory) Stability properties –Stability for arbitrary delay, capacity, routing & load –Good performance Negligible queueing delay & loss introduced by the protocol Fast response

16. Implementation and Experiment

17. Implementation First Version (demonstrated in SuperComputing Conf, Nov 2002): Sender-side kernel modification (Good for File sharing service) Challenges: –Effects ignored in theory –Large window size and high speed

18. SCinet Caltech-SLAC experiments netlab.caltech.edu/FAST SC2002 Baltimore, Nov 2002 Network Topology SunnyvaleBaltimore Chicago Geneva 3000km 1000km 7000km C. Jin, D. Wei, S. Low FAST Team and Partners FAST TCP  Standard MTU  Peak window = 14,255 pkts  Throughput averaged over > 1hr 925 Mbps single flow/GE card 9.28 petabit-meter/sec 1.89 times LSR 8.6 Gbps with 10 flows 34.0 petabit-meter/sec 6.32 times LSR 21TB in 6 hours with 10 flows Highlights 1 2 1 2 7 9 10 Geneva-Sunnyvale Baltimore-Sunnyvale FAST I2 LSR #flows

19. FAST BMPS flowsBmps Peta Thruput Mbps Distance km Delay ms MTU B Duration s Transfer GB Path Alaska- Amsterdam 9.4.2002 14.9240112,272--130.625Fairbanks, AL – Amsterdam, NL MS-ISI 29.3.2000 25.389575,626-4,470828.4 MS, WA – ISI, Va Caltech-SLAC 19.11.2002 19.2892510,0371801,5003,600387CERN - Sunnyvale Caltech-SLAC 19.11.2002 218.031,79710,0371801,5003,600753CERN - Sunnyvale Caltech-SLAC 18.11.2002 724.176,1233,948851,50021,60015,396Baltimore - Sunnyvale Caltech-SLAC 19.11.2002 931.357,9403,948851,5004,0303,725Baltimore - Sunnyvale Caltech-SLAC 20.11.2002 1033.998,6093,948851,50021,60021,647Baltimore - Sunnyvale Mbps = 10 6 b/s; GB = 2 30 bytes C. Jin, D. Wei, S. Low FAST Team and Partners

20. FAST BMPS flowsBmps Peta Thruput Mbps Distance km Delay ms MTU B Duration s Transfer GB Path Alaska- Amsterdam 9.4.2002 14.9240112,272--130.625Fairbanks, AL – Amsterdam, NL MS-ISI 29.3.2000 25.389575,626-4,470828.4 MS, WA – ISI, Va Caltech-SLAC 19.11.2002 19.2892510,0371801,5003,600387CERN - Sunnyvale Caltech-SLAC 19.11.2002 218.031,79710,0371801,5003,600753CERN - Sunnyvale Mbps = 10 6 b/s; GB = 2 30 bytes C. Jin, D. Wei, S. Low FAST Team and Partners

21. FAST BMPS flowsBmps Peta Thruput Mbps Distance km Delay ms MTU B Duration s Transfer GB Path Alaska- Amsterdam 9.4.2002 14.9240112,272--130.625Fairbanks, AL – Amsterdam, NL MS-ISI 29.3.2000 25.389575,626-4,470828.4 MS, WA – ISI, Va Caltech-SLAC 19.11.2002 19.2892510,0371801,5003,600387CERN - Sunnyvale Caltech-SLAC 19.11.2002 218.031,79710,0371801,5003,600753CERN - Sunnyvale Mbps = 10 6 b/s; GB = 2 30 bytes C. Jin, D. Wei, S. Low FAST Team and Partners

22. FAST BMPS flow s Bmps Peta Thruput Mbps Distance km Delay ms MTU B Duration s Transfer GB Path Alaska- Amsterdam 9.4.2002 14.9240112,272--130.625Fairbanks, AL – Amsterdam, NL MS-ISI 29.3.2000 25.389575,626-4,470828.4 MS, WA – ISI, Va Caltech-SLAC 19.11.2002 19.2892510,0371801,5003,600387CERN - Sunnyvale Caltech-SLAC 19.11.2002 218.031,79710,0371801,5003,600753CERN - Sunnyvale Caltech-SLAC 18.11.2002 724.176,1233,948851,50021,60015,396Baltimore - Sunnyvale Caltech-SLAC 19.11.2002 931.357,9403,948851,5004,0303,725Baltimore - Sunnyvale Caltech-SLAC 20.11.2002 1033.998,6093,948851,50021,60021,647Baltimore - Sunnyvale Mbps = 10 6 b/s; GB = 2 30 bytes C. Jin, D. Wei, S. Low FAST Team and Partners

23. FAST Aggregate throughput 1 flow 2 flows 7 flows 9 flows 10 flows Average utilization 95% 92% 90% 88% FAST Standard MTU Utilization averaged over > 1hr 1hr 6hr1.1hr 6hr C. Jin, D. Wei, S. Low

24. FAST vs Linux TCP (2.4.18-3) Linux TCP Linux TCP FAST Average utilization 19% 27% 92% FAST Standard MTU Utilization averaged over 1hr txq=100txq=10000 95% 16% 48% Linux TCP Linux TCP FAST 2G 1G C. Jin (Caltech)

25. Trial Deployment FAST Kernel Installed: SLAC: Les Cottrell, etc. www-iepm.slac.stanford.edu/monitoring/bulk/fast FermiLab: Michael Ernst, etc. Coming soon: 10-Gbps NIC Testing (Sunnyval - CERN) Internet2 …

26. Detailed Information: Home Page: http://Netlab.caltech.edu/FASThttp://Netlab.caltech.edu/FAST Theory: http://netlab.caltech.edu/FAST/overview.html http://netlab.caltech.edu/FAST/overview.html Implementation & Testing: http://netlab.caltech.edu/FAST/software.html http://netlab.caltech.edu/FAST/software.html Publications: http://netlab.caltech.edu/FAST/publications.html http://netlab.caltech.edu/FAST/publications.html

27. FAST netlab.caltech.edu/FAST Theory D. Choe (Postech/Caltech), J. Doyle, S. Low, F. Paganini (UCLA), J. Wang, Z. Wang (UCLA) Prototype C. Jin, D. Wei Experiment/facilities –Caltech: J. Bunn, C. Chapman, C. Hu (Williams/Caltech), H. Newman, J. Pool, S. Ravot (Caltech/CERN), S. Singh –CERN: O. Martin, P. Moroni –Cisco: B. Aiken, V. Doraiswami, R. Sepulveda, M. Turzanski, D. Walsten, S. Yip –DataTAG: E. Martelli, J. P. Martin-Flatin –Internet2: G. Almes, S. Corbato –Level(3): P. Fernes, R. Struble –SCinet: G. Goddard, J. Patton –SLAC: G. Buhrmaster, R. Les Cottrell, C. Logg, I. Mei, W. Matthews, R. Mount, J. Navratil, J. Williams –StarLight: T. deFanti, L. Winkler –TeraGrid: L. Winkler Major sponsors ARO, CACR, Cisco, DataTAG, DoE, Lee Center, NSF Acknowledgments

28. Thanks Questions?