1. FAST TCP Steven Low CS/EE netlab.CALTECH.edu Oct 2003

2. Congestion Control & Routing Steven Low netlab.CALTECH.edu Nov 2002

3. Can TCP/IP Maximize Utility Jiantao Wang Lun Li Steven Low John Doyle netlab.CALTECH.edu Nov 2002

4. FAST Protocols for Ultrascale Networks netlab.caltech.edu/FAST Internet: distributed feedback control system TCP: adapts sending rate to congestion AQM: feeds back congestion information R f (s) R b (s) xy pq TCPAQM Theory Calren2/Abilene Chicago Amsterdam CERN Geneva SURFNet StarLight WAN in Lab Caltech research & production networks Multi-Gbps 50-200ms delay Experiment Students Choe (Postech/CIT) Hu (Williams) J. Wang (CDS) Z.Wang (UCLA) Wei (CS) Industry Doraiswami (Cisco) Yip (Cisco) Faculty Doyle (CDS,EE,BE) Low (CS,EE) Newman (Physics) Paganini (UCLA) Staff/Postdoc Bunn (CACR) Jin (CS) Ravot (Physics) Singh (CACR) Partners CERN, Internet2, CENIC, StarLight/UI, SLAC, AMPATH, Cisco People 155Mb/s slow start equilibrium FAST recovery FAST retransmit time out 10Gb/s Implementation

5. netlab.caltech.edu Outline Motivation Network model FAST TCP Equilibrium Stability Experiments TCP/IP Applications TCP/AQM IP Transmission WWW, Email, Napster, FTP, … Ethernet, ATM, POS, WDM, …

6. netlab.caltech.edu High Energy Physics Large global collaborations 2000 physicists from 150 institutions in >30 countries 300-400 physicists in US from >30 universities & labs SLAC has 500TB data by 4/2002, worlds largest database Typical file transfer ~1 TB At 622Mbps: ~ 4 hrs At 2.5Gbps: ~ 1 hr At 10Gbps: ~15min Gigantic elephants! LHC (Large Hadron Collider) at CERN, to open 2007 Generate data at PB (10 15 B)/sec Filtered in realtime by a factor of 10 6 to 10 7 Data stored at CERN at 100MB/sec Many PB of data per year To rise to Exabytes (10 18 B) in a decade

7. netlab.caltech.edu HEP high speed network … that must change

8. netlab.caltech.edu HEP Network (DataTAG) NL SURFnet GENEVA UK SuperJANET4 ABILEN E ESNET CALRE N It GARR-B GEANT NewYork Fr Renater STAR-TAP STARLIGHT Wave Triangle 2.5 Gbps Wavelength Triangle 2002 10 Gbps Triangle in 2003 Newman (Caltech)

9. netlab.caltech.edu Performance at large windows ns-2 simulation 10Gbps capacity = 155Mbps, 622Mbps, 2.5Gbps, 5Gbps, 10Gbps; 100 ms round trip latency; 100 flows J. Wang (Caltech, June 02) 27% txq=100txq=10000 95% 1G Linux TCP Linux TCP FAST 19% average utilization capacity = 1Gbps; 180 ms round trip latency; 1 flow C. Jin, D. Wei, S. Ravot, etc (Caltech, Nov 02) DataTAG Network: CERN (Geneva) – StarLight (Chicago) – SLAC/Level3 (Sunnyvale) txq=100

10. netlab.caltech.edu Network upgrade 2001-06 01 155 02 622 03 2.5 04 5 05 10

11. netlab.caltech.edu Projected performance Ns-2: capacity = 155Mbps, 622Mbps, 2.5Gbps, 5Gbps, 10Gbps 100 sources, 100 ms round trip propagation delay 01 155 02 622 03 2.5 04 5 05 10 J. Wang (Caltech)

12. netlab.caltech.edu Outline Motivation Network model FAST TCP Equilibrium Stability Experiments TCP/IP Applications TCP/AQM IP Transmission WWW, Email, Napster, FTP, … Ethernet, ATM, POS, WDM, …

13. netlab.caltech.edu Congestion Control ~ W packets per RTT Lost packet detected by missing ACK Congestion signal: delay and loss RTT time Source Destination 12W12W12W data ACKs 12W

14. netlab.caltech.edu Congestion control x i (t) p l (t) Example congestion measure p l (t) Loss (Reno) Queueing delay (Vegas)

15. netlab.caltech.edu TCP/AQM Congestion control is a distributed asynchronous algorithm to share bandwidth It has two components TCP: adapts sending rate (window) to congestion AQM: adjusts & feeds back congestion information They form a distributed feedback control system Equilibrium & stability depends on both TCP and AQM And on delay, capacity, routing, #connections p l (t) x i (t) TCP: Reno Vegas AQM: DropTail RED REM/PI AVQ

16. netlab.caltech.edu Network model c1c1 c2c2 Network Links l of capacities c l Sources s L(s) - links used by source s U s (x s ) - utility if source rate = x s x1x1 x2x2 x3x3

17. netlab.caltech.edu Network model F1F1 FNFN G1G1 GLGL R f (s) R b (s) TCP Network AQM x y q p

18. netlab.caltech.edu for every RTT { if W/RTT min – W/RTT < then W ++ if W/RTT min – W/RTT > then W -- } queue size Vegas model Fi:Fi: Gl:Gl: Link queueing delay E2E queueing delay

19. netlab.caltech.edu Vegas model F1F1 FNFN G1G1 GLGL R f (s) R b (s) TCP Network AQM x y q p

20. netlab.caltech.edu Outline Motivation Network model FAST TCP Equilibrium Stability Experiments TCP/IP Applications TCP/AQM IP Transmission WWW, Email, Napster, FTP, … Ethernet, ATM, POS, WDM, …

21. netlab.caltech.edu Methodology Protocol (Reno, Vegas, RED, REM/PI…) Equilibrium Performance Throughput, loss, delay Fairness Utility Dynamics Local stability Cost of stabilization

22. netlab.caltech.edu Model c1c1 c2c2 Network Links l of capacities c l Sources s L(s) - links used by source s U s (x s ) - utility if source rate = x s x1x1 x2x2 x3x3

23. netlab.caltech.edu Summary: duality model Flow control problem (Kelly, Malloo, Tan 98) TCP/AQM Maximize utility with different utility functions Primal-dual algorithm Reno, Vegas DropTail, RED, REM Result (L 00): (x*,p*) primal-dual optimal iff

24. netlab.caltech.edu Example utility functions

25. netlab.caltech.edu Game interpretation Source s : Link l :

26. netlab.caltech.edu Synchronous convergence Theorem (L & Lapsley 99) Provided R has full row rank & U s strictly concave: Gradient projection algorithm of dual problem Converges to optimal primal-dual solutions if Limit point: unique Pareto optimal Nash equilibrium

27. netlab.caltech.edu Asynchronous convergence Sources and links update & compute at different times with different frequencies using delayed info Theorem (L & Lapsley 99) Converges in asynchronous environment with smaller

28. netlab.caltech.edu Persistent congestion Vegas exploits buffer process to compute prices (queueing delays) Persistent congestion due to Coupling of buffer & price Error in propagation delay estimation Consequences Excessive backlog Unfairness to older sources Theorem (Low, Peterson, Wang 02) A relative error of i in propagation delay estimation distorts the utility function to

29. netlab.caltech.edu Equilibrium of Vegas Network Link queueing delays: p l Queue length: c l p l Sources Throughput: x i E2E queueing delay : q i Packets buffered: Utility funtion: U i (x) = i d i log x Proportional fairness

30. netlab.caltech.edu Validation (L. Wang, Princeton) Single link, capacity = 6 pkt/ms, s = 2 pkts/ms, d s = 10 ms With finite buffer: Vegas reverts to Reno Without estimation errorWith estimation error

31. netlab.caltech.edu Validation (L. Wang, Princeton) Source rates (pkts/ms) #src1 src2 src3 src4 src5 15.98 (6) 22.05 (2) 3.92 (4) 30.96 (0.94) 1.46 (1.49) 3.54 (3.57) 40.51 (0.50) 0.72 (0.73) 1.34 (1.35) 3.38 (3.39) 50.29 (0.29) 0.40 (0.40) 0.68 (0.67) 1.30 (1.30) 3.28 (3.34) #queue (pkts)baseRTT (ms) 1 19.8 (20) 10.18 (10.18) 2 59.0 (60)13.36 (13.51) 3127.3 (127)20.17 (20.28) 4237.5 (238)31.50 (31.50) 5416.3 (416)49.86 (49.80)

32. netlab.caltech.edu Methodology Protocol (Reno, Vegas, RED, REM/PI…) Equilibrium Performance Throughput, loss, delay Fairness Utility Dynamics Local stability Cost of stabilization

33. netlab.caltech.edu Theorem (Low et al, Infocom02) Reno/RED is locally stable if Stability: Reno/RED F1F1 FNFN G1G1 GLGL R f (s) R b (s) TCP Network AQM x y q p TCP: Small Small c Large N RED: Small Large delay

34. netlab.caltech.edu Stability: scalable control F1F1 FNFN G1G1 GLGL R f (s) R b (s) TCP Network AQM x y q p Theorem (Paganini, Doyle, L, CDC01) Provided R is full rank, feedback loop is locally stable for arbitrary delay, capacity, load and topology

35. netlab.caltech.edu Stability: Stabilized Vegas F1F1 FNFN G1G1 GLGL R f (s) R b (s) TCP Network AQM x y q p Theorem (Choe & L, Infocom03) Provided R is full rank, feedback loop is locally stable if

36. netlab.caltech.edu Stability: Stabilized Vegas F1F1 FNFN G1G1 GLGL R f (s) R b (s) TCP Network AQM x y q p Theorem (Choe & L, Infocom03) Provided R is full rank, feedback loop is locally stable if

37. netlab.caltech.edu Stability: FAST F1F1 FNFN G1G1 GLGL R f (s) R b (s) TCP Network AQM x y q p Application Stabilized TCP with current routers Queueing delay as congestion measure has right scaling Incremental deployment with ECN

38. netlab.caltech.edu Outline Motivation Network model FAST TCP Equilibrium Stability Experiments TCP/IP Applications TCP/AQM IP Transmission WWW, Email, Napster, FTP, … Ethernet, ATM, POS, WDM, …

39. netlab.caltech.edu Window control algorithm Theorem (Jin, Wei, L 03) In absence of delay Mapping from w(t) to w(t+1) is contraction Global exponential convergence Full utilization after finite time Utility function: i log x i (proportional fairness)

40. netlab.caltech.edu Network (Sylvain Ravot, caltech/CERN)

41. netlab.caltech.edu FAST BMPS Internet2 Land Speed Record FAST 1 2 1 2 7 9 10 Geneva-Sunnyvale Baltimore-Sunnyvale #flows FAST Standard MTU Throughput averaged over > 1hr

42. netlab.caltech.edu FAST BMPS flowsBmps Peta Thruput Mbps Distance km Delay ms MTU B Duration s Transfer GB Path Alaska- Amsterdam 9.4.2002 14.9240112,272--130.625Fairbanks, AL – Amsterdam, NL MS-ISI 29.3.2000 25.389575,626-4,470828.4MS, WA – ISI, Va Caltech-SLAC 19.11.2002 19.2892510,0371801,5003,600387CERN - Sunnyvale Caltech-SLAC 19.11.2002 218.031,79710,0371801,5003,600753CERN - Sunnyvale Caltech-SLAC 18.11.2002 724.176,1233,948851,50021,60015,396Baltimore - Sunnyvale Caltech-SLAC 19.11.2002 931.357,9403,948851,5004,0303,725Baltimore - Sunnyvale Caltech-SLAC 20.11.2002 1033.998,6093,948851,50021,60021,647Baltimore - Sunnyvale Mbps = 10 6 b/s; GB = 2 30 bytes

43. netlab.caltech.edu Aggregate throughput 1 flow 2 flows 7 flows 9 flows 10 flows Average utilization 95% 92% 90% 88% FAST Standard MTU Utilization averaged over > 1hr 1hr 6hr 1.1hr6hr

44. netlab.caltech.edu FAST vs Linux TCP flowsBmps Peta Thruput Mbps Distance km Delay ms MTU B Duration s Transfer GB Path Linux TCP txqueulen=100 11.8618510,0371801,500360078CERN - Sunnyvale Linux TCP txqueulen=10000 12.6726610,0371801,5003600111CERN - Sunnyvale FAST 19.11.2002 19.2892510,0371801,5003600387CERN - Sunnyvale Linux TCP txqueulen=100 23.1831710,0371801,5003600133CERN - Sunnyvale Linux TCP txqueulen=10000 29.3593110,0371801,5003600390CERN - Sunnyvale FAST 19.11.2002 218.031,79710,0371801,5003600753CERN - Sunnyvale Mbps = 10 6 b/s; GB = 2 30 bytes; Delay = propagation delay Linux TCP expts: Jan 28-29, 2003

45. netlab.caltech.edu Aggregate throughput Linux TCP Linux TCP FAST Average utilization 19% 27% 92% FAST Standard MTU Utilization averaged over 1hr txq=100txq=10000 95% 16% 48% Linux TCP Linux TCP FAST 2G 1G

46. SCinet Caltech-SLAC experiments netlab.caltech.edu/FAST SC2002 Baltimore, Nov 2002 Acknowledgments Prototype C. Jin, D. Wei Theory D. Choe (Postech/Caltech), J. Doyle, S. Low, F. Paganini (UCLA), J. Wang, Z. Wang (UCLA) Experiment/facilities Caltech: J. Bunn, C. Chapman, C. Hu (Williams/Caltech), H. Newman, J. Pool, S. Ravot (Caltech/CERN), S. Singh CERN: O. Martin, P. Moroni Cisco: B. Aiken, V. Doraiswami, R. Sepulveda, M. Turzanski, D. Walsten, S. Yip DataTAG: E. Martelli, J. P. Martin-Flatin Internet2: G. Almes, S. Corbato Level(3): P. Fernes, R. Struble SCinet: G. Goddard, J. Patton SLAC: G. Buhrmaster, R. Les Cottrell, C. Logg, I. Mei, W. Matthews, R. Mount, J. Navratil, J. Williams StarLight: T. deFanti, L. Winkler Major sponsors ARO, CACR, Cisco, DataTAG, DoE, Lee Center, NSF

47. netlab.caltech.edu New Internet tech 153,000 times faster than modem March 18, 2003 Software breaks data-transfer record March 27, 2003 Connections that leaves broadband in the dust April 7, 2003 Press on FAST FAST protocol supercharges networks March 27, 2003 (UK) >50 articles, 10 countries Pushing the speed limit (Space.com) April 9, 2003 Technology Quarterly June 21, 2003 Goodbye net gridlock June, 2003 My Point of View June, 2003 June 5, 2003c New system can speed up web downloads June 5, 2003c Promise of ultra-fast downloads June 5, 2003c June 5, 2003

48. netlab.caltech.edu Dynamic sharing: 3 flows FASTLinux Dynamic sharing on Dummynet capacity = 800Mbps delay=120ms 3 flows iperf throughput Linux 2.4.x (HSTCP: UCL)

49. netlab.caltech.edu Dynamic sharing: 3 flows FASTLinux HSTCPSTCP Steady throughput

50. netlab.caltech.edu FASTLinux throughput loss queue STCPHSTCP Dynamic sharing on Dummynet capacity = 800Mbps delay=120ms 14 flows iperf throughput Linux 2.4.x (HSTCP: UCL) 30min

51. netlab.caltech.edu FASTLinux throughput loss queue STCPHSTCP 30min Room for mice ! HSTCP

52. netlab.caltech.edu Outline Motivation Network model FAST TCP Equilibrium Stability Experiments TCP/IP Applications TCP/AQM IP Transmission WWW, Email, Napster, FTP, … Ethernet, ATM, POS, WDM, …

53. netlab.caltech.edu Protocol Decomposition Applications TCP/AQM IP Transmission WWW, Email, Napster, FTP, … Ethernet, ATM, POS, WDM, … Power control Maximize channel capacity Shortest-path routing Minimize path costs Duality model (Kelly, Low et al) Maximize aggregate utility HOT (Doyle et al) Minimize user response time Heavy-tailed file sizes

54. netlab.caltech.edu Network model F1F1 FNFN G1G1 GLGL R R T TCP Network AQM x y q p Reno, Vegas DT, RED, … IP routing

55. netlab.caltech.edu Motivation

56. netlab.caltech.edu Motivation Can TCP/IP maximize utility? Shortest path routing!

57. netlab.caltech.edu TCP-AQM/IP Theorem (Wang, et al 03) Primal problem is NP-hard Proof Reduce integer partition to primal problem Given: integers {c 1, …, c n } Find: set A s.t.

58. netlab.caltech.edu TCP-AQM/IP Theorem (Wang, et al 03) Primal problem is NP-hard Achievable utility of TCP/IP? Stability? Duality gap? Conclusion: Inevitable tradeoff between achievable utility routing stability

59. netlab.caltech.edu Ring network destination r Single destination Instant convergence of TCP/IP Shortest path routing Link cost = p l (t) + d l pricestatic TCP/AQM IP r(0) p l (0) r(1) p l (1) … r(t), r(t+1), … routing

60. netlab.caltech.edu Ring network destination r TCP/AQM IP r(0) p l (0) r(1) p l (1) … r(t), r(t+1), … Stability: r ? Utility: V ? r* : optimal routing V* : max utility

61. netlab.caltech.edu Ring network destination r Theorem (Infocom 2003) No duality gap Unstable if = 0 starting from any r(0), subsequent r(t) oscillates between 0 and 1 link cost = p l (t) + d l Stability: r ? Utility: V ?

62. netlab.caltech.edu Ring network destination r link cost = p l (t) + d l Theorem (Infocom 2003) Solve primal problem asymptotically as Stability: r ? Utility: V ?

63. netlab.caltech.edu Ring network destination r link cost = p l (t) + d l Theorem (Infocom 2003) large: globally unstable small: globally stable medium: depends on r(0) Stability: r ? Utility: V ?

64. netlab.caltech.edu General network Conclusion: Inevitable tradeoff between achievable utility routing stability random graph 20 nodes, 200 links Achievable utility

65. netlab.caltech.edu FAST TCP: motivation, architecture, algorithms, performance. submitted for publication, July 1, 2003 -release: August 2003 Inquiry: fast-support@cs.caltech.edufast-support@cs.caltech.edu FAST Project Review Caltech, Oct 27-28, 2003 netlab.caltech.edu/FAST