1. Utility, Fairness, TCP/IP Steven Low CS/EE netlab.CALTECH.edu Feb 2004

2. Acknowledgments  Caltech Bunn, Choe, Doyle, Jin, Newman, Ravot, Singh, J. Wang, Wei  UCLA Paganini, Z. Wang  CERN Martin  SLAC Cottrell  Internet2 Almes, Shalunov  Cisco Aiken, Doraiswami, Yip  Level(3) Fernes  LANL Wu

3. Protocol Decomposition Applications TCP/AQM IP Transmission WWW, Email, Napster, FTP, … Ethernet, ATM, POS, WDM, … Topology, power control Maximize capacity Shortest-path routing Minimize path costs Duality model (Kelly, Low et al) Maximize aggregate utility HOT (Doyle et al) Minimize user response time Heavy-tailed file sizes

4. Outline  Network model  FAST TCP Equilibrium Stability Implementation Experiments  TCP/IP interaction  Fairness-efficiency Applications TCP/AQM IP Transmission WWW, Email, Napster, FTP, … Ethernet, ATM, POS, WDM, …

5. Performance at large windows ns-2 simulation 10Gbps capacity = 155Mbps, 622Mbps, 2.5Gbps, 5Gbps, 10Gbps; 100 ms round trip latency; 100 flows J. Wang (Caltech, June 02) 27% txq=100txq=10000 95% 1G Linux TCP Linux TCP FAST 19% average utilization capacity = 1Gbps; 180 ms round trip latency; 1 flow C. Jin, D. Wei, S. Ravot, etc (Caltech, Nov 02) DataTAG Network: CERN (Geneva) – StarLight (Chicago) – SLAC/Level3 (Sunnyvale) txq=100

6. Congestion control x i (t) p l (t) Example congestion measure p l (t) Loss (Reno) Queueing delay (Vegas)

7. TCP/AQM  Congestion control is a distributed asynchronous algorithm to share bandwidth  It has two components TCP: adapts sending rate (window) to congestion AQM: adjusts & feeds back congestion information  They form a distributed feedback control system Equilibrium & stability depends on both TCP and AQM And on delay, capacity, routing, #connections p l (t) x i (t) TCP: Reno Vegas AQM: DropTail RED REM/PI AVQ

8. Network model F1F1 FNFN G1G1 GLGL R f (s) R b ’ (s) TCP Network AQM x y q p

9. Network model F1F1 FNFN G1G1 GLGL R f (s) R b ’ (s) TCP Network AQM x y q p

10. Outline  Network model  FAST TCP Equilibrium Stability Implementation Experiments  TCP/IP interaction  Fairness-efficiency Applications TCP/AQM IP Transmission WWW, Email, Napster, FTP, … Ethernet, ATM, POS, WDM, …

11. Methodology Protocol (Reno, Vegas, RED, REM/PI…) Equilibrium Performance Throughput, loss, delay Fairness Dynamics Local stability Global stability

12. Network model F1F1 FNFN G1G1 GLGL R R T TCP Network AQM x y q p Reno, Vegas DT, RED, … IP routing

13. Duality model Primal-dual algorithm:

14. Duality Model of TCP Primal-dual algorithm: Reno, VegasDropTail, RED, REM Source algorithm iterates on rates Link algorithm iterates on prices With different utility functions

15. Summary: duality model Flow control problem (Kelly, Malloo, Tan 98) TCP/AQM Maximize utility with different utility functions Primal-dual algorithm Reno, Vegas DropTail, RED, REM Result (L 00): (x*,p*) primal-dual optimal iff

16. Example utility functions FAST, STCP (Mo, Walrand 00)

17. Methodology Protocol (Reno, Vegas, RED, REM/PI…) Equilibrium Performance Throughput, loss, delay Fairness Dynamics Local stability Global stability

18. Theorem (Low et al, Infocom’02) Reno/RED is locally stable if Stability: Reno/RED F1F1 FNFN G1G1 GLGL R f (s) R b ’ (s) TCP Network AQM x y q p TCP: Small  Small c Large N RED: Small  Large delay

19. Stability: scalable control F1F1 FNFN G1G1 GLGL R f (s) R b ’ (s) TCP Network AQM x y q p Theorem (Paganini, Doyle, L, CDC’01) Provided R is full rank, feedback loop is locally stable for arbitrary delay, capacity, load and topology

20. Linear Stability: scalable control Theorem (Paganini, Doyle, Low, CDC’01) Provided R is full rank, feedback loop is locally stable for arbitrary delay, capacity, load and topology Globally stable in presence of delay?

21. Stability: Stabilized Vegas F1F1 FNFN G1G1 GLGL R f (s) R b ’ (s) TCP Network AQM x y q p Theorem (Choe & L, Infocom’03) Provided R is full rank, feedback loop is locally stable if

22. Stability: Stabilized Vegas F1F1 FNFN G1G1 GLGL R f (s) R b ’ (s) TCP Network AQM x y q p Application  Stabilized TCP with current routers  Queueing delay as congestion measure has right scaling  Incremental deployment with ECN

23. Outline  Network model  FAST TCP Equilibrium Stability Implementation Experiments  TCP/IP interaction  Fairness-efficiency Applications TCP/AQM IP Transmission WWW, Email, Napster, FTP, … Ethernet, ATM, POS, WDM, …

24. Reno TCP  Packet level Designed and implemented first  Flow level Understood afterwards  Flow level dynamics determines Equilibrium: performance, fairness Stability  Design flow level equilibrium & stability  Implement flow level goals at packet level

25. Packet level ACK: W  W + 1/W Loss: W  W – 0.5W  Reno AIMD(1, 0.5) ACK: W  W + a(w)/W Loss: W  W – b(w)W  HSTCP AIMD(a(w), b(w)) ACK: W  W + 0.01 Loss: W  W – 0.125W  STCP MIMD(a, b)  FAST

26. Flow level: Reno, HSTCP, STCP, FAST  Similar flow level equilibrium  = 1.225 (Reno), 0.120 (HSTCP), 0.075 (STCP) pkts/sec

27. Flow level: Reno, HSTCP, STCP, FAST  Different gain  and utility U i They determine equilibrium and stability  Different congestion measure p i Loss probability (Reno, HSTCP, STCP) Queueing delay (Vegas, FAST)  Common flow level dynamics window adjustment control gain flow level goal =

28. Implementation strategy  Common flow level dynamics window adjustment control gain flow level goal =  Small adjustment when close, large far away Need to estimate how far current state is wrt target Scalable  Window adjustment independent of p i Depends only on current window Difficult to scale

29. Difficulties at large window  Equilibrium problem Packet level: AI too slow, MI too drastic Flow level: required loss probability too small  Dynamic problem Packet level: must oscillate on binary signal Flow level: unstable at large window 5

30. FAST TCP Theorem (Jin, Wei, L ‘03) In absence of delay at a single link  Mapping from w(t) to w(t+1) is contraction  Global exponential convergence  Full utilization after finite time  Utility function:  i log x i (proportional fairness)

31. Outline  Network model  FAST TCP Equilibrium Stability Implementation Experiments  TCP/IP interaction  Fairness-efficiency Applications TCP/AQM IP Transmission WWW, Email, Napster, FTP, … Ethernet, ATM, POS, WDM, …

32. Network (Sylvain Ravot, caltech/CERN)

33. FAST TCP util: 95% Linux TCP util: 19% 1Gbps path; 180 ms RTT; 1 flow Jin, Wei, Ravot, etc (Caltech, Nov 02)

34. FAST BMPS Internet2 Land Speed Record FAST 1 2 1 2 7 9 10 Geneva-Sunnyvale Baltimore-Sunnyvale #flows FAST  Standard MTU  Throughput averaged over > 1hr

35. Aggregate throughput 1 flow 2 flows 7 flows 9 flows 10 flows Average utilization 95% 92% 90% 88% FAST  Standard MTU  Utilization averaged over > 1hr 1hr 6hr 1.1hr6hr

36. SCinet Caltech-SLAC experiments netlab.caltech.edu/FAST SC2002 Baltimore, Nov 2002 Acknowledgments  Prototype C. Jin, D. Wei  Theory D. Choe (Postech/Caltech), J. Doyle, S. Low, F. Paganini (UCLA), J. Wang, Z. Wang (UCLA)  Experiment/facilities Caltech: J. Bunn, C. Chapman, C. Hu (Williams/Caltech), H. Newman, J. Pool, S. Ravot (Caltech/CERN), S. Singh CERN: O. Martin, P. Moroni Cisco: B. Aiken, V. Doraiswami, R. Sepulveda, M. Turzanski, D. Walsten, S. Yip DataTAG: E. Martelli, J. P. Martin-Flatin Internet2: G. Almes, S. Corbato Level(3): P. Fernes, R. Struble SCinet: G. Goddard, J. Patton SLAC: G. Buhrmaster, R. Les Cottrell, C. Logg, I. Mei, W. Matthews, R. Mount, J. Navratil, J. Williams StarLight: T. deFanti, L. Winkler  Major sponsors ARO, CACR, Cisco, DataTAG, DoE, Lee Center, NSF

37. Dynamic sharing: 3 flows FASTLinux Dynamic sharing on Dummynet  capacity = 800Mbps  delay=120ms  3 flows  iperf throughput  Linux 2.4.x (HSTCP: UCL)

38. Dynamic sharing: 3 flows FASTLinux HSTCPSTCP Steady throughput

39. FASTLinux throughput loss queue STCPHSTCP Dynamic sharing on Dummynet  capacity = 800Mbps  delay=120ms  14 flows  iperf throughput  Linux 2.4.x (HSTCP: UCL) 30min

40. FASTLinux throughput loss queue STCPHSTCP 30min Room for mice ! HSTCP

41. Aggregate throughput ideal performance Dummynet: cap = 800Mbps; delay = 50-200ms; #flows = 1-14; 29 expts

42. Aggregate throughput small window 800pkts large window 8000 Dummynet: cap = 800Mbps; delay = 50-200ms; #flows = 1-14; 29 expts

43. Fairness Jain’s index HSTCP ~ Reno Dummynet: cap = 800Mbps; delay = 50-200ms; #flows = 1-14; 29 expts

44. Stability Dummynet: cap = 800Mbps; delay = 50-200ms; #flows = 1-14; 29 expts stable in diverse scenarios

45. Outline  Network model  FAST TCP Equilibrium Stability Implementation Experiments  TCP/IP interaction  Fairness-efficiency Applications TCP/AQM IP Transmission WWW, Email, Napster, FTP, … Ethernet, ATM, POS, WDM, …

46. Network model F1F1 FNFN G1G1 GLGL R R T TCP Network AQM x y q p Reno, Vegas DT, RED, … IP routing

47. Motivation

48. Can TCP/IP maximize utility? Shortest path routing!

49. TCP-AQM/IP Theorem (Wang, et al 03) Primal problem is NP-hard Proof Reduce integer partition to primal problem Given: integers {c 1, …, c n } Find: set A s.t.

50. TCP-AQM/IP Theorem (Wang, et al 03) Primal problem is NP-hard Achievable utility of TCP/IP? Stability? Duality gap? Conclusion: Inevitable tradeoff between achievable utility routing stability

51. Ring network destination r Single destination Instant convergence of TCP/IP Shortest path routing Link cost =  p l (t) +  d l pricestatic TCP/AQM IP r(0)  p l (0) r(1)  p l (1) … r(t), r(t+1), … routing

52. Ring network destination r TCP/AQM IP r(0)  p l (0) r(1)  p l (1) … r(t), r(t+1), … Stability: r  ? Utility: V  ? r* : optimal routing V* : max utility

53. Ring network destination r Theorem (Infocom 2003) “No” duality gap Unstable if  = 0 starting from any r(0), subsequent r(t) oscillates between 0 and 1 link cost =  p l (t) +  d l Stability: r  ? Utility: V  ?

54. Ring network destination r link cost =  p l (t) +  d l Theorem (Infocom 2003) Solve primal problem asymptotically as Stability: r  ? Utility: V  ?

55. Ring network destination r link cost =  p l (t) +  d l Theorem (Infocom 2003)  large: globally unstable  small: globally stable  medium: depends on r(0) Stability: r  ? Utility: V  ?

56. General network Conclusion: Inevitable tradeoff between achievable utility routing stability random graph 20 nodes, 200 links Achievable utility

57. Outline  Network model  FAST TCP Equilibrium Stability Implementation Experiments  TCP/IP interaction  Fairness-efficiency Applications TCP/AQM IP Transmission WWW, Email, Napster, FTP, … Ethernet, ATM, POS, WDM, …

58. TCP/AQM: duality model Flow control problem (Kelly, Malloo, Tan 98) TCP/AQM Maximize utility with different utility functions (L 00): (x*,p*) primal-dual optimal iff Primal-dual algorithm Reno, Vegas, FASTDT, RED, REM/PI, AVQ

59. Fairness Identify allocation with  An allocation is fairer if its  is larger (Mo, Walrand 00)

60. Fairness  maximum throughput  proportional fairness  min delay fairness  infinity  maxmin fairness (Mo, Walrand 00)

61. Efficiency Unique optimal rate x () An allocation is efficient if T () is large

62. Conjecture T () is nonincreasing i.e. a fair allocation is always inefficient

63. Example 1 Conjecture T () is nonincreasing i.e. a fair allocation is always inefficient 1/(L+1) L/(L+1) 1/2 maxmin proportional

64. Example 1 Conjecture T () is nonincreasing i.e. a fair allocation is always inefficient

65. Example 2 Conjecture T () is nonincreasing i.e. a fair allocation is always inefficient

66. Example 3 Conjecture T () is nonincreasing i.e. a fair allocation is always inefficient

67. Intuition “The fundamental conflict between achieving flow fairness and maximizing overall system throughput….. The basic issue is thus the trade-off between these two conflicting criteria.” Luo,etc.(2003), ACM MONET

68. Results  Theorem: Necessary & sufficient condition for general network  Corollary 1: true if N(R)=1 1/(L+1) L/(L+1) 1/2 maxmin proportional

69. Results  Theorem: Necessary & sufficient condition for general network  Corollary 1: true if N(R)=1

70. Results  Theorem: Necessary & sufficient condition for general network  Corollary 2: true if N(R)=2 2 long flows pass through same# links

71. Counter-example  There exists a network such that dT/d > 0 for almost all >0  Intuition Large  favors expensive flows Long flows may not be expensive  Maxmin may be more efficient than proportional fairness

72. Counter-example  Theorem: Given any  0 >0, there exists network where  Compact example

73. netlab.caltech.edu/FAST