FAST TCP Speaker: Ray Veune: Room 1026 Date: 25 th October, 2003 Time:10:00am

Motivation Demand for ultrascale networking Demand for ultrascale networking HENP (High Energy and Nuclear Physics) HENP (High Energy and Nuclear Physics) Data volumes of tens of Petabytes (10 15 ) to Exabytes (10 18 ) Data volumes of tens of Petabytes (10 15 ) to Exabytes (10 18 ) Require Terabit/sec (10 15 bit/sec or 1000Gbit/sec) Require Terabit/sec (10 15 bit/sec or 1000Gbit/sec) Scalability problem of TCP Scalability problem of TCP Losses must be extremely rare Losses must be extremely rare TCP must induce loss TCP must induce loss Underutilization and oscillation Underutilization and oscillation

Scalability problem of TCP extremely loss packet loss possibility Rate = 1.3 * MTU / (RTT * sqrt(Loss)) Rate = 1.3 * MTU / (RTT * sqrt(Loss)) MTU = 1500bytes, RTT = 10ms MTU = 1500bytes, RTT = 10ms Bandwidth Loss rate 10Mbps 2.43 x Mbps 2.43 x Gbps 2.43 x Gbps 2.43 x Tbps 2.43 x Tbps (RTT:100ms) 3.80 x

Scalability problem of TCP inevitable packet loss TCP needs to create losses TCP needs to create losses Single bit network feedback signal Single bit network feedback signal

Scalability problem of TCP Underutilization and oscillation AIMD (1, 0.5) AIMD (1, 0.5) At large window size (in excess of 10,000 pkts): At large window size (in excess of 10,000 pkts): Halving window on loss event is too drastic Halving window on loss event is too drastic Increasing window by one packet per RTT is too conservative Increasing window by one packet per RTT is too conservative

FAST TCP Achievement CERN (European Organization for Nuclear Research) sent 1.1 Terabytes of data at 5.44 Gbps CERN (European Organization for Nuclear Research) sent 1.1 Terabytes of data at 5.44 Gbps Full-length DVD film in 7 seconds !! Full-length DVD film in 7 seconds !!

FAST TCP Flow based vs Packet based Flow based vs Packet based Network delay vs Packet loss Network delay vs Packet loss TCP-Vegas vs TCP-Reno TCP-Vegas vs TCP-Reno Stabilized Vegas Stabilized Vegas

TCP Vegas Techniques New Retransmission Mechanism New Retransmission Mechanism Congestion Avoidance Mechanism Congestion Avoidance Mechanism Modified Slow-Start Mechanism Modified Slow-Start Mechanism

TCP Vegas New Retransmission Mechanism Timeout Timeout n duplicate ACKs n duplicate ACKs

TCP Vegas New Retransmission Mechanism Check time record for the first duplicate packet Check time record for the first duplicate packet non-duplicate ACKs first or second after retransmission non-duplicate ACKs first or second after retransmission Catch other segment lost previous to retransmission Catch other segment lost previous to retransmission

TCP Vegas Congestion Avoidance Mechanism Detect network delay by monitoring RTT Detect network delay by monitoring RTT BaseRTT and ActualRTT BaseRTT and ActualRTT SourceDest.Router

Expected = WindowSize / BaseRTT Expected = WindowSize / BaseRTT Diff = Expected - Actual Diff = Expected - Actual Diff >> 0, decrease sending rate Diff >> 0, decrease sending rate Diff = 0, increase sending rate Diff = 0, increase sending rate α< Diff < β α< Diff < β TCP Vegas Congestion Avoidance Mechanism αβ Diff

Extra buffers occupied Extra buffers occupied BaseRTT: 100ms, segment size: 1KB BaseRTT: 100ms, segment size: 1KB Expected = WindowSize / BaseRTT α = 30KB/s, β=60KB/s α=> 30KBps * 100ms / 1KB = 3 α=> 30KBps * 100ms / 1KB = 3 β=> 60KBps * 100ms / 1KB = 6 β=> 60KBps * 100ms / 1KB = 6

TCP Vegas Congestion Avoidance Mechanism α=β α=β Diff <α Diff <α increase one segment per RTT increase one segment per RTT Diff =α Diff =α no change in windows size no change in windows size Diff >α Diff >α decrease one segment per RTT decrease one segment per RTT

TCP Vegas Slow-Start Mechanism TCP-reno TCP-reno Send two segment for each ACK received Send two segment for each ACK received Exponential growth every RTT Exponential growth every RTT TCP-Vegas TCP-Vegas Exponential growth every alternative RTT Exponential growth every alternative RTT γthreshold γthreshold Diff >γ Diff >γ Changes from slow-start mode to linear I/D mode Changes from slow-start mode to linear I/D mode

Stability of TCP Vegas Network model Set of L links with finite capacities c Set of L links with finite capacities c c = (c l, l  L) c = (c l, l  L) N sources indexed by r N sources indexed by r Each source r uses a set of link defined by the L  N routing matrix Each source r uses a set of link defined by the L  N routing matrix R lr = { 1if source r uses link l 0otherwise

Stability of TCP Vegas Network model For each link l, the congestion measure p l (t) is call price For each link l, the congestion measure p l (t) is call price For each source r, it maintains a rate x r (t) in packets/sec For each source r, it maintains a rate x r (t) in packets/sec Equilibrium forward delay from source r to link l :  Equilibrium forward delay from source r to link l :  Equilibrium backward delay from link l to source r :  Equilibrium backward delay from link l to source r :  lr

Stability of TCP Vegas Network model Aggregate price source r observes in its path Aggregate price source r observes in its path q r (t) =  R lr p l (t -  ) q r (t) =  R lr p l (t -  ) Aggregate source rate link l observes Aggregate source rate link l observes y l (t) =  R lr x r (t -  ) y l (t) =  R lr x r (t -  ) x1(t) x2(t) p1(t) p2(t) p3(t)p4(t) l lr r

Stability of TCP Vegas Network model Tr denote equilibrium RTT Tr denote equilibrium RTT  +  = Tr,  l  L Dynamical system of TCP Vegas Dynamical system of TCP Vegas  p l (t) =( y l ( t ) – c l ) / c l if p l (t) > 0 ( y l ( t ) – c l ) / c l ) + if p l (t) = 0  x r (t) = 1/Tr 2 (t) sgn( 1 – x r (t)q r (t) /  r d r ) Tr (t) = d r + q r ( t ) Wheresgn(z) = 1 if z > 0, -1 if z 0, -1 if z < 0 and 0 if z = 0 (z) + = max { 0, z } lr

Stability of TCP Vegas Approximate model  x r (t) = 1/Tr 2 (t) sgn( 1 – x r (t)q r (t) /  r d r )  x r (t) = 1/Tr 2 (t) sgn( 1 – x r (t)q r (t) /  r d r ) sgn(z)  2/  tan -1 (  z) sgn(z)  2/  tan -1 (  z)        x r (t) = (2/  Tr 2 (t))tan -1  (1 – x r (t)q r (t) /  r d r )  x r (t) = (2/  Tr 2 (t))tan -1  (1 – x r (t)q r (t) /  r d r )

Stability of TCP Vegas Approximate model In equilibrium, the source rate x r * and aggregate price q r * satisfy In equilibrium, the source rate x r * and aggregate price q r * satisfy x r * q r * =  r d r

Stability of TCP Vegas Theorem 1 Suppose for all r, k 0 Tr  max r Tr for some k 0. Suppose for all r, k 0 Tr  max r Tr for some k 0. Let M be an upper bound on the number of links in the path of any source, M  max r  l R lr. Let M be an upper bound on the number of links in the path of any source, M  max r  l R lr. The Vegas model is locally asymptotically stable around the equilibrium point (x r *, y l *, p l *, q r * ) if The Vegas model is locally asymptotically stable around the equilibrium point (x r *, y l *, p l *, q r * ) if max r x r * Tr sinc  (ň / x r * Tr ) <  / Mk 0 2 ň = 2  /  ň = 2  /  Let  (a) be the unique solution in ( 0,  /2) of  tan  = a as a strictly increasing function of a Let  (a) be the unique solution in ( 0,  /2) of  tan  = a as a strictly increasing function of a sinc  = sin  /  sinc  = sin  / 

Stability of TCP Vegas Theorem 1 max r x r * Tr sinc  (ň / x r * Tr ) <  / Mk 0 2 max r x r * Tr sinc  (ň / x r * Tr ) <  / Mk 0 2  () is strictly increasing  () is strictly increasing sinc() is strictly decreasing sinc() is strictly decreasing LHS is strictly increasing in windows size x r * Tr LHS is strictly increasing in windows size x r * Tr Theorem 1: Stability condition impose a limit on max windows size Theorem 1: Stability condition impose a limit on max windows size

Stability of TCP Vegas Corollary 2 max r x r * Tr sinc  (ň / x r * Tr ) <  / Mk 0 2 max r x r * Tr sinc  (ň / x r * Tr ) <  / Mk 0 2 All source has the same target queue length,  r d r =  for all r All source has the same target queue length,  r d r =  for all r Corollary 2: LHS is strictly increasing in q r * / Tr, implying a lower bound on queueing delay Corollary 2: LHS is strictly increasing in q r * / Tr, implying a lower bound on queueing delay qr* /Tr sinc  ( ) Tr qr* ň . min r > Mk 0 2

Stability of TCP Vegas Corollary 3 Since  () 2 / , k 0  1 Since  () 2 / , k 0  1 Corollary 3: min r q r * / Tr > 2M /  Corollary 3: min r q r * / Tr > 2M /  If M  2, then RHS bigger than 1, since Tr = d r + q r * If M  2, then RHS bigger than 1, since Tr = d r + q r * M = 1 M = 1 The stability condition cannot be satisfied if a source has more than one link The stability condition cannot be satisfied if a source has more than one link Sufficient in multilink case Sufficient in multilink case Necessary and sufficient in single-link-homogeneous- source case Necessary and sufficient in single-link-homogeneous- source case

Stability of TCP Vegas Single link with homogeneous source A single link of capacity c, A single link of capacity c, Shared by N homogeneous source, Shared by N homogeneous source, with round trip propagation delay d with round trip propagation delay d

Stability of TCP Vegas Single link with homogeneous source From corollary 3: q r * / Tr > 2 /  for all r From corollary 3: q r * / Tr > 2 /  for all r Tr / q r * <  /2, since Tr = d + q r * Tr / q r * <  /2, since Tr = d + q r * d / q r * d d < (  /2 – 1) q r * Since q r * =  / x r * => (  N)/c Since q r * =  / x r * => (  N)/c cd < (  /2 – 1)  N cd < (  /2 – 1)  N Conclusion: bandwidth delay product should be small Conclusion: bandwidth delay product should be small

Stabilized Vegas  x r (t) = (2/  Tr 2 (t))tan -1  (1 – x r (t)q r (t) /  r d r )  x r (t) = (2/  Tr 2 (t))tan -1  (1 – x r (t)q r (t) /  r d r )  x r (t) = (w/Tr 2 (t))tan -1  r (t)(1 – x r (t)q r (t)/  r d r -  r (t)  q r (t))  x r (t) = (w/Tr 2 (t))tan -1  r (t)(1 – x r (t)q r (t)/  r d r -  r (t)  q r (t)) 1 – x r ( t ) q r ( t ) /  r d r 1 – x r ( t ) q r ( t ) /  r d r 1 – x r ( t ) q r ( t ) /  r d r -  r ( t )  q r ( t ) 1 – x r ( t ) q r ( t ) /  r d r -  r ( t )  q r ( t )  r ( t ) = (1 /  ) ( Tr( t ) / q r ( t ) )  r ( t ) = (1 /  ) ( Tr( t ) / q r ( t ) )  r ( t ) = (   / w ) ( x r ( t ) Tr( t ) )  r ( t ) = (   / w ) ( x r ( t ) Tr( t ) )

Stabilized Vegas The gain  r ( t ) serves as a normalization to  q r ( t ) The gain  r ( t ) serves as a normalization to  q r ( t ) Additional differential term  r ( t )  q r ( t ) anticipates the future of q r ( t ) Additional differential term  r ( t )  q r ( t ) anticipates the future of q r ( t ) Without: x r ( t ) will increase if x r (t)q r (t)<  r d r Without: x r ( t ) will increase if x r (t)q r (t)<  r d r Even x r (t)q r (t)/  r d r is small, x r ( t ) may decrease if prices are rapidly growing Even x r (t)q r (t)/  r d r is small, x r ( t ) may decrease if prices are rapidly growing

Stabilized Vegas Stability condition for stabilized Vegas Stability condition for stabilized Vegas where  = tan -1 ( (2  )/(1-  ) ) Stabilized Vegas can choose a small Stabilized Vegas can choose a small ( a>0,  (0,1) ) such that RHS can be larger for better stability of the original Vegas cd 0,  (0,1) ) such that RHS can be larger for better stability of the original Vegas cd < (  /2 – 1)  N   2 +  2 a 2  2 + a 2 cd < ( - 1 )  N

Simulation Results Reno/RenoReno/VegasRegas/RenoVegas/Vegas Throughput (KB/s) 60/10961/12366/11974/131 Throughput Ratio 1.00/ / / /1.20 Retranmissions (KB) 30/2243/1.81.5/1.80.3/0.1 Retransmit Ratios 1.00/ / / /0.01 One-on-One (300KB and 1MB) Transfers c = 200KB/s 50ms delay

Simulation Results  = 20  = 20 N = 100 flows N = 100 flows Fixed packet size of 1KB Fixed packet size of 1KB FIFO /w Droptail, queue capacity = FIFO /w Droptail, queue capacity = ( a,  ) = ( 0.5, ) ( a,  ) = ( 0.5, )

Simulation Results c = 100 pkts/s and d = 10ms

Simulation Results c = 1000 pkts/s and d = 10ms

Simulation Results c = 100 pkts/s and d = 100ms

Simulation Results

Experimental Results FAST TCP was first demonstrated publicly in during the SuperComputing Conference (SC2002) in Baltimore, MD, in November 16 – Caltech-SLAC research team CERN DataTAG StarLight TeraGrid Cisco Level(3).

Experimental Results

Experimental Results Throughput and utilization #flowThroughputMbpsutilizationDelaymsDistancekmDurations Bmps Transfer GB (266) 95% (27%) 18010,0373, (2.67) 387 (111) 2 1,797 (931) 92% (48%) 18010,0373, (9.35) 753 (390) 76,12390%853,94821, ,396 97,94090%853,9484, , ,60988%853,94821, ,647 SC2002 FAST experimental result Current TCP implementation in Linux v on Jan 27-28, 2003

FAST TCP Conclusion Problem of current TCP Problem of current TCP Equilibrium Equilibrium Dynamic problem Dynamic problem FAST TCP FAST TCP Equation-based control with queuing delay Equation-based control with queuing delay TCP Vegas TCP Vegas Stabilized Vegas Stabilized Vegas

Q&A

Thank you