Buffer requirements for TCP: queueing theory & synchronization analysis Gaurav RainaDamon Wischik CambridgeUCL

TCP if (seqno > _last_acked) { if (!_in_fast_recovery) { _last_acked = seqno; _dupacks = 0; inflate_window(); send_packets(now); _last_sent_time = now; return; } if (seqno < _recover) { uint32_t new_data = seqno - _last_acked; _last_acked = seqno; if (new_data < _cwnd) _cwnd -= new_data; else _cwnd=0; _cwnd += _mss; retransmit_packet(now); send_packets(now); return; } uint32_t flightsize = _highest_sent - seqno; _cwnd = min(_ssthresh, flightsize + _mss); _last_acked = seqno; _dupacks = 0; _in_fast_recovery = false; send_packets(now); return; } if (_in_fast_recovery) { _cwnd += _mss; send_packets(now); return; } _dupacks++; if (_dupacks!=3) { send_packets(now); return; } _ssthresh = max(_cwnd/2, (uint32_t)(2 * _mss)); retransmit_packet(now); _cwnd = _ssthresh + 3 * _mss; _in_fast_recovery = true; _recover = _highest_sent; } time [0-8 sec] traffic rate [0-100 kB/sec]

Motivation: buffer size Internet routers have buffers, to accomodate bursts in traffic. How big do the buffers need to be? –3 GByte? Rule of thumb is C*RTT—what Cisco does today –300 MByte? C*RTT/√N [Appenzeller, Keslassy, McKeown, 2004 ] –30 kByte? constant buffer size, independent of line rate Large buffers are unsustainable: –Data volumes double every 10 months –CPU speeds double every 18 months –Memory access speeds double every 10 years

Buffers & synchronization Desynchronized TCP flows: –aggregate traffic is smooth –small buffers are OK Synchronized TCP flows: –aggregate traffic is bursty –large buffers are needed + + = + + = aggregate traffic rate time individual flow rates

Network traffic model When there are many TCP flows, the average traffic rate x t varies smoothly, according to a differential equation [Misra, Gong, Towsley, 2000] –involving average round trip time RTT –and packet loss probability p t time aggregate traffic rate desynchronizedsynchronized

Network traffic model When there are many TCP flows, the average traffic rate x t varies smoothly, according to a differential equation [Misra, Gong, Towsley, 2000] –involving average round trip time RTT –and packet loss probability p t The packet loss probability p t satisfies a differential equation, which depends on buffer size: [Raina, W., 2005] –either for small buffers (max queueing delay « RTT) –or for large buffers (max queueing delay  RTT) time queue size

Analysis Write down differential equations –for average TCP traffic rate x t –for queue dynamics and loss prob p t taking account of buffer size Calculate –average link utilization –average queue occupancy/delay –extent of synchronization and consequent loss of utilization, and jitter

Small buffers [20-50 pkts] Large buffers [>50ms queue delay] low- bandwidth flows [window <5pkts] slightly synchronized; high utilization; low delay/jitter. desynchronized; high utilization; high delay, low jitter. high- bandwidth flows [window >10pkts] desynchronized; moderate utilization; low delay/jitter. severely synchronized; high utilization; high delay/jitter. Results There is a virtuous circle: desynchronization permits small buffers; small buffers induce desynchronization.

Illustration: 20 flows Standard TCP, single bottleneck link, no AQM service C=1.2 kpkt/sec, RTT=200 ms, #flows N=20 B=20 pkts (small buffer) B=54 pkts (intermediate buffer) B=240 pkts (large buffer)

Illustration: 200 flows Standard TCP, single bottleneck link, no AQM service C=12 kpkt/sec, RTT=200 ms, #flows N=200 B=20 pkts (small buffer) B=170 pkts (intermediate buffer) B=2,400 pkts (large buffer)

Illustration: 2000 flows Standard TCP, single bottleneck link, no AQM service C=120 kpkt/sec, RTT=200 ms, #flows N=2000 B=20 pkts (small buffer) B=537 pkts (intermediate buffer) B=24,000 pkts (large buffer)

Limitations/concerns Surely bottlenecks are at the access network, not the core network? –Unwise to rely on this! –The small-buffer theory works fine for as few as 20 flows Perhaps TCP is constrained by receiver window, not by congestion window? –Unwise to rely on this! Probably there are congestion-constrained flows which cause congestion –Would small buffers hurt flows that are constrained by receiver window? Not if they respond appropriately.

Further work These results have been partially validated (Internet2 & Level3) Full validation needs –goodly amount of traffic –full measurement kit –ability to control buffer size Alternatives to TCP –should pace packet injections –can achieve desynchronization at all window sizes

Small buffers As line rate increases –queue size fluctuates more and more rapidly so p t depends only on the instantaneous utilization at time t can model packet arrivals by a Poisson process of rate x t [Cao+Ramanan 2002] –queue size distribution depends only on link utilization, not on line rate queue size [0-15 pkt] time [0-5 sec] queueing delay 19 ms queueing delay 1.9 ms queueing delay 0.19 ms

Large buffers (queueing delay 200 ms) When Nx t <C the queue size is small (C=line rate) No packet drops, so TCP increases x t queue size [0-160 pkt] time [0-10 sec]

Large buffers (queueing delay 200 ms) When Nx t <C the queue size is small (C=line rate) No packet drops, so TCP increases x t When Nx t >C the queue fills up and packets begin to get dropped queue size [0-160 pkt] time [0-10 sec]

Large buffers (queueing delay 200 ms) When Nx t <C the queue size is small (C=line rate) No packet drops, so TCP increases x t When Nx t >C the queue fills up and packets begin to get dropped TCP may ‘overshoot’, leading to synchronization queue size [0-160 pkt] time [0-10 sec]

Large buffers (queueing delay 200 ms) Drop probability depends on both average traffic rate x t and queue size q t queue size [0-160 pkt] time [0-10 sec]

System summary x t = average traffic rate at time t p t = packet loss probability at time t C = line rate B = buffer size RTT = round trip time N=# flows TCP traffic model Small buffer queueing model (though this is sensitive to traffic statistics) Large buffer queueing model

Stability/instability analysis For some values of RTT*C/N, the dynamical system is stable For others it is unstable and there are oscillations (i.e. the flows are partially synchronized) When it is unstable, we can calculate the amplitude of the oscillations time traffic rate Nx t /C

Instability plot traffic intensity Nx/C log 10 of pkt loss probability p TCP throughput equation queue equation extent of oscillations in Nx/C

Instability plot RTT*C/N=4 pkts RTT*C/N=20 pkts RTT*C/N=100 pkts traffic intensity Nx/C log 10 of pkt loss probability p

Alternative buffer-sizing rules Intermediate buffers buffer = bandwidth*delay / sqrt(#flows) or Large buffers, no AQM buffer = bandwidth*delay Large buffers with RED buffer=bandwidth*delay*{¼,1,4} Small buffers, no AQM buffer={10,20,50} pkts Small buffers, no AQM, ScalableTCP buffer={50,1000} pkts