1. Shivkumar Kalyanaraman Rensselaer Polytechnic Institute 1 TCP, TCP Congestion Control and Common AQM Schemes: Quick Revision Shivkumar Kalyanaraman Rensselaer Polytechnic Institute shivkuma@ecse.rpi.edu http://www.ecse.rpi.edu/Homepages/shivkuma Based in part upon slides of Prof. Raj Jain (OSU), Srini Seshan (CMU), J. Kurose (U Mass), I.Stoica (UCB)

2. Shivkumar Kalyanaraman Rensselaer Polytechnic Institute 2 q Quick introduction to TCP Services q TCP Reliability Model, Mechanisms q TCP Congestion Control Model and Mechnisms q TCP Versions: Reno, NewReno, SACK, Vegas etc q AQM schemes: common goals, RED, … Overview

3. Shivkumar Kalyanaraman Rensselaer Polytechnic Institute 3 Multiplexing / demultiplexing application transport network M P2 application transport network receiver H t H n segment M application transport network P1 MMM P3 P4 segment header application-layer data source port #dest port # 32 bits header/payload fields

4. Shivkumar Kalyanaraman Rensselaer Polytechnic Institute 4 Checksum Sender: q Treat segment contents as sequence of 16-bit integers q Checksum: addition (1’s complement sum) of segment contents q Sender puts checksum value into UDP checksum field Receiver: q Compute checksum of received segment q Check if computed checksum equals checksum field value: q NO - error detected q YES - no error detected. But maybe errors nonetheless? Goal: detect “errors” (e.g., flipped bits) in transmitted segment (I.e., payload + header) Note: IP only has a header checksum.

5. Shivkumar Kalyanaraman Rensselaer Polytechnic Institute 5 Introduction to TCP q Communication abstraction: close equivalent to UNIX file-system interface => programmer productivity! q Reliable q Ordered q Point-to-point q Byte-stream q Full duplex q Flow and congestion controlled

6. Shivkumar Kalyanaraman Rensselaer Polytechnic Institute 6 TCP Header Source portDestination port Sequence number Acknowledgement Advertised windowHdrLen Flags 0 ChecksumUrgent pointer Options (variable) Data Flags: SYN FIN RESET PUSH URG ACK

7. Shivkumar Kalyanaraman Rensselaer Polytechnic Institute 7 Principles of Reliable Data Transfer q Characteristics of unreliable channel will determine complexity of reliable data transfer protocol (rdt)

8. Shivkumar Kalyanaraman Rensselaer Polytechnic Institute 8 Temporal Redundancy Model Packets Sequence Numbers CRC or Checksum Status Reports ACKs NAKs, SACKs Bitmaps Packets FEC information Retransmissions Timeout

9. Shivkumar Kalyanaraman Rensselaer Polytechnic Institute 9 Types of errors and effects q Forward channel bit-errors (garbled packets) q Forward channel packet-errors (lost packets) q Reverse channel bit-errors (garbled status reports) q Reverse channel bit-errors (lost status reports) q Protocol-induced effects: q Duplicate packets q Duplicate status reports q Out-of-order packets q Out-of-order status reports q Out-of-range packets/status reports (in window-based transmissions)

10. Shivkumar Kalyanaraman Rensselaer Polytechnic Institute 10 Reliability Mechanisms… q Mechanisms: q Checksum: detects corruption in pkts & acks q ACK: “packet correctly received” q Duplicate ACK: “packet incorrectly received” q Sequence number: identifies packet or ack q 1-bit sequence number used both in forward & reverse channel q Timeout only at sender q Provides reliable transmission over: q An error-free channel q A forward & reverse channel with bit-errors q Detects duplicates of packets/acks q NAKs eliminated q Forward & reverse channel w/ packet-errors (loss)

11. Shivkumar Kalyanaraman Rensselaer Polytechnic Institute 11 Example: Three-Way Handshake q TCP connection-establishment: 3-way-handshake necessary and sufficient for unambiguous setup/teardown even under conditions of loss, duplication, and delay

12. Shivkumar Kalyanaraman Rensselaer Polytechnic Institute 12 Stop-and-Wait (Handshake) Efficiency Data Ack Data t frame t prop  = t prop t frame = Distance/Speed of Signal Frame size /Bit rate = Distance  Bit rate Frame size  Speed of Signal = 1 2  + 1 U= 2t prop +t frame t frame U  Light in vacuum = 300 m/  s Light in fiber = 200 m/  s Electricity = 250 m/  s No loss or bit-errors!

13. Shivkumar Kalyanaraman Rensselaer Polytechnic Institute 13 “Sliding Window” Protocols Data Ack t frame t prop U= Nt frame 2t prop +t frame = N 2  +1 1 if N>2  +1 Note: no loss or bit-errors!

14. Shivkumar Kalyanaraman Rensselaer Polytechnic Institute 14 Receiver Sender Sliding Window: Details …… Sent & AckedSent Not Acked OK to SendNot Usable …… Max acceptable Receiver window Max ACK receivedNext seqnum Received & AckedAcceptable Packet Not Usable Sender window Next expected

15. Shivkumar Kalyanaraman Rensselaer Polytechnic Institute 15 acknowledgedsentto be sentoutside window Source Port Dest. Port Sequence Number Acknowledgment HL/Flags Window D. Checksum Urgent Pointer Options.. Source Port Dest. Port Sequence Number Acknowledgment HL/Flags Window D. Checksum Urgent Pointer Options.. Packet Sent Packet Received App write Window Flow Control: Header

16. Shivkumar Kalyanaraman Rensselaer Polytechnic Institute 16 Go-Back-N Retransmission q If you hear of packet loss, retransmit the whole window! q k-bit seq # in pkt header q Allows upto N = 2 k – 1 packets in-flight, unacked q ACK(n): ACKs all pkts up to, including seq # n - “cumulative ACK” q Sender may receive duplicate ACKs q Robust to ack losses on the reverse channel q Can pinpoint the first packet lost, but cannot identify blocks of lost packets in window q One timer for oldest-in-flight pkt

17. Shivkumar Kalyanaraman Rensselaer Polytechnic Institute 17 Selective Repeat: Sender, Receiver Windows

18. Shivkumar Kalyanaraman Rensselaer Polytechnic Institute 18 Timeout and RTT Estimation q Problem: q Unlike a physical link, the RTT of a logical link can vary, quite substantially q How long should timeout be ? q Too long => underutilization q Too short => wasteful retransmissions q Solution: adaptive timeout: based on a good estimate of maximum current value of RTT

19. Shivkumar Kalyanaraman Rensselaer Polytechnic Institute 19 How to estimate max RTT? q RTT = prop + queuing delay q Queuing delay highly variable q So, different samples of RTTs will give different random values of queuing delay q Chebyshev’s Theorem: q MaxRTT = Avg RTT + k*Deviation q Error probability is less than 1/(k**2) q Result true for ANY distribution of samples q In TCP: q Timeout = AverageRTT + 4*Deviation q Rounded up to timer granularity (50-500 ms)

20. Shivkumar Kalyanaraman Rensselaer Polytechnic Institute 20 Recap: Stability of a Multiplexed System Average Input Rate > Average Output Rate => system is unstable! How to ensure stability ? 1.Reserve enough capacity so that demand is less than reserved capacity 2.Dynamically detect overload and adapt either the demand or capacity to resolve overload

21. Shivkumar Kalyanaraman Rensselaer Polytechnic Institute 21 Congestion Problem in Packet Switching A B C 10 Mbs Ethernet 1.5 Mbs 45 Mbs D E statistical multiplexing queue of packets waiting for output link  If capacity is sized to be less than peak demand (statistical muxing!), need to either reserve resources or dynamically detect/adapt to overload for stability

22. Shivkumar Kalyanaraman Rensselaer Polytechnic Institute 22 Congestion: A Close-up View q knee – point after which q throughput increases very slowly q delay increases fast q cliff – point after which q throughput starts to decrease very fast to zero (congestion collapse) q delay approaches infinity q Note (in an M/M/1 queue) q delay = 1/(1 – utilization) Load Throughput Delay kneecliff congestion collapse packet loss

23. Shivkumar Kalyanaraman Rensselaer Polytechnic Institute 23 Congestion Control vs. Congestion Avoidance q Congestion control goal q stay left of cliff q Congestion avoidance goal q stay left of knee q Right of cliff: q Congestion collapse q Increase in network load results in decrease of useful work done Load Throughput kneecliff congestion collapse

24. Shivkumar Kalyanaraman Rensselaer Polytechnic Institute 24 End-to-End Congestion Control q 1. End-to-end model: q End-system estimate the timing and degree of congestion and reduces its demand appropriately q Intermediate nodes relied upon to send timely and appropriate penalty indications (eg: packet loss rate) during congestion q Enhanced routers could send more accurate congestion signals (eg: early congestion notifications, I.e. ECNs) q Key: trust and complexity resides at end-systems q Issue: What about misbehaving flows?

25. Shivkumar Kalyanaraman Rensselaer Polytechnic Institute 25 Packet Conservation: Self-clocking PrPr PbPb ArAr AbAb Receiver Sender AsAs q Implications of ack-clocking: q More batching of acks => bursty traffic q Less batching leads to a large fraction of Internet traffic being just acks (overhead)

26. Shivkumar Kalyanaraman Rensselaer Polytechnic Institute 26 Additive Increase/Multiplicative Decrease (AIMD) Policy q Assumption: decrease policy must (at minimum) reverse the load increase over-and-above efficiency line q Implication: decrease factor should be conservatively set to account for any congestion detection lags etc x0x0 x1x1 x2x2 Efficiency Line Fairness Line User 1’s Allocation x 1 User 2’s Allocation x 2

27. Shivkumar Kalyanaraman Rensselaer Polytechnic Institute 27 TCP Congestion Control q Maintains three variables: q cwnd – congestion window q rcv_win – receiver advertised window q ssthresh – threshold size (used to update cwnd) q Rough estimate of knee point… q For sending use: win = min(rcv_win, cwnd)

28. Shivkumar Kalyanaraman Rensselaer Polytechnic Institute 28 TCP: Slow Start q Whenever starting traffic on a new connection, or whenever increasing traffic after congestion was experienced: q Set cwnd =1 q Each time a segment is acknowledged increment cwnd by one (cwnd++). q Does Slow Start increment slowly? Not really. In fact, the increase of cwnd is exponential!! q Window increases to W in RTT * log 2 (W)

29. Shivkumar Kalyanaraman Rensselaer Polytechnic Institute 29 Slow Start Example q The congestion window size grows very rapidly q TCP slows down the increase of cwnd when cwnd >= ssthresh ACK for segment 1 segment 1 cwnd = 1 cwnd = 2 segment 2 segment 3 ACK for segments 2 + 3 cwnd = 4 segment 4 segment 5 segment 6 segment 7 ACK for segments 4+5+6+7 cwnd = 8

30. Shivkumar Kalyanaraman Rensselaer Polytechnic Institute 30 Slow Start Sequence Plot Time Sequence No...... Window doubles every round

31. Shivkumar Kalyanaraman Rensselaer Polytechnic Institute 31 Congestion Avoidance q Goal: maintain operating point at the left of the cliff: q How? q additive increase: starting from the rough estimate (ssthresh), slowly increase cwnd to probe for additional available bandwidth q multiplicative decrease: cut congestion window size aggressively if a loss is detected. q If cwnd > ssthresh then each time a segment is acknowledged increment cwnd by 1/cwnd i.e. (cwnd += 1/cwnd).

32. Shivkumar Kalyanaraman Rensselaer Polytechnic Institute 32 Congestion Avoidance Sequence Plot Time Sequence No Window grows by 1 every round

33. Shivkumar Kalyanaraman Rensselaer Polytechnic Institute 33 Slow Start/Congestion Avoidance Eg. q Assume that ssthresh = 8 Roundtrip times Cwnd (in segments) ssthresh

34. Shivkumar Kalyanaraman Rensselaer Polytechnic Institute 34 Putting Everything Together: TCP Pseudo-code Initially: cwnd = 1; ssthresh = infinite; New ack received: if (cwnd < ssthresh) /* Slow Start*/ cwnd = cwnd + 1; else /* Congestion Avoidance */ cwnd = cwnd + 1/cwnd; Timeout: (loss detection) /* Multiplicative decrease */ ssthresh = win/2; cwnd = 1; while (next < unack + win) transmit next packet; where win = min(cwnd, flow_win); unacknext win seq #

35. Shivkumar Kalyanaraman Rensselaer Polytechnic Institute 35 The big picture Time cwnd Timeout Slow Start Congestion Avoidance

36. Shivkumar Kalyanaraman Rensselaer Polytechnic Institute 36 Packet Loss Detection: Timeout Avoidance q Wait for Retransmission Time Out (RTO) q What’s the problem with this? q Because RTO is a performance killer q In BSD TCP implementation, RTO is usually more than 1 second q the granularity of RTT estimate is 500 ms q retransmission timeout is at least two times of RTT q Solution: Don’t wait for RTO to expire q Use alternate mechanism for loss detection q Fall back to RTO only if these alternate mechanisms fail.

37. Shivkumar Kalyanaraman Rensselaer Polytechnic Institute 37 Fast Retransmit q Resend a segment after 3 duplicate ACKs q Recall: a duplicate ACK means that an out-of sequence segment was received q Notes: q duplicate ACKs due packet reordering! q if window is small don’t get duplicate ACKs! ACK 1 segment 1 cwnd = 1 cwnd = 2 segment 2 segment 3 ACK 3 cwnd = 4 segment 4 segment 5 segment 6 segment 7 ACK 1 3 duplicate ACKs ACK 4

38. Shivkumar Kalyanaraman Rensselaer Polytechnic Institute 38 Fast Recovery (Simplified) q After a fast-retransmit set cwnd to ssthresh/2 q i.e., don’t reset cwnd to 1 q But when RTO expires still do cwnd = 1 q Fast Retransmit and Fast Recovery  implemented by TCP Reno; most widely used version of TCP today

39. Shivkumar Kalyanaraman Rensselaer Polytechnic Institute 39 Fast Retransmit and Fast Recovery q Retransmit after 3 duplicated acks q prevent expensive timeouts q No need to slow start again q At steady state, cwnd oscillates around the optimal window size. Time cwnd Slow Start Congestion Avoidance

40. Shivkumar Kalyanaraman Rensselaer Polytechnic Institute 40 Fast Retransmit Time Sequence No Duplicate Acks Retransmission X

41. Shivkumar Kalyanaraman Rensselaer Polytechnic Institute 41 Multiple Losses Time Sequence No Duplicate Acks Retransmission X X X X Now what? (TCP Versions)

42. Shivkumar Kalyanaraman Rensselaer Polytechnic Institute 42 Time Sequence No X X X X TCP Versions: Tahoe Tahoe: set window to 1, and do slow start! No timeout…

43. Shivkumar Kalyanaraman Rensselaer Polytechnic Institute 43 TCP Versions: Reno Time Sequence No X X X X Now what? - timeout Reno: Recover 1 packet loss ok, but multiple loss => timeout

44. Shivkumar Kalyanaraman Rensselaer Polytechnic Institute 44 TCP Reno (Jacobson 1990) SS time window CA SS: Slow Start CA: Congestion Avoidance Fast retransmission/fast recovery

45. Shivkumar Kalyanaraman Rensselaer Polytechnic Institute 45 NewReno Time Sequence No X X X X Now what? – partial ack recovery NewReno: Recover multiple losses in successive RTTs using notion of partial ack”. No timeout.

46. Shivkumar Kalyanaraman Rensselaer Polytechnic Institute 46 SACK q Basic problem is that cumulative acks only provide little information q Alt: Selective Ack for just the packet received q What if selective acks are lost?  carry cumulative ack also! q Implementation: Bitmask of packets received q Selective acknowledgement (SACK) q Only provided as an optimization for retransmission q Fall back to cumulative acks to guarantee correctness and window updates

47. Shivkumar Kalyanaraman Rensselaer Polytechnic Institute 47 SACK Time Sequence No X X X X Now what? – send retransmissions as soon as detected

48. Shivkumar Kalyanaraman Rensselaer Polytechnic Institute 48 TCP Congestion Control Summary q Sliding window limited by receiver window. q Dynamic windows: slow start (exponential rise), congestion avoidance (additive rise), multiplicative decrease. q Ack clocking q Adaptive timeout: need mean RTT & deviation q Timer backoff and Karn’s algo during retransmission q Go-back-N or Selective retransmission q Cumulative and Selective acknowledgements q Timeout avoidance: Fast Retransmit

49. Shivkumar Kalyanaraman Rensselaer Polytechnic Institute 49 Queuing Disciplines q Each router must implement some queuing discipline q Queuing allocates bandwidth and buffer space: q Bandwidth: which packet to serve next (scheduling) q Buffer space: which packet to drop next (buff mgmt) q Queuing also affects latency Class C Class B Class A Traffic Classes Traffic Sources Drop Scheduling Buffer Management

50. Shivkumar Kalyanaraman Rensselaer Polytechnic Institute 50 Typical Internet Queuing q FIFO + drop-tail q Simplest choice q Used widely in the Internet q FIFO (first-in-first-out) q Implies single class of traffic q Drop-tail q Arriving packets get dropped when queue is full regardless of flow or importance q Important distinction: q FIFO: scheduling discipline q Drop-tail: drop (buffer management) policy

51. Shivkumar Kalyanaraman Rensselaer Polytechnic Institute 51 FIFO + Drop-tail Problems q FIFO Issues: In a FIFO discipline, the service seen by a flow is convoluted with the arrivals of packets from all other flows! q No isolation between flows: full burden on e2e control q No policing: send more packets  get more service q Drop-tail issues: q Routers are forced to have have large queues to maintain high utilizations q Larger buffers => larger steady state queues/delays q Synchronization: end hosts react to same events because packets tend to be lost in bursts q Lock-out: a side effect of burstiness and synchronization is that a few flows can monopolize queue space

52. Shivkumar Kalyanaraman Rensselaer Polytechnic Institute 52 Queue Management Ideas q Synchronization, lock-out: q Random drop: drop a randomly chosen packet q Drop front: drop packet from head of queue q High steady-state queuing vs burstiness: q Early drop: Drop packets before queue full q Do not drop packets “too early” because queue may reflect only burstiness and not true overload q Misbehaving vs Fragile flows: q Drop packets proportional to queue occupancy of flow q Try to protect fragile flows from packet loss (eg: color them or classify them on the fly) q Drop packets vs Mark packets: q Dropping packets interacts w/ reliability mechanisms q Mark packets: need to trust end-systems to respond!

53. Shivkumar Kalyanaraman Rensselaer Polytechnic Institute 53 Packet Drop Dimensions Aggregation Per-connection state Single class Drop position Head Tail Random location Class-based queuing Early dropOverflow drop

54. Shivkumar Kalyanaraman Rensselaer Polytechnic Institute 54 Random Early Detection (RED) Min thresh Max thresh Average Queue Length min th max th max P 1.0 Avg queue length P(drop)

55. Shivkumar Kalyanaraman Rensselaer Polytechnic Institute 55 Random Early Detection (RED) q Maintain running average of queue length q Low pass filtering q If avg Q < min th do nothing q Low queuing, send packets through q If avg Q > max th, drop packet q Protection from misbehaving sources q Else mark (or drop) packet in a manner proportional to queue length & bias to protect against synchronization q P b = max p (avg - min th ) / (max th - min th ) q Further, bias P b by history of unmarked packets q P a = P b /(1 - count*P b )

56. Shivkumar Kalyanaraman Rensselaer Polytechnic Institute 56 RED Issues q Issues: q Breaks synchronization well q Extremely sensitive to parameter settings q Wild queue oscillations upon load changes q Fail to prevent buffer overflow as #sources increases q Does not help fragile flows (eg: small window flows or retransmitted packets) q Does not adequately isolate cooperative flows from non-cooperative flows q Isolation: q Fair queuing achieves isolation using per-flow state q RED penalty box: Monitor history for packet drops, identify flows that use disproportionate bandwidth

57. Shivkumar Kalyanaraman Rensselaer Polytechnic Institute 57 REM Athuraliya & Low 2000 q Main ideas q Decouple congestion & performance measure q “Price” adjusted to match rate and clear buffer q Marking probability exponential in `price’ REM RED Avg queue 1

58. Shivkumar Kalyanaraman Rensselaer Polytechnic Institute 58 Comparison of AQM Performance DropTail queue = 94% RED min_th = 10 pkts max_th = 40 pkts max_p = 0.1 REM queue = 1.5 pkts utilization = 92%  = 0.05,  = 0.4,  = 1.15

59. Shivkumar Kalyanaraman Rensselaer Polytechnic Institute 59 Area = 2w 2 /3 What is TCP Throughput?  Each cycle delivers 2w 2 /3 packets  Assume: each cycle delivers 1/p packets = 2w 2 /3 q Delivers 1/p packets followed by a drop  => Loss probability = p/(1+p) ~ p if p is small. q Hence t window 2w/3 w = (4w/3+2w/3)/2 4w/3 2w/3

60. Shivkumar Kalyanaraman Rensselaer Polytechnic Institute 60 Law q Equilibrium window size q Equilibrium rate  Empirically constant a ~ 1 q Verified extensively through simulations and on Internet q References q T.J.Ott, J.H.B. Kemperman and M.Mathis (1996) q M.Mathis, J.Semke, J.Mahdavi, T.Ott (1997) q T.V.Lakshman and U.Mahdow (1997) q J.Padhye, V.Firoiu, D.Towsley, J.Kurose (1998)