1. How to get good TCP performance over asymmetric networks Hari Balakrishnan MIT Lab for Computer Science http://wind.lcs.mit.edu/ hari@lcs.mit.edu With: Venkata Padmanabhan (Microsoft Research) and Randy Katz (UC Berkeley)

2. Motivation Many emerging (in some cases, “emerged!”) networks are asymmetric –Cable modems, “wireless” cable, (A)DSL –Direct Broadcast Satellites –Packet radio networks How does TCP behave over asymmetric networks? –TCP relies on ACK feedback to work, which might be an issue

3. When is a network asymmetric? Bandwidth: 10- 1000 X more in forward direction Latency: Due to MAC protocol interactions Loss-rate: Higher loss-rate in one direction The network characteristics in one direction significantly affect performance in the other Sender Receiver Forward Reverse Router

4. Why is this a problem for TCP? TCP relies on ACKs for reliability and for congestion control New, cumulative ACKs “slide” and “bump up” window, allowing new transmissions (ACK clocking) Smooth ACK flow  smooth transmits; flaky ACK flow  bursty transmits 7 4 3 6 5 Cumulative Acknowledgments (ACK) 1 2 3 4 2

5. 0 TCP Overview –Window-based algorithm to determine sustainable rate –Upon congestion, reduce window –“ACK clocking” sends data smoothly 8 1 1 7 6 1 3 4 1 2 lost 1 5 Timeout = srtt + 4 * rtt_deviation (coarse-grained) Fast retransmissions when duplicate ACKs arrive 1. Loss recovery 2. Congestion control 0 10 9

6. TCP Dynamics Data ACKs Window Fast retransmission Duplicate ACKs Sequence number (bytes) Time (s) RTT

7. Typical Performance Goal: To bridge the gap between perceived and rated performance

8. Bandwidth Asymmetry Shows up in asymmetric access technologies –ADSL, (wireless) cable modems, DBS Low-bandwidth ACK channel ACKs don’t come back fast enough or get lost Problem: Imperfect ACK feedback degrades TCP performance

9. Problems Sender Receiver Forward ACKs Router Bottleneck Router 0 Data 8 Data 11 Data 10 Data 9 3214567 1. Acks arrive slowly (large buffer) 25 471 36 2. Acks are dropped (small buffer) 1 Data 3. Acks are queued behind data packets Lakshman & Madhow 97 Kalampoukas et al. 97 Balakrishnan et al. 97 Performance depends on 1. Normalized asymmetry ratio 2. Bottleneck buffer sizes 3. Competing reverse traffic

10. Hybrid Wireless Cable 0 1 2 3 4 5 6 020406080100120140160180200 TCP Throughput (Mbps) 10 Mbps Ethernet 28.8 C-SLIP 28.8 SLIP 9.6 C-SLIP 9.6 SLIP Return channel speed and latency affects performance Receiver socket buffer size (KB)

11. Latency Asymmetry: Packet Radio Networks Internet PT ER FH GW MH Modem PR ER PT Fixed Host Ethernet Radios Poletop Radios Mobile Host RTS CTS Half-duplex radios Synchronization before communication

12. MAC Protocol Interactions Internet PT ER FH GW MH Modem PR ER PT Fixed Host Ethernet Radios Poletop Radios Mobile Host Pkt Ack RTS No response Exponential backoff Problem: Large, variable communication latency

13. Problem: Large Round-Trip Time Variations Metricom Ricochet Wireless Network Mean rtt = 2.45s, std deviation = 1.5s  long timeout! Long idle periods after multiple losses (~ 20 Kbps) In contrast, UDP throughput = 50-64 Kbps ACK flow affects data latency Fast retransmissions Timeouts

14. Solutions Problems arise because of imperfect ACK feedback 1. Reduce frequency of acks –ACK Filtering (AF) –ACK Congestion Control (ACC) 2. Handle infrequent acks –Sender Adaptation (SA) –ACK Reconstruction (AR) General solutions for different asymmetric situations

15. ACK Filtering (AF) 975 1113 375 Forward Router 1 Server Client Router Purge all redundant, cumulative ACKs from constrained reverse queue No persistent filter state Used in conjunction with sender adaptation or ACK reconstruction

16. Server Client Router 8 Data 21 Data 22 Data 20 18 16 14 Data 19 10 Delack factor = 2 Adaptive extension of TCP delayed ACKs Forward ACK Congestion Control (ACC)

17. Sender Router Data 22 Data 12 Data RED marking of ECN bit (Explicit Congestion Notification) Delack factor = 2 Receiver Forward

18. ACK Congestion Control (ACC) Router Data Data 40 Data 22 Echo ECN marking to receiver Delack factor = 2 Forward Sender Receiver

19. ACK Congestion Control (ACC) Router Data 42 Data 43 Data 41 3640 Data 40 Delack factor = 4 Forward Sender Receiver

20. Sender Adaptation (SA) Infrequent ACKs slow window growth Sender tends to be bursty Server 1 915 1. cwnd += 8 cwnd += 8/cwnd Incr window by amount acked 19202122... 2. Regulation: pace packets out at rate estimated by cwnd/srtt This reduces burstiness Forward Receiver

21. ACK Reconstruction (AR) 975 1 1113 375 ACK filterACK reconstructor 753 9 Server Forward Client Regenerate ACKs at other end of reverse link Shield sender from gaps in ACK sequence (no TCP changes needed) AR rate determined by input ACK rate and target ACK spacing Reconstructor state is soft 1

22. Performance: Bandwidth Asymmetry –TCP transfers in the forward direction alone –Maximum window size 100 KB; no forward losses –Header compression helps –Large reverse buffer hurts for Reno and ACC –Fairness greatly improves using AF and ACC

23. Multihop Wireless Simulations –1 to 3 wireless hops on path –Radio turnaround time = 3-12 ms –Radio queue size = 10 packets –Exponential backoff in multiples of 20 ms slots ER PT Receiver Sender 100 Kbps, 10 ms 10 Mbps 1 ms PT

24. Single TCP Transfer AF reduces chances that peer radio is busy –MAC backoffs less frequent RTT std deviation reduces from 1.5s to 0.6s Throughput (Kbps) AF: 20-35% throughput improvement compared to Reno

25. Concurrent Transfers Metrics: utilization and fairness Simultaneous connections over 2-hop network –Predictable and consistent performance with AF Long timeouts cause unpredictable performance AF: 25% improvement in fairness over Reno

26. Combining Technologies Internet server Internet Hybrid PoP Client 10 Mbps, 2 ms Wireless transmitter PT ER Requests & ACKs Web data Cable forward channel with packet radio reverse channel Workload: Multiple concurrent Web-like transfers Result: Ack filtering greatly improves scaling behavior (average completion time vs. # of concurrent transactions)

27. Status Extensive simulation results under various bandwidth and latency asymmetric conditions Implementation and evaluation under BSD/OS 3.0 (code available) Documentation –Paper to appear ACM MONET (extended, revised version from MOBICOM 97 –Internet-draft submitted (PILC WG)

28. Cross-layer optimizations A way of performing protocol-specific optimizations at link-layer Lightweight, soft-state agents in network infrastructure State is soft: not essential for correctness; improves performance Easy to implement! Examples include ACK filters and reconstructors

29. Summary Asymmetry affects TCP performance –ACK channel affects forward throughput Asymmetry comes in various flavors –Bandwidth asymmetry (technology) –Latency asymmetry (MAC interactions) Fortunately, good performance can be obtained using a two-pronged approach –Reduce frequency of ACKs (AF, ACC) –Handle infrequent ACKs (SA, AR) Cross-layer optimizations are A Good Thing!