How to get good TCP performance over asymmetric networks Hari Balakrishnan MIT Lab for Computer Science With: Venkata Padmanabhan (Microsoft Research) and Randy Katz (UC Berkeley)
Motivation Many emerging (in some cases, “emerged!”) networks are asymmetric –Cable modems, “wireless” cable, (A)DSL –Direct Broadcast Satellites –Packet radio networks How does TCP behave over asymmetric networks? –TCP relies on ACK feedback to work, which might be an issue
When is a network asymmetric? Bandwidth: X more in forward direction Latency: Due to MAC protocol interactions Loss-rate: Higher loss-rate in one direction The network characteristics in one direction significantly affect performance in the other Sender Receiver Forward Reverse Router
Why is this a problem for TCP? TCP relies on ACKs for reliability and for congestion control New, cumulative ACKs “slide” and “bump up” window, allowing new transmissions (ACK clocking) Smooth ACK flow smooth transmits; flaky ACK flow bursty transmits Cumulative Acknowledgments (ACK)
0 TCP Overview –Window-based algorithm to determine sustainable rate –Upon congestion, reduce window –“ACK clocking” sends data smoothly lost 1 5 Timeout = srtt + 4 * rtt_deviation (coarse-grained) Fast retransmissions when duplicate ACKs arrive 1. Loss recovery 2. Congestion control
TCP Dynamics Data ACKs Window Fast retransmission Duplicate ACKs Sequence number (bytes) Time (s) RTT
Typical Performance Goal: To bridge the gap between perceived and rated performance
Bandwidth Asymmetry Shows up in asymmetric access technologies –ADSL, (wireless) cable modems, DBS Low-bandwidth ACK channel ACKs don’t come back fast enough or get lost Problem: Imperfect ACK feedback degrades TCP performance
Problems Sender Receiver Forward ACKs Router Bottleneck Router 0 Data 8 Data 11 Data 10 Data Acks arrive slowly (large buffer) Acks are dropped (small buffer) 1 Data 3. Acks are queued behind data packets Lakshman & Madhow 97 Kalampoukas et al. 97 Balakrishnan et al. 97 Performance depends on 1. Normalized asymmetry ratio 2. Bottleneck buffer sizes 3. Competing reverse traffic
Hybrid Wireless Cable TCP Throughput (Mbps) 10 Mbps Ethernet 28.8 C-SLIP 28.8 SLIP 9.6 C-SLIP 9.6 SLIP Return channel speed and latency affects performance Receiver socket buffer size (KB)
Latency Asymmetry: Packet Radio Networks Internet PT ER FH GW MH Modem PR ER PT Fixed Host Ethernet Radios Poletop Radios Mobile Host RTS CTS Half-duplex radios Synchronization before communication
MAC Protocol Interactions Internet PT ER FH GW MH Modem PR ER PT Fixed Host Ethernet Radios Poletop Radios Mobile Host Pkt Ack RTS No response Exponential backoff Problem: Large, variable communication latency
Problem: Large Round-Trip Time Variations Metricom Ricochet Wireless Network Mean rtt = 2.45s, std deviation = 1.5s long timeout! Long idle periods after multiple losses (~ 20 Kbps) In contrast, UDP throughput = Kbps ACK flow affects data latency Fast retransmissions Timeouts
Solutions Problems arise because of imperfect ACK feedback 1. Reduce frequency of acks –ACK Filtering (AF) –ACK Congestion Control (ACC) 2. Handle infrequent acks –Sender Adaptation (SA) –ACK Reconstruction (AR) General solutions for different asymmetric situations
ACK Filtering (AF) Forward Router 1 Server Client Router Purge all redundant, cumulative ACKs from constrained reverse queue No persistent filter state Used in conjunction with sender adaptation or ACK reconstruction
Server Client Router 8 Data 21 Data 22 Data Data Delack factor = 2 Adaptive extension of TCP delayed ACKs Forward ACK Congestion Control (ACC)
Sender Router Data 22 Data 12 Data RED marking of ECN bit (Explicit Congestion Notification) Delack factor = 2 Receiver Forward
ACK Congestion Control (ACC) Router Data Data 40 Data 22 Echo ECN marking to receiver Delack factor = 2 Forward Sender Receiver
ACK Congestion Control (ACC) Router Data 42 Data 43 Data Data 40 Delack factor = 4 Forward Sender Receiver
Sender Adaptation (SA) Infrequent ACKs slow window growth Sender tends to be bursty Server cwnd += 8 cwnd += 8/cwnd Incr window by amount acked Regulation: pace packets out at rate estimated by cwnd/srtt This reduces burstiness Forward Receiver
ACK Reconstruction (AR) ACK filterACK reconstructor Server Forward Client Regenerate ACKs at other end of reverse link Shield sender from gaps in ACK sequence (no TCP changes needed) AR rate determined by input ACK rate and target ACK spacing Reconstructor state is soft 1
Performance: Bandwidth Asymmetry –TCP transfers in the forward direction alone –Maximum window size 100 KB; no forward losses –Header compression helps –Large reverse buffer hurts for Reno and ACC –Fairness greatly improves using AF and ACC
Multihop Wireless Simulations –1 to 3 wireless hops on path –Radio turnaround time = 3-12 ms –Radio queue size = 10 packets –Exponential backoff in multiples of 20 ms slots ER PT Receiver Sender 100 Kbps, 10 ms 10 Mbps 1 ms PT
Single TCP Transfer AF reduces chances that peer radio is busy –MAC backoffs less frequent RTT std deviation reduces from 1.5s to 0.6s Throughput (Kbps) AF: 20-35% throughput improvement compared to Reno
Concurrent Transfers Metrics: utilization and fairness Simultaneous connections over 2-hop network –Predictable and consistent performance with AF Long timeouts cause unpredictable performance AF: 25% improvement in fairness over Reno
Combining Technologies Internet server Internet Hybrid PoP Client 10 Mbps, 2 ms Wireless transmitter PT ER Requests & ACKs Web data Cable forward channel with packet radio reverse channel Workload: Multiple concurrent Web-like transfers Result: Ack filtering greatly improves scaling behavior (average completion time vs. # of concurrent transactions)
Status Extensive simulation results under various bandwidth and latency asymmetric conditions Implementation and evaluation under BSD/OS 3.0 (code available) Documentation –Paper to appear ACM MONET (extended, revised version from MOBICOM 97 –Internet-draft submitted (PILC WG)
Cross-layer optimizations A way of performing protocol-specific optimizations at link-layer Lightweight, soft-state agents in network infrastructure State is soft: not essential for correctness; improves performance Easy to implement! Examples include ACK filters and reconstructors
Summary Asymmetry affects TCP performance –ACK channel affects forward throughput Asymmetry comes in various flavors –Bandwidth asymmetry (technology) –Latency asymmetry (MAC interactions) Fortunately, good performance can be obtained using a two-pronged approach –Reduce frequency of ACKs (AF, ACC) –Handle infrequent ACKs (SA, AR) Cross-layer optimizations are A Good Thing!