Mark Claypool, Feng Li and Jae Chung BBR' - An Implementation of Bottleneck Bandwidth and Round-trip Time Congestion Control for ns-3 Mark Claypool, Feng Li and Jae Chung In Proceedings of the Workshop on ns-3 (WNS3) Mangalore, India, June 2018 http://www.cs.wpi.edu/~claypool/papers/bbr-prime/

Revisit model of congested link Introduction TCP dominant protocol on Internet Yesterday Designed in an era of limited resources – low network bandwidths, small queues Packet loss a sign of congestion Today Router queues can be large (buffer bloat) Packet loss can be from wireless errors https://image.slidesharecdn.com/bufferbloatpresentation-140602070926-phpapp02/95/effekterna-av-bufferbloat-och-latency-p-internet-4-638.jpg?cb=1401693007 https://image.slidesharecdn.com/7-140402115120-phpapp02/95/7-wang-jingyuan-tcpfit-4-638.jpg?cb=1396439521 Revisit model of congested link

Congestion and Bottlenecks

Congestion and Bottlenecks

Congestion and Bottlenecks

Congestion and Bottlenecks

Congestion and Bottlenecks Bandwidth-Delay Product

Estimating Optimal Point

Estimating Optimal Point To see both max BW and min RTT  Must probe on both sides of BDP!

Estimating Optimal Point N. Cardwell, Y. Cheng, C.S. Gunn, S.H. Yeganeh, and V. Jacobson. BBR: Congestion-Based Congestion Control,Communications of the ACM, 60(2), February 2017. BBR – Bottleneck Bandwidth and Round-trip time Model network Each ACK, update max BW and min RTT Control sending based on model Probe for max BW and min RTT Pace near estimate BW Keep inflight near BDP Goal: high throughput with small bottleneck queue

Outline Introduction (done) BBR (next) BBR’ Validation Evaluation Conclusions

BBR – Walk towards max BW, min RTT

BBR – STARTUP: exponential search

BBR – DRAIN: drain queue built up in STARTUP

BBR – PROBE_BW: explore max BW, drain, cruise

BBR – PROBE_RTT: check if min RTT changed

BBR – State Machine 2 Phases of probing Warm-up Steady-state 2 Phases of probing Increase inflight to get max BW Lower inflight to get min RTT  Apply during Warm-up and Steady-state Warm-up Startup – ramp up quickly (ala Slow Start) Drain – drain built up queue Steady-state ProbeBW – cycle pacing rate to probe BW ProbeRTT – if needed, coordinated dip to probe RTT

BBR – Median Latency versus Buffer Size BBR vs. CUBIC. Synthetic bulk download. 8 flows, bottleneck 128 Kb/s, RTT 40 ms

BBR – Mean Goodput versus Loss Rate BBR vs. CUBIC. Synthetic bulk download. 1 flow, bottleneck 100 Mb/s, RTT 100 ms

BBR Status Status of BBR v1.0 Actively working on BBR v2.0 Deployed widely at Google Open source for Linux TCP and QUIC Documented in IETF Internet Drafts ACK aggregation (e.g., WiFi) Packet loss signals Dynamics of sharing with loss- based congestion control FreeBSD TCP @ NetFlix https://datatracker.ietf.org/meeting/101/materials/slides-101-iccrg-an-update-on-bbr-work-at-google-00

ns – The Network Simulator https://www.nsnam.org/ Discrete event simulators ns-1 (LBNL) 1995 – 2001 ns-2 (DARPA) 2000 – 2010 ns-3 (NSF and INRIA) 2008 – present Educational and Research use Yearly workshop (WNS3) Share ideas, code, research https://tinyurl.com/y7gof6p6 Does not support BBR! Implementation of BBR for ns-3 

Outline Introduction (done) BBR (done) BBR’ (next) Validation Evaluation Conclusions

BBR’ Implementation Protocol states BDP computation N. Cardwell, Y. Cheng, S. H. Yeganeh, and V. Jacobson. BBR Congestion Control, IETF Draft, July 2017. Y. Cheng, N. Cardwell, S. H. Yeganeh, and V. Jacobson. Delivery Rate Estimation, IETF Draft, July 2017. Developed based on specifications: Protocol states Solution: state machine BDP computation Solution: compute each ACK Delivery rate estimation Solution: hook to record when send Packet pacing Solution: “hijack” at TCP layer

BBR’ Implementation BBR’ APP TCP BBR’: On ACK, update RTT & BW BBR’: Compute & set pacing rate BBR’: Compute BDP & set cwnd Device ready to send?  Callback to APP APP: call send() TCP: enqueue packet TCP: timer triggers paced send BBR’: on send, book-keeping TCP: actually send TcpBbr BulkSend- Application BbrStateMachine TCP TcpSocketBase TcpSocketState

BBR’ Implementation BBR’ APP TCP BBR’: On ACK, update RTT & BW BBR’: Compute & set pacing rate BBR’: Compute BDP & set cwnd Device ready to send?  Callback to APP APP: call send() TCP: enqueue packet TCP: timer triggers paced send BBR’: on send, book-keeping TCP: actually send TcpBbr BulkSend- Application BbrStateMachine PktsAcked() 1 TCP TcpSocketBase TcpSocketState

BBR’ Implementation BBR’ APP TCP BBR’: On ACK, update RTT & BW BBR’: Compute & set pacing rate BBR’: Compute BDP & set cwnd Device ready to send?  Callback to APP APP: call send() TCP: enqueue packet TCP: timer triggers paced send BBR’: on send, book-keeping TCP: actually send TcpBbr BulkSend- Application BbrStateMachine PktsAcked() 1 TCP TcpSocketBase TcpSocketState SetPacingRate() 2

BBR’ Implementation BBR’ APP TCP BBR’: On ACK, update RTT & BW tcb → m_cwnd APP 3 BBR’: On ACK, update RTT & BW BBR’: Compute & set pacing rate BBR’: Compute BDP & set cwnd Device ready to send?  Callback to APP APP: call send() TCP: enqueue packet TCP: timer triggers paced send BBR’: on send, book-keeping TCP: actually send TcpBbr BulkSend- Application BbrStateMachine PktsAcked() 1 TCP TcpSocketBase TcpSocketState SetPacingRate() 2

BBR’ Implementation BBR’ APP TCP BBR’: On ACK, update RTT & BW tcb → m_cwnd APP 3 BBR’: On ACK, update RTT & BW BBR’: Compute & set pacing rate BBR’: Compute BDP & set cwnd Device ready to send?  Callback to APP APP: call send() TCP: enqueue packet TCP: timer triggers paced send BBR’: on send, book-keeping TCP: actually send TcpBbr BulkSend- Application BbrStateMachine PktsAcked() 1 TCP TcpSocketBase TcpSocketState DataSend() SetPacingRate() 4 2

BBR’ Implementation BBR’ APP TCP BBR’: On ACK, update RTT & BW tcb → m_cwnd APP 3 BBR’: On ACK, update RTT & BW BBR’: Compute & set pacing rate BBR’: Compute BDP & set cwnd Device ready to send?  Callback to APP APP: call send() TCP: enqueue packet TCP: timer triggers paced send BBR’: on send, book-keeping TCP: actually send TcpBbr BulkSend- Application BbrStateMachine PktsAcked() 1 TCP TcpSocketBase TcpSocketState Send() DataSend() SetPacingRate() 5 4 2

BBR’ Implementation BBR’ APP TCP BBR’: On ACK, update RTT & BW tcb → m_cwnd APP 3 BBR’: On ACK, update RTT & BW BBR’: Compute & set pacing rate BBR’: Compute BDP & set cwnd Device ready to send?  Callback to APP APP: call send() TCP: enqueue packet TCP: timer triggers paced send BBR’: on send, book-keeping TCP: actually send TcpBbr BulkSend- Application BbrStateMachine PktsAcked() 1 TCP TcpSocketBase TcpSocketState Send() DataSend() SetPacingRate() 5 4 2 SendDataPacket() 6

BBR’ Implementation BBR’ APP TCP BBR’: On ACK, update RTT & BW tcb → m_cwnd APP 3 BBR’: On ACK, update RTT & BW BBR’: Compute & set pacing rate BBR’: Compute BDP & set cwnd Device ready to send?  Callback to APP APP: call send() TCP: enqueue packet TCP: timer triggers paced send BBR’: on send, book-keeping TCP: actually send TcpBbr BulkSend- Application BbrStateMachine PktsAcked() 1 TCP TcpSocketBase TcpSocketState Send() DataSend() SetPacingRate() 5 4 2 (Timer) SendDataPacket() 6 PacePackets() 7

BBR’ Implementation BBR’ APP TCP BBR’: On ACK, update RTT & BW tcb → m_cwnd APP 3 BBR’: On ACK, update RTT & BW BBR’: Compute & set pacing rate BBR’: Compute BDP & set cwnd Device ready to send?  Callback to APP APP: call send() TCP: enqueue packet TCP: timer triggers paced send BBR’: on send, book-keeping TCP: actually send TcpBbr BulkSend- Application BbrStateMachine Send() 8 PktsAcked() 1 TCP TcpSocketBase TcpSocketState Send() DataSend() SetPacingRate() 5 4 2 (Timer) SendDataPacket() 6 PacePackets() 7

SendDataPacketReal() BBR’ Implementation BBR’ tcb → m_cwnd APP 3 BBR’: On ACK, update RTT & BW BBR’: Compute & set pacing rate BBR’: Compute BDP & set cwnd Device ready to send?  Callback to APP APP: call send() TCP: enqueue packet TCP: timer triggers paced send BBR’: on send, book-keeping TCP: actually send TcpBbr BulkSend- Application BbrStateMachine Send() 8 PktsAcked() 1 TCP TcpSocketBase TcpSocketState Send() DataSend() SetPacingRate() 5 4 2 (Timer) SendDataPacket() 6 PacePackets() 7 SendDataPacketReal() 9

SendDataPacketReal() BBR’ Implementation BBR’ tcb → m_cwnd APP 3 BBR’: On ACK, update RTT & BW BBR’: Compute & set pacing rate BBR’: Compute BDP & set cwnd Device ready to send?  Callback to APP APP: call send() TCP: enqueue packet TCP: timer triggers paced send BBR’: on send, book-keeping TCP: actually send TcpBbr BulkSend- Application BbrStateMachine Send() 8 PktsAcked() 1 TCP TcpSocketBase TcpSocketState Send() DataSend() SetPacingRate() 5 4 2 (Timer) SendDataPacket() 6 PacePackets() 7 SendDataPacketReal() 9

Outline Introduction (done) BBR (done) BBR’ (done) Validation (next) Evaluation Conclusions

Verification: BBR’ – Basic Bottleneck Single flow. Bottleneck: 10 Mb/s, 22 ms RTT BBR’ works as expected (based on specification)

Verification: BBR’ – Basic Bottleneck (Estimated versus Actual) BBR’ accurately infers RTT and BW

Validation: BBR’ versus BBR – Steady State Single flow. Bottleneck: 10 Mb/s, 40 ms RTT BBR’ looks similar to BBR

Validation: BBR’ versus BBR – Bwidth Increase Single flow. Bottleneck: 40 ms RTT. Time 20: 10 Mb/s  20 Mb/s BBR’ looks similar to BBR

Validation: BBR’ versus BBR – Bwidth Decrease Single flow. Bottleneck: 40 ms RTT. Time 40: 20 Mb/s  10 Mb/s BBR’ looks similar to BBR

Outline Introduction (done) BBR (done) BBR’ (done) Validation (done) Evaluation (next) Conclusions

Evaluation: BBR’ vs. CUBIC – Wired BBR’ has similar Throughput, but lower RTT

Evaluation: BBR’ vs. CUBIC – Wireless 5 km Evaluation: BBR’ vs. CUBIC – Wireless Results for 4G LTE similar to wired

Evaluation: BBR’ vs. CUBIC – Wireless 0 – 20 km Evaluation: BBR’ vs. CUBIC – Wireless BBR’ has similar Throughput, but lower RTT with distance

Apoova Lad and Chengle Zhang for early work Conclusions Evolution of networks needs evolution of TCP BBR promising, but needs more vetting ns-3 network simulator great for vetting, but no BBR BBR’ – Implementation of BBR for ns-3 Use existing CC code base Integrate with other NS layers Validation – looks like BBR Evaluation compared to CUBIC Similar throughputs Lower RTTS Acknowledgements Apoova Lad and Chengle Zhang for early work

Future Work Incorporate more BBR features Application limited flows Adverse Internet conditions – packet reordering, ACK loss Adjustments of BBR parameters Pacing Gain rates Evaluation over more network conditions Competing protocol stacks Wireless (WiFi and LTE) Queue sizes (small, super large)

Mark Claypool, Feng Li and Jae Chung BBR' - An Implementation of Bottleneck Bandwidth and Round-trip Time Congestion Control for ns-3 Mark Claypool, Feng Li and Jae Chung In Proceedings of the Workshop on ns-3 (WNS3) Mangalore, India, June 2018 http://www.cs.wpi.edu/~claypool/papers/bbr-prime/