1. Multipath TCP Costin Raiciu University Politehnica of Bucharest Joint work with: Mark Handley, Damon Wischik, University College London Olivier Bonaventure, Sébastien Barré, Université catholique de Louvain and many many others Christoph Paasch Université catholique de Louvain Thanks to

2. Networks are becoming multipath Mobile devices have multiple wireless connections

3. Networks are becoming multipath

5. Datacenters have redundant topologies

6. Networks are becoming multipath Servers are multi-homed Client

7. How do we use these networks? TCP. Used by most applications, offers byte-oriented reliable delivery, adjusts load to network conditions [Labovits et al – Internet Interdomain traffic – Sigcomm 2010]

8. TCP is single path A TCP connection Uses a single-path in the network regardless of network topology Is tied to the source and destination addresses of the endpoints

9. Mismatch between network and transport creates problems

10. Poor Performance for Mobile Users 3G celltower

11. Poor Performance for Mobile Users 3G celltower

12. Poor Performance for Mobile Users 3G celltower

13. Poor Performance for Mobile Users 3G celltower Offload to WiFi

14. Poor Performance for Mobile Users 3G celltower All ongoing TCP connections die

15. Collisions in datacenters [Fares et al - A Scalable, Commodity Data Center Network Architecture - Sigcomm 2008]

16. Single-path TCP collisions reduce throughput [Raiciu et. Al – Sigcomm 2011]

17. Multipath TCP

18. Multipath TCP (MPTCP) is an evolution of TCP that can effectively use multiple paths within a single transport connection Supports unmodified applications Works over today’s networks Standardized at the IETF (almost there)

19. Multipath TCP components Connection setup Sending data over multiple paths Encoding control information Dealing with (many) middleboxes Congestion control [Raiciu et. al – NSDI 2012] [Wischik et. al – NSDI 2011]

20. Multipath TCP components Connection setup Sending data over multiple paths Encoding control information Dealing with (many) middleboxes Congestion control [Raiciu et. al – NSDI 2012] [Wischik et. al – NSDI 2011]

21. MPTCP Connection Management SYN MP_CAPABLE X

22. MPTCP Connection Management SYN/ACK MP_CAPABLE Y

23. MPTCP Connection Management SUBFLOW 1 CWND Snd.SEQNO Rcv.SEQNO FLOW Y

24. MPTCP Connection Management SYN JOIN Y SUBFLOW 1 CWND Snd.SEQNO Rcv.SEQNO FLOW Y

25. MPTCP Connection Management SYN/ACK JOIN X SUBFLOW 1 CWND Snd.SEQNO Rcv.SEQNO FLOW Y

26. MPTCP Connection Management SUBFLOW 2 CWND Snd.SEQNO Rcv.SEQNO SUBFLOW 1 CWND Snd.SEQNO Rcv.SEQNO FLOW Y

27. TCP Packet Header Source PortDestination Port Sequence Number Acknowledgment Number Receive Window Header Length ReservedCode bits Checksum Options Urgent Pointer Data Bit 0Bit 15Bit 16Bit 31 20 Bytes 0 - 40 Bytes

28. TCP Packet Header Source PortDestination Port Sequence Number Acknowledgment Number Receive Window Header Length ReservedCode bits Checksum Options Urgent Pointer Data Bit 0Bit 15Bit 16Bit 31 20 Bytes 0 - 40 Bytes

29. Sequence Numbers Packets go multiple paths. – Need sequence numbers to put them back in sequence. – Need sequence numbers to infer loss on a single path. Options: – One sequence space shared across all paths? – One sequence space per path, plus an extra one to put data back in the correct order at the receiver?

30. Sequence Numbers One sequence space per path is preferable. – Loss inference is more reliable. – Some firewalls/proxies expect to see all the sequence numbers on a path. Outer TCP header holds subflow sequence numbers. – Where do we put the data sequence numbers?

31. MPTCP Packet Header Source PortDestination Port Sequence Number Acknowledgment Number Receive Window Header Length ReservedCode bits Checksum Options Urgent Pointer Data Bit 0Bit 15Bit 16Bit 31 20 Bytes 0 - 40 Bytes Subflow Data sequence number Data ACK

32. MPTCP Operation DATA SEQ 1000 DSEQ 10000 options …… SUBFLOW 2 CWND Snd.SEQNO Rcv.SEQNO SUBFLOW 1 CWND Snd.SEQNO Rcv.SEQNO FLOW Y

33. MPTCP Operation DATA SEQ 1000 DSEQ 10000 options …… SUBFLOW 2 CWND Snd.SEQNO Rcv.SEQNO SUBFLOW 1 CWND Snd.SEQNO Rcv.SEQNO FLOW Y

34. MPTCP Operation DATA SEQ 1000 DSEQ 10000 options ……DATA SEQ 5000 DSEQ 11000 options …… SUBFLOW 2 CWND Snd.SEQNO Rcv.SEQNO SUBFLOW 1 CWND Snd.SEQNO Rcv.SEQNO FLOW Y

35. MPTCP Operation DATA SEQ 1000 DSEQ 10000 options ……DATA SEQ 5000 DSEQ 11000 options …… SUBFLOW 2 CWND Snd.SEQNO Rcv.SEQNO SUBFLOW 1 CWND Snd.SEQNO Rcv.SEQNO FLOW Y

36. MPTCP Operation DATA SEQ 1000 DSEQ 10000 options ……DATA SEQ 5000 DSEQ 11000 options …… SUBFLOW 2 CWND Snd.SEQNO Rcv.SEQNO SUBFLOW 1 CWND Snd.SEQNO Rcv.SEQNO FLOW Y

37. MPTCP Operation ACK 2000 Data ACK 11000 …… SUBFLOW 2 CWND Snd.SEQNO Rcv.SEQNO SUBFLOW 1 CWND Snd.SEQNO Rcv.SEQNO DATA SEQ 5000 DSEQ 11000 options …… FLOW Y

38. MPTCP Operation DATA SEQ 5000 DSEQ 11000 options …… SUBFLOW 2 CWND Snd.SEQNO Rcv.SEQNO SUBFLOW 1 CWND Snd.SEQNO Rcv.SEQNO FLOW Y

39. MPTCP Operation DATA SEQ 5000 DSEQ 11000 options …… SUBFLOW 2 CWND Snd.SEQNO Rcv.SEQNO SUBFLOW 1 CWND Snd.SEQNO Rcv.SEQNO FLOW Y

40. MPTCP Operation DATA SEQ 2000 DSEQ 11000 options …… SUBFLOW 2 CWND Snd.SEQNO Rcv.SEQNO SUBFLOW 1 CWND Snd.SEQNO Rcv.SEQNO FLOW Y

41. Multipath TCP Congestion Control

42. Packet switching ‘pools’ circuits. Multipath ‘pools’ links TCP controls how a link is shared. How should a pool be shared? Two circuitsA link Two separate links A pool of links 42

43. To be fair, Multipath TCP should take as much capacity as TCP at a bottleneck link, no matter how many paths it is using. Design goal 1: Multipath TCP should be fair to regular TCP at shared bottlenecks To be fair, Multipath TCP should take as much capacity as TCP at a bottleneck link, no matter how many subflows it is using. A multipath TCP flow with two subflows Regular TCP 43

44. To be fair, Multipath TCP should take as much capacity as TCP at a bottleneck link, no matter how many paths it is using. Design goal 2: MPTCP should use efficient paths Each flow has a choice of a 1-hop and a 2-hop path. How should split its traffic? 12Mb/s 44

45. To be fair, Multipath TCP should take as much capacity as TCP at a bottleneck link, no matter how many paths it is using. Design goal 2: MPTCP should use efficient paths If each flow split its traffic 1:1... 8Mb/s 12Mb/s 45

46. To be fair, Multipath TCP should take as much capacity as TCP at a bottleneck link, no matter how many paths it is using. Design goal 2: MPTCP should use efficient paths If each flow split its traffic 2:1... 9Mb/s 12Mb/s 46

47. To be fair, Multipath TCP should take as much capacity as TCP at a bottleneck link, no matter how many paths it is using. Design goal 2: MPTCP should use efficient paths If each flow split its traffic 4:1... 10Mb/s 12Mb/s 47

48. To be fair, Multipath TCP should take as much capacity as TCP at a bottleneck link, no matter how many paths it is using. Design goal 2: MPTCP should use efficient paths If each flow split its traffic ∞:1... 12Mb/s 48

49. To be fair, Multipath TCP should take as much capacity as TCP at a bottleneck link, no matter how many paths it is using. Design goal 3: MPTCP should get at least as much as TCP on the best path Design Goal 2 says to send all your traffic on the least congested path, in this case 3G. But this has high RTT, hence it will give low throughput. wifi path: high loss, small RTT 3G path: low loss, high RTT 49

50. Maintain a congestion window w. Increase w for each ACK, by 1/ w Decrease w for each drop, by w/ 2 How does TCP congestion control work? 50

51. Maintain a congestion window w r, one window for each path, where r ∊ R ranges over the set of available paths. Increase w r for each ACK on path r, by Decrease w r for each drop on path r, by w r / 2 51 How does MPTCP congestion control work?

52. Maintain a congestion window w r, one window for each path, where r ∊ R ranges over the set of available paths. Increase w r for each ACK on path r, by Decrease w r for each drop on path r, by w r / 2 52 How does MPTCP congestion control work? Goal 2

53. Maintain a congestion window w r, one window for each path, where r ∊ R ranges over the set of available paths. Increase w r for each ACK on path r, by Decrease w r for each drop on path r, by w r / 2 53 How does MPTCP congestion control work? Goals 1&3

54. Maintain a congestion window w r, one window for each path, where r ∊ R ranges over the set of available paths. Increase w r for each ACK on path r, by Decrease w r for each drop on path r, by w r / 2 54 How does MPTCP congestion control work?

55. Applications of Multipath TCP

56. At a multihomed web server, MPTCP tries to share the ‘pooled access capacity’ fairly. 56 100Mb/s 2 TCPs @ 50Mb/s 4 TCPs @ 25Mb/s

57. At a multihomed web server, MPTCP tries to share the ‘pooled access capacity’ fairly. 57 100Mb/s 2 TCPs @ 33Mb/s 1 MPTCP @ 33Mb/s 4 TCPs @ 25Mb/s

58. At a multihomed web server, MPTCP tries to share the ‘pooled access capacity’ fairly. 58 100Mb/s 2 TCPs @ 25Mb/s 2 MPTCPs @ 25Mb/s 4 TCPs @ 25Mb/s The total capacity, 200Mb/s, is shared out evenly between all 8 flows.

59. At a multihomed web server, MPTCP tries to share the ‘pooled access capacity’ fairly. 59 100Mb/s 2 TCPs @ 22Mb/s 3 MPTCPs @ 22Mb/s 4 TCPs @ 22Mb/s The total capacity, 200Mb/s, is shared out evenly between all 9 flows. It’s as if they were all sharing a single 200Mb/s link. The two links can be said to form a 200Mb/s pool.

60. At a multihomed web server, MPTCP tries to share the ‘pooled access capacity’ fairly. 60 100Mb/s 2 TCPs @ 20Mb/s 4 MPTCPs @ 20Mb/s 4 TCPs @ 20Mb/s The total capacity, 200Mb/s, is shared out evenly between all 10 flows. It’s as if they were all sharing a single 200Mb/s link. The two links can be said to form a 200Mb/s pool.

61. At a multihomed web server, MPTCP tries to share the ‘pooled access capacity’ fairly. 61 100Mb/s 5 TCPs First 0, then 10 MPTCPs 15 TCPs time [min] throughput per flow [Mb/s] We confirmed in experiments that MPTCP nearly manages to pool the capacity of the two access links. Setup: two 100Mb/s access links, 10ms delay, first 20 flows, then 30.

62. At a multihomed web server, MPTCP tries to share the ‘pooled access capacity’ fairly. 62 100Mb/s 5 TCPs First 0, then 10 MPTCPs 15 TCPs MPTCP makes a collection of links behave like a single large pool of capacity — i.e. if the total capacity is C, and there are n flows, each flow gets throughput C / n.

63. Multipath TCP can pool datacenter networks Instead of using one path for each flow, use many random paths Don’t worry about collisions. Just don’t send (much) traffic on colliding paths

64. Multipath TCP in data centers

66. MPTCP better utilizes the FatTree network

67. MPTCP on EC2 Amazon EC2: infrastructure as a service – We can borrow virtual machines by the hour – These run in Amazon data centers worldwide – We can boot our own kernel A few availability zones have multipath topologies – 2-8 paths available between hosts not on the same machine or in the same rack – Available via ECMP

68. Amazon EC2 Experiment 40 medium CPU instances running MPTCP For 12 hours, we sequentially ran all-to-all iperf cycling through: – TCP – MPTCP (2 and 4 subflows)

69. MPTCP improves performance on EC2 Same Rack

70. Implementing Multipath TCP in the Linux Kernel

71. Linux Kernel MPTCP About 10000 lines of code in the Linux Kernel Initially started by Sébastien Barré Now, 3 actively working on Linux Kernel MPTCP Christoph Paasch Fabien Duchêne Gregory Detal Freely available at http://mptcp.info.ucl.ac.be

72. MPTCP-session creation

73. Application creates regular TCP- sockets

74. MPTCP-session creation

75. The Kernel creates the Meta-socket

76. MPTCP creating new subflows

77. The Kernel handles the different MPTCP subflows

78. MPTCP Performance with apache 100 simultaneous HTTP-Requests, total of 100000

79. MPTCP Performance with apache 100 simultaneous HTTP-Requests, total of 100000

80. MPTCP Performance with apache 100 simultaneous HTTP-Requests, total of 100000

81. MPTCP on multicore architectures Flow-to-core affinity steers all packets from one TCP-flow to the same core. MPTCP has lots of L1/L2 cache-misses because the individual subflows are steered to different CPU-cores

82. MPTCP on multicore architectures

83. Solution: Send all packets from the same MPTCP-session to the same CPU-core Based on Receive-Flow-Steering implementation in Linux (Author: Tom Herbert from Google)

84. MPTCP on multicore architectures

85. Multipath TCP on Mobile Devices

86. MPTCP over WiFi/3G

87. TCP over WiFi/3G

88. MPTCP over WiFi/3G

95. WiFi to 3G handover with Multipath TCP A mobile node may lose its WiFi connection. Regular TCP will break! Some applications support recovering from a broken TCP (HTTP-Header Range) Thanks to the REMOVE_ADDR-option, MPTCP is able to handle this without the need for application support.

96. WiFi to 3G handover with Multipath TCP

103. Related Work Multipath TCP has been proposed many times before – First by Huitema (1995),CMT, pTCP, M-TCP, … You can solve mobility differently – At different layer: Mobile IP, HTTP range – At transport layer: Migrate TCP, SCTP You can deal with datacenter collisions differently – Hedera (Openflow + centralized scheduling)

104. Multipath topologies need multipath transport Multipath TCP can be used by unchanged applications over today’s networks MPTCP moves traffic away from congestion, making a collection of links behave like a single pooled resource

105. Backup Slides

106. Packet-level ECMP in datacenters

107. Maintain a congestion window w r, one window for each path, where r ∊ R ranges over the set of available paths. Increase w r for each ACK on path r, by Decrease w r for each drop on path r, by w r / 2 107 How does MPTCP congestion control work? Design goals 1&3: At any potential bottleneck S that path r might be in, look at the best that a single-path TCP could get, and compare to what I’m getting.

108. Maintain a congestion window w r, one window for each path, where r ∊ R ranges over the set of available paths. Increase w r for each ACK on path r, by Decrease w r for each drop on path r, by w r / 2 108 How does MPTCP congestion control work? Design goal 2: We want to shift traffic away from congestion. To achieve this, we increase windows in proportion to their size.