1. Reliable Transport Protocols Should Forbid Reneging Nasif Ekiz Paul D. Amer, Professor Preethi Natarajan Ertugrul Yilmaz Jon Leighton Abu Rahman sponsored by U.S. Army Research Lab

2. Outline transport layer selective acknowledgments (SACKs) – what are SACKs? – how are SACKs renegable? what is the gain in making selective acks non-renegable? what is the penalty in making selective acks non-renegable? 2 Conclusion: selective acks for reliable transport protocols (e.g., TCP, SCTP) should be non-renegable

3. Transport layer receive buffer 34579111213 ordered data (ACKed) out-of-order data (SACKed) available space receiving application Internet data sender receive buffer

4. Types of acknowledgments For ordered data - cumulative ACK n – bytes [... to n-1 ] (TCP) [RFC 793] – segments [... to n ] (SCTP) [RFC 2960] For out-of-order data - selective ACK (SACK) m-n – bytes [ m to n-1 ] (TCP) [RFC 2018] – segments [ m to n ] (SCTP) [RFC 2960] 4  prevent unnecessary retransmissions during loss recovery  improve throughput when multiple losses in same window

5. Types of ACKs data sender receive buffer data receiver 3 1 1 ACK 1 2 ACK 2 2 4 ACK 2, SACK 4-4 4 5 ACK 2, SACK 4-5 45 6 456 ACK 2, SACK 4-6 ACK 6 3 6345 5

6. What is reneging? 6 [RFC 2018]: “The SACK option is advisory, in that, while it notifies the data sender that the data receiver has received the indicated segments, the data receiver is permitted to later discard data which have been reported in a SACK option.”  discarding SACKed data before delivery to the receiver application (or socket) is “reneging”  TCP and SCTP allow reneging - data sender retains copies of all SACKed data until ACKed

7. data sender receiver buffer 1 1 2 3 4 5 6 456 data receiver ACK 1 ACK 2, SACK 4-4 ACK 2, SACK 4-5 ACK 2 2 4 45 ACK 2, SACK 4-6 7 Reneging example 3 ACK 3 3 ACK 3, SACK 7-7 7 7 OS needs memory and reneges

8. Summary of reneging Reneging : data receiver SACKs data, and later discards it (i.e., SACK information is “advisory”, not a delivery guarantee) Reneging is discouraged but permitted Data sender keeps data in a send buffer until cumulatively ACKed (i.e., cum ack is a guarantee) 8 Special Case for SCTP – out-of-order data already delivered to the application is non-renegable by definition

9. Special case for SCTP: unordered data data sender receive buffer 2 data receiver 1 1 ACK 1 3 3 ACK 1, SACK 3-3 3 5 5 ACK 1, SACK 3-5 9 3 4 4 ACK 1, SACK 3-4 unordered 6 ? ?

10. Special case for SCTP: unordered data data sender receive buffer 1 3 3 3 4 5 1 2 3 4 5 data receiver ACK 1 ACK 1, SACK 3-3 2 32 5 ACK 6 ACK 1, SACK 3-4 ACK 1, SACK 3-5 7 7 ACK 7 10 unordered 3 5 6 3 56 ACK 1, SACK 3-6 unordered

11. We argue that tolerating reneging is wrong Suppose SACKs were a guarantee of delivery, not advisory – All SACKs are non-renegable (NR-SACKs) – Data receiver takes responsibility for all selectively acked data – Data sender can remove NR-SACKed data from send buffer 11 Part I: What is the gain in forbidding selective acks?  always improved send buffer utilization (TCP and SCTP) “Non-renegable selective acks for SCTP” Int'l Conf on Network Protocols (ICNP), Orlando, 10/08  sometimes improved throughput (SCTP) “Throughput analysis of Non-Renegable Selective Acks for SCTP” Computer Communications, 33(16), 10/10

12. Send buffer utilization Send buffer consists of two types of data: – Necessary (renegable) - N – Unnecessary (non-renegable) - U Send buffer utilization = N / (N+U) 12

13. Send buffer utilization (SACK) 13 data sender receive buffer data receiver send buffer 1 1 ACK 1 1 100% 2 21 3 3 ACK 1, SACK 3-3 231 100% 4 4 ACK 1, SACK 3-4 4231 100% 5 5 ACK 1, SACK 3-5 4235 100% send buffer blocking 2 ACK 5 4235 25% 2 4235 75% 4235 25% 4235 50% 6 6 100%

14. Send buffer utilization (NR-SACK) data sender receiver buffer data receiver send buffer 1 1 ACK 1 1 100% ACK 7 2 2 6257 100% 7268 ACK 8 8 8 9 8279 9 100% ACK 9 2 21 100% 4 ACK 1, NR-SACK 3-4 4231 100% 4 5 ACK 1, NR-SACK 3-5 4235 100% 5 5246 6 ACK 1, NR-SACK 3-6 6 3 ACK 1, NR-SACK 3-3 231 100% 3 6257 7 ACK 1, NR-SACK 3-7 7 10 928 100% 10 ACK 10 100% 11 108911 no send buffer blocking

15. NR-SACK ns-2 simulation 15

16. NR-SACK FreeBSD implementation 16

17. Send buffer utilization (ns-2) NR-SACK As traffic load increases, NR-SACKs better utilize send buffer Send Buffer Utilization 17 SACK SACK 64K SACK 32K ∞

18. Send buffer utilization (FreeBSD) 18 Send Buffer Utilization ∞

19. Throughput gains (ns-2) (only for SCTP not TCP) 19 NR-SACKs never do worse than SACKs

20. 20 Changing TCP or SCTP to non-reneging protocol is easy: SACK semantics changed from advisory to permanent if data receiver needs to renege, data receiver MUST RESET the connection (  this is the penalty) Let’s assume transport protocols are designed to forbid data reneging Part II: What is penalty in forbidding selective acks? We argue that tolerating reneging is wrong

21. Suppose reneging occurs 1 in 100,000 TCP (or SCTP) flows Case A (current practice): reneging allowed –99,999 non-reneging connections underutilize send buffer (and for SCTP may achieve lower throughput) –1 reneging connection continues (maybe...) Case B (proposed change): reneging forbidden –99,999 connections have equal or better send buffer utilization (and for SCTP potential greater throughput) –1 reneging connection is RESET Hypothesis: “data reneging rarely if ever occurs in practice” Data reneging has never been studied –Does data reneging happen or not? –If reneging happens, how often? How big is the penalty? answer: depends on how often reneging happens

22. receive buffer 1 3 1 2 3 3 45 5 2 2 Data SenderData Receiver Router OS reneges State of receive buffer Detecting TCP reneging at a router 4 ACK 1, SACK 3-4 3 4 6 3 456 ACK 1, SACK 3-6 7 7 ACK 2, SACK 7-7 ACK 2, SACK 3-6 ? reneging detected

23. Model to detect reneging Current state (C) and new SACK (N) are compared 4 possibilities: SACK 12-17 SACK 12-15 NewCurrent SACK 12-13 SACK 12-17 SACK 22-25 SACK 12-17 SACK 15-20

24. Model to detect reneging Current state (C) New SACK (N) Reneging (R)

25. Model to detect reneging CAIDA* trace TCP flow filter Reneg Detect tshark editcap mergecap ~4600 lines of C code ACK reordering check TCP flows with SACKs reneging? yes or no.pcap *Cooperative Association for Internet Data Analysis

26. Model verification RenegDetect was tested with synthetic TCP flows – created reneging flows with text2pcap – all reneging flows were identified correctly RenegDetect was tested with real TCP flows from CAIDA Internet traces – at first, reneging seemed to occur frequently – on closer inspection, we found that many SACK implementations are incorrect ! “Misbehaviors in TCP SACK Generation” (Ekiz, Rahman, Amer) (ACM SIGCOMM Computer Communication Review, April 2011)

27. Misbehaviors in SACK generation 7 misbehaviors are observed in CAIDA traces We designed TBIT tests to verify SACK generation 27 OS’s tested RenegDetect updated to identify misbehaviors

28. Example TBIT test

29. Incorrect SACK implementations Operating System Misbehavior ABCD EFG FreeBSD 5.3, 5.4 Y Y Linux 2.2.20 (Debian 3)Y Linux 2.4.18 (Red Hat 8)Y Linux 2.4.22 (Fedora 1)Y Linux 2.6.12 (Ubuntu 5.10)Y Linux 2.6.15 (Ubuntu 6.06)Y Linux 2.6.18 (Debian 4)Y OpenBSD 4.2, 4.5, 4.6, 4.7YY OpenSolaris 2008.05YY OpenSolaris 2009.06YY Solaris 10Y Windows 2000YYYYY Windows XPYYYYY Windows Server 2003YYYYY Windows VistaYY Windows Server 2008YY Windows 7YY

30. Event A: TCP flow reneges Hypothesis: We want to design an experiment which rejects H 0 with 95% confidence to conclude Our experiment will observe n TCP flows hoping to NOT find even a single instance of reneging Using MAPLE, n ≥ 299,572 Experiment design – how to “prove” reneging does not happen?

31. Questions ? (thank you) 31

32. TCP Send buffer utilization (SACK) 32 data sender receiver buffer 1 2 3 4 5 6 2 data receiver ACK 1 ACK 1, SACK 3-3 ACK 5 ACK 1, SACK 3-4 ACK 1, SACK 3-5 send buffer 1 21 231 4231 4235 4235 4235 4235 6 100% 75% 50% 25% 100% send buffer blocking 1 3 32 45 3 4 3 45 3 45 3 45 3 45 ACK 6 6

33. TCP Send buffer utilization (NR-SACK) 33 NO send buffer blocking

34. Data reneging in OSes Reneging in Linux (version 2.6.28.7) – tcp_prune_ofo_queue() deletes out-of-order data Reneging in FreeBSD, Mac OS – net.inet.tcp.do_tcpdrain sysctl turns reneging on/off – tcp_drain() deletes out-of-order data

35. Data reneging in Linux

36. 3. Inferring the state of receive buffer TCP Segments with n SACK options Enough space for another SACK option Not enough space for another SACK option n=1~88%0% n=2~11%0% n=30.7%0.20% n=4n/a0.15% Total number of TCP segments780,798 (100%)

37. 3. Inferring the state of receive buffer TCP Segments with n SACK options Enough space for another SACK option Not enough space for another SACK option n=1~88%0% n=2~11%0% n=30.7%0.20% n=4n/a0.15% Total number of TCP segments780,798 (100%)

38. NR-SACK Negotiation INIT – NR-SACKs Supported INIT-ACK – NR-SACKs Supported COOKIE-ACK COOKIE-ECHO SCTP Data Transfer SCTP Association Startup DATA NR-SACKs (cum-ack,gap-ack, nr-gap-ack, dup-TSN) 38

39. NR-SACK Chunk Type = 0x10 Chunk Flags Chunk Length Cumulative TSN Ack Advertised Receiver Window Credit Number of NR-Gap-Ack Blocks = M Number of Gap Ack Blocks = N Gap Ack Block #1 Start Gap Ack Block #1 End Duplicate TSN 1 Duplicate TSN X Gap Ack Block #N Start Gap Ack Block #N End Number of Duplicate TSNs = X Reserved NR-Gap Ack Block #1 Start NR-Gap Ack Block #1 End NR-Gap Ack Block #M Start NR-Gap Ack Block #M End ACK SACK NR-SACK D-SACK 39

40. When is data non-renegable? case 2: multistreaming data sender receiver buffer data receiver SID : Stream Identifier SSN : Stream Sequence Number 40

41. When is data non-renegable? case 2: multistreaming data sender receiver buffer 1 1 data receiver ACK 1 SID: 1 SSN: 1 SID : Stream Identifier SSN : Stream Sequence Number 41

42. When is data non-renegable? case 2: multistreaming data sender receiver buffer 1 1 2 data receiver ACK 1 SID: 1 SSN: 1 SID: 2 SSN: 1 SID : Stream Identifier SSN : Stream Sequence Number 42

43. When is data non-renegable? case 2: multistreaming data sender receiver buffer 1 3 1 2 3 data receiver ACK 1 ACK 1, SACK 3-3 SID: 1 SSN: 1 SID: 2 SSN: 1 SID: 1 SSN: 2 SID : Stream Identifier SSN : Stream Sequence Number 43

44. When is data non-renegable? case 2: multistreaming data sender receiver buffer 1 3 4 1 2 3 4 data receiver ACK 1 ACK 1, SACK 3-3 ACK 1, SACK 3-4 SID: 1 SSN: 1 SID: 2 SSN: 1 SID: 1 SSN: 2 SID: 2 SSN: 2 SID : Stream Identifier SSN : Stream Sequence Number 44

45. When is data non-renegable? case 2: multistreaming data sender receiver buffer 1 3 4 1 2 3 4 5 data receiver ACK 1 ACK 1, SACK 3-3 ACK 1, SACK 3-4 SID: 1 SSN: 1 SID: 2 SSN: 1 SID: 1 SSN: 2 SID: 2 SSN: 2 SID: 1 SSN: 3 SID : Stream Identifier SSN : Stream Sequence Number 45 ? ?

46. When is data non-renegable? case 2: multistreaming data sender receiver buffer 1 3 4 4 5 1 2 3 4 5 data receiver ACK 1 ACK 1, SACK 3-3 ACK 1, SACK 3-4 ACK 1, SACK 3-5 SID: 1 SSN: 1 SID: 2 SSN: 1 SID: 1 SSN: 2 SID: 2 SSN: 2 SID: 1 SSN: 3 SID : Stream Identifier SSN : Stream Sequence Number 46

47. When is data non-renegable? case 2: multistreaming data sender receiver buffer 1 3 4 4 4 5 6 1 2 3 4 5 6 data receiver ACK 1 ACK 1, SACK 3-3 ACK 1, SACK 3-4 ACK 1, SACK 3-5 ACK 1, SACK 3-6 SID: 1 SSN: 1 4 6 SID: 2 SSN: 1 SID: 1 SSN: 2 SID: 2 SSN: 2 SID: 1 SSN: 3 SID: 2 SSN: 3 SID : Stream Identifier SSN : Stream Sequence Number 47

48. When is data non-renegable? case 2: multistreaming data sender receiver buffer 1 3 4 4 4 5 6 1 2 3 4 5 6 2 42 6 data receiver ACK 1 ACK 1, SACK 3-3 ACK 6 ACK 1, SACK 3-4 ACK 1, SACK 3-5 ACK 1, SACK 3-6 SID: 1 SSN: 1 4 6 SID: 2 SSN: 1 SID: 1 SSN: 2 SID: 2 SSN: 2 SID: 1 SSN: 3 SID: 2 SSN: 3 SID: 2 SSN: 1 SID : Stream Identifier SSN : Stream Sequence Number 48

49. When is data non-renegable? case 2: multistreaming data sender receiver buffer 1 3 4 4 4 5 6 1 2 3 4 5 6 7 2 42 6 7 data receiver ACK 1 ACK 1, SACK 3-3 ACK 6 ACK 1, SACK 3-4 ACK 1, SACK 3-5 ACK 1, SACK 3-6 SID: 1 SSN: 1 4 6 ACK 7 SID: 2 SSN: 1 SID: 1 SSN: 2 SID: 2 SSN: 2 SID: 1 SSN: 3 SID: 2 SSN: 3 SID: 2 SSN: 1 SID: 1 SSN: 4 SID : Stream Identifier SSN : Stream Sequence Number 49

50. When is data non-renegable? Case 3: OS guarantee Data Sender Receiver Buffer Data Receiver 50

51. When is data non-renegable? Case 3: OS guarantee Data Sender Receiver Buffer 1 1 Data Receiver ACK 1 51

52. When is data non-renegable? Case 3: OS guarantee Data Sender Receiver Buffer 1 1 2 Data Receiver ACK 1 52

53. When is data non-renegable? Case 3: OS guarantee Data Sender Receiver Buffer 1 3 1 2 3 Data Receiver ACK 1 ACK 1, SACK 3-3 53

54. When is data non-renegable? Case 3: OS guarantee Data Sender Receiver Buffer 1 3 3 4 1 2 3 4 Data Receiver ACK 1 ACK 1, SACK 3-3 ACK 1, SACK 3-4 54

55. When is data non-renegable? Case 3: OS guarantee Data Sender Receiver Buffer 1 3 3 3 4 45 1 2 3 4 5 Data Receiver ACK 1 ACK 1, SACK 3-3 ACK 1, SACK 3-4 ACK 1, SACK 3-5 55

56. When is data non-renegable? Case 3: OS guarantee Data Sender Receiver Buffer 1 3 3 3 3 4 4 4 5 5 1 2 3 4 5 6 6 Data Receiver ACK 1 ACK 1, SACK 3-3 ACK 1, SACK 3-4 ACK 1, SACK 3-5 ACK 1, SACK 3-6 3 456 56

57. When is data non-renegable? Case 3: OS guarantee Data Sender Receiver Buffer 1 3 3 3 3 4 4 4 5 5 1 2 3 4 5 6 2 2 6 Data Receiver ACK 1 ACK 1, SACK 3-3 ACK 6 ACK 1, SACK 3-4 ACK 1, SACK 3-5 ACK 1, SACK 3-6 3 456 3 456 57

58. When is data non-renegable? Case 3: OS guarantee Data Sender Receiver Buffer 1 3 3 3 3 4 4 4 5 5 1 2 3 4 5 6 7 2 6 7 Data Receiver ACK 1 ACK 1, SACK 3-3 ACK 6 ACK 1, SACK 3-4 ACK 1, SACK 3-5 ACK 1, SACK 3-6 3 456 ACK 7 23 456 58

59. When is data non-renegable? Case 3: OS guarantee Data Sender Receiver Buffer 1 3 3 3 3 4 4 4 5 5 1 2 3 4 5 6 7 2 6 7 Data Receiver ACK 1 ACK 1, SACK 3-3 ACK 6 ACK 1, SACK 3-4 ACK 1, SACK 3-5 ACK 1, SACK 3-6 3 456 ACK 7 * * OS guarantees not to renege 23 456 59