1. 1 Debugging Path MTU Discovery Failures - Techniques and Results Internet2 Member Meeting September 21, 2005 Matthew Luckie, Kenjiro Cho, Bill Owens

2. 2 Large Packets == GOOD TCP performance interrupts Jumbo Packets (9000 bytes) == BETTER Fragmentation == REALLY BAD Background - Why Do We Care?

3. 3 If we want to avoid fragmentation, we must know the size of the largest packet the path will permit. This is especially critical when we go above 1500 bytes. Simple concept - send using your local MTU, set the Don’t Fragment bit, and watch for errors. Optimizations - have the routers tell you what their next-hop MTU is, to save searching. Enter PMTUD

4. 4 Most of the time this works... but only because most of the world is limited to 1500 bytes. ICMP packets are not first-class citizens: firewalled disabled rate-limited The general consensus is that PMTUD fails in too many paths. So how do we deploy jumbo hosts? PMTUD and Jumbo

5. 5 ICMP Black Hole no feedback from the path MTU Mismatches mixed configuration, or a low-MTU device ‘hidden’ between two L3 devices No Suggested MTU not fatal, but slows down the process Private IP Addresses error messages blocked by ingress filters PMTUD Failure Modes

6. 6 SYN ACK SYN-ACK HTTP GET HTTP Data Frag Reqd. HTTP Data Frag Reqd. TCP 3-Way Handshake SrcDstFWGW 14801500

7. 7 A. Medina, M. Allman, S. Floyd. (2005) Measuring the Evolution of Transport Protocols in the Internet 17% of 81776 targets failed at PMTUD 35% of 500 ‘popular’ websites failed Did not find –any– which tried smaller packets Assumed middle-boxes filtering ICMP 41% of targets did PMTUD 30% did not attempt PMTUD The Reverse-Path Problem

8. 8 Topology NYSERNet - GigE 9000 byte connected server in NYC Abilene - GigE 9000 byte connected server in Chicago Targets - NLANR Active Measurement Project (AMP) hosts, 1500 byte connected Testbed

9. 9 Software - scamper originally developed to probe IPv6 paths and find tunnels high-speed parallel traceroute with advanced MTU determination capabilities IPv4 and IPv6 capable http://www.wand.net.nz/scamper/ Testbed

10. 10 Initial Path Determination begin with a traceroute using small packets infer the forward path determine which hops will send ICMP feedback, so we can distinguish between all packets being silently discarded, and just large ones PMTUD Failure Diagnostics

11. 11 Next-Hop MTU Search internal table of likely MTU values start with local MTU skip down to 1500 on failure if not 1500, skip down to 1454 to check for the lower bound, set when feedback is obtained if lower bound is a table value and upper bound is below the next largest, try +1 byte to confirm if the next table value is below the upper bound, try it if between table values, fall back to a binary search PMTUD Failure Diagnostics

12. 12 Inferring MTU without feedback perform next-hop MTU search to determine the PMTU TTL search to find the hop where the MTU decreases silently can locate the problem, but not the reason - ICMP disabled, filtered, blocked by RFC 1918? PMTUD Failure Diagnostics

13. 13 SrcDstR1*R3 1500 1480 1500 1. TTL 255, Size 1500 2. TTL 255, Size 1500 3. TTL 255, Size 1454 ICMP Port Unreach 4. 5. TTL 255, Size 1480 ICMP Port Unreach 6. 7. TTL 255, Size 1492 8. TTL 255, Size 1492 9. TTL 255, Size 1481 10. TTL 255, Size 1481

14. 14 SrcDstR1*R3 1500 14801500 11. 12. TTL 3, Size 1500 13. TTL 3, Size 1500 Probe: TTL 1, Size 1500 Reply: ICMP Time Exceeded

15. 15 Inferring MTU with invalid feedback packet too big message with no next-hop MTU, or larger than the probe use the bad message to speed up finding the actual MTU, working down through the table PMTUD Failure Diagnostics

16. 16 Test runs on April 28, 2005 147 targets 30% PMTUD failures, in four groups: no ICMP to packet too big message incorrect packet too big target machine with MTU mismatch Results

17. 17 Results NYSERNetnms1-chinIntersectionTotal Target Count:147 - Reachable:136 (93%)134 (91%)134- Failures:41 (30%)40 (30%)25- No ICMP Messages:6 (6)5 (5)4 (4)7 unique No PTB Messages:26 (17)27 (18)13 (13)22 unique Incorrect PTB:2 (2) 2 unique Target MTU Mismatch:7 (7)6 (6) 2 unique The number on the left is the number of AMP targets on a path with this failure mode. The number in brackets is the number of unique failure points.

18. 18 No ICMP Messages 7 failures (6 at 1500, 1 at 1536) two due to ingress filters: one originated ICMP with 127.0.0.1 one originated with RFC 1918 address another due to an “Internet-Free Zone” another caused by a routing issue - routers in the forward path had no route back to our test source Results

19. 19 No Packet Too Big Messages 22 hops sent TTL Expired, but no PTB 16 at 1500 4 at 4472, 2 at 4540, 1 at 4470, 1 at 2002 22 distinct problem locations obtained technical diagnosis for seven, two were fixed before an answer could be obtained two main causes: no IP unreachables, but does not suppress TTL Expired MTU mismatches Results

20. 20 Incorrect Packet Too Big Messages two hops at one location sent a message with incorrect next- hop MTU we sent 9000 PTB said to send 4586 actually the path could only handle 4472 Results

21. 21 Target MTU Mismatches seven AMP machines had 1500 byte interfaces, but were on a router interface that forwarded packets larger than 1500 these machines shouldn’t receive packets larger than 1500... but tcpdump verified that two did: one at 1506, another at 2016 Results

22. 22 Specific Cases Circuit Cross Connect - hidden 4470 limit invalid next-hop MTU - 4458 limit, 4470 message VoIP phone with integrated switch different v4 and v6 MTUs Results

23. 23 Matt Zekauskas collected nms1-chin (Chicago) dataset WIDE funded development of scamper for a year NLANR/NMA is funded by NSF ANI-0129677 Acknowledgements

24. 24 RFC 1191 - Path MTU Discovery RFC 1435 - IESG Advice from Experience with Path MTU Discovery RFC 1981 - Path MTU Discovery for IP version 6 RFC 2923 - TCP Problems with Path MTU Discovery Fragmentation Considered Harmful http://www.hpl.hp.com/techreports/Compaq-DEC/WRL-87-3.html Fragmentation Considered Very Harmful http://www.psc.edu/~jheffner/drafts/draft-mathis-frag-harmful-XX.html PMTUD Working Group http://www.psc.edu/~mathis/MTU/pmtud/ References