1. Multipath tracing with Paris Traceroute
Brice Augustin
Timur Friedman and Renata teixeira
Laboratoire LIP6 – CNRS
Université Pierre et Marie Curie – Paris 6
In this talk I will describe an extension to Paris traceroute to trace all paths under load balancing

2. Introduction
Traceroute measures a path between two hosts in an IP network
It is widely used
Diagnosis of connectivity problems [Paxson 1999]
Inferrence of network properties [Faloutsos 1999]
Internet maps [Rocketfuel 2002]
Traceroute is a tool to measure the IP-level path from a source to a destination in the internet.
It is widely used, from the diagnosis of network problems to the assemblage of internet maps.

3. Contributions
Previously [IMC 2006]: identified traceroute deficiencies on load balanced paths
Measured paths are:
Inaccurate (fixed)
Incomplete (this remained to be done)
Built a new traceroute: Paris traceroute
Added a probing algorithm to trace all paths
In a previous work, we identified two severe deficiencies with classic traceroute under load balancing. The first problem was that traceroute measures inaccurate paths. This leads to many measurement artifacts like loops, cycles, false links. We fixed this problem by building a tool called Paris traceroute which we presented in a paper to IMC The second problem, which we had not addressed, was that traceroute also measures incomplete paths, because there is no longer a single path from source to destination. In this work, we present a probing strategy to detect all paths under load balancing.

4. Traceroute under load balancing
Actual topology: A C E
Dst
Src L
TTL = 2 B D
TTL = 3
Inferred topology: A C
Now lets see what happens when the path traverses a load balancer L with two load-balanced paths. Imagine that at TTL 2, traceroute sends a probe which takes the upper path and discovers A. Then at TTL 3 the probe takes the lower path and discovers B. This simple example illustrates the two deficiencies of traceroute: first, the path is inaccurate since there is a false link. The link A – D does not exist in the real topology. Second, the path is incomplete, since traceroute missed nodes B and C, as well as their related links.
Per-packet, per-flow? Happens even under per-flow?
False link
Src E
Dst L B D
Missing nodes and links

5. Tracing a single path [IMC 2006]
Solves the problem with per-flow load balancing
Probes have the same flow identifier
Port 1
Port 1 A C
Port 1
We addressed the problem of inaccuracy in a paper in IMC2006. Paris traceroute maintains the flow identifier of the probes, in order to take the same path under per-flow load balancing. There is still the problem of per-packet load balancers, but we found that they are infrequent in our traces. Now we can trace a clean path, but we can only trace one path at a time. Multiple traceroutes towards different destinations, but all traversing the same load balancer, will eventually discover all the paths. This would be the case for Internet cartography applications, which trace towards thousands, or hundreds of thousands destinations. So why would be need a tool to enumerate the paths? Simply because Internet cartography is NOT the most common way people use traceroute. Usually people use it to trace to a single destination. If you allow to trace a single path one at a time, you may diagnose the wrong path, for instance. E
Dst
Src L
TTL = 2
TTL = 3 B D
TTL = 1

6. Tracing all the paths Multipath Detection Algorithm (MDA)
Change the probing strategy
At each hop:
Send probes with a different flow identifier
Send enough probes to enumerate all interfaces with a high degree of confidence
Classify load balancers: per-flow or per-packet
That’s why we place our work in the context of tracing from a single source to a single destination. We suggest a new goal for route tracing in this context and propose an adaptive probing strategy. Instead of sending a fixed number of probes per hop as does classic traceroute, we adapt this number according what has been discovered just before. Instead of expecting a single path, we explore the set of possible flow identifiers to enumerate the nexthops of load balancers with a high degree of confidence. Finally, we enrich the output with valuable information on the type of load balancing encountered. Let us now describe how this algorithm works with a simple example.

7. MDA - interfaces enumeration
Steps:
Port 11
Port 8
Port 2
Port 4
Port 1
Nexthops of L? ? A
Suppose 2 interfaces
Send 6 probes through L
Src L
Port 9
Port 10
Port 7
Responses from 2 interfaces
Port 5
Port 6
Port 3
Suppose actually 3 interfaces B ?
Imagine we want to trace the paths in our previous example, with a load balancer L and two load-balanced paths. At a given step of the discovery, we will have discovered L. We now want to enumerate with a high degree of confidence, the nexthops of L. That is to say, all the interfaces that are directly reachable through L.
In the left part you see the topology discovered by PT. In the right part, the progression of the algorithm.
We know that there is at least 1 interface out there, even without sending a probe. So we first suppose that L has two nexthops. For a confidence degree of 95%, we have to send 6 probes to rule out this hypothesis. Thus we send those 6 probes, each with a different flow identifier (different port number). The first one may take the upper path, and discover A. The second one takes the upper path one more time, but the third one takes the lower paths and reveals B.
Once we have received responses from the 6 probes, we will have discovered 2 interfaces. Our hypothesis is confirmed. We then suppose that there exists a third interface, and that we have not yet sent enough probes to see it.
Here again with a 95% degree of confidence, we send 11 probes through L. None of them reveals the third interface. We can rule out our hypothesis, stop probing, and say with a high degree of confidence that L has only 2 nexthops, A and B.
Send 5 more probes through L
TTL = 1
No third interface
Stop probing ?
Parameters:
95% confidence degree

8. MDA – hop by hop progression
Port 12
Port 11
Port 4
Port 8
Port 2
Port 1
Port 11
Port 8 C ?
Port 2
Port 4
Port 1 ? A C
Src ? L E
Dst
TTL = 2
Next steps: ? B D
We have to repeat the previous algorithm for each interface we discover, in order to progress in the path and explore all its branches.
Now lets see how one can enumerate the nexthops of A. We can apply the previous algorithm, but we first have to find enough flow identifiers that traverse A. In practice, we have already discovered several flow identifiers during the probing of L. The others go to B. But this is not enough, as we need 6 probes to rule out the presence of 2 nexthops. A first part consists in testing additional flow identifiers and select those which traverse A and NOT B. Then we use those identifiers to enumerate the nexthops of A. As usual, we suppose there are 2 nexthops, but receive responses from a single interface C. We rule out the hypothesis and conclude that A is not a load balancer.
Nexthops of A?
Find enough probes traversing A
Enumerate interfaces

9. MDA - load balancer classification
Port 1
Port 1
Steps:
Port 1
Port 1
Port 1
Port 1 ? A
Per-packet or
per-flow
Src L
Port 1
Port 1 B
A first part of the algorithm is the interface enumeration. But once we have discovered a load balancer, we also have to classify it. Indeed, varying the flow identifier of the probes reveals both per-packet and per-flow load balancers. We first make the hypothesis that L uses the per-packet technique. We can rule out it by sending a few additional probes though the load balancer, this time having the SAME flow identifier.
If we receive responses from a single interface, we conclude that this is a per-flow load balancer, because we have verified that the flow control has an impact on the path the packets take. Otherwise, we conclude that it is a per-packet one.
Suppose per-packet
Send 6 identical probes
Responses from 2 interfaces
Suppose per-packet
Send 6 identical probes
Responses from 1 interface
TTL = 1

10. Measurement infrastructure
5000 reachable destinations
Measurements
1 round takes 100 minutes
Paris traceroute
Classic traceroute
Paris traceroute
Classic traceroute
We now have a tool able to trace the paths to a destination. We tested it by running measurements from a single source located in Paris, towards 5000 randomly selected destinations. For each destinations, we successively launch a Paris traceroute using our adaptive probing, followed by a classic traceroute. We take the point of view of an end-user who wants to discover the path from his machine, towards, say a web server. Our goal is to compare the paths discovered by both implementations of traceroute.
This is a very limited evaluation of our technique. We now have more results, and tested it from multiple vantage points.
Paris
Paris traceroute
INTERNET
Source

11. Paris traceroute performance20
Discovered (+) + 15
Missed (-) * 10
Paris traceroute finds up to 16 more interfaces
201 cases where Paris traceroute missed 1 or more interfaces
Number of interfaces 5
Paths with load balancing:
per-flow: 30%
per-packet: 2%
This plot shows the number of interfaces discovered or missed by Paris traceroute, in comparison with classic traceroute.
The X axis represents each of the 1700 destinations where we detected load balancing. In the Y axis, in red we plot the number of interfaces that Paris traceroute discovered, but that classic traceroute missed. In the majority of cases Paris traceroute finds one more interface than classic traceroute. We also found cases where it finds much more interfaces, up to 16.
In red we see the cases where PT fails at finding all the interfaces, which shows how difficult it is to do accurate internet measurements. Routing changes between the measurements can explain that. Routers that limit the number of ICMP responses they send.
Paris traceroute finds one more interface than classic traceroute
- 5
-10
Destination

12. The effects of load balancing
1000
UDP probing
ICMP probing
100
Most hops have a small number of responding interfaces
ICMP probing usually discovers fewer interfaces
# distinct hops
Some hops have up to 16 interfaces 10
This figure is a kind of « raw view » of the effects of load balancing. When there is load balancing there are several responding interfaces at each traceroute hop. This figure plots the number of responding interfaces that Paris traceroute sees at each hop. In the X axis, you have the number of interfaces per hop (we only show hops with multiple interfaces).
The Y axis represents the number of hops with the given number of interfaces. With UDP probing, in blue, most hops have a small number of interfaces. But some hops of up to 16 interfaces. We observed such case at a peering point between a tier-1 and a brazilian autonomous system. Something surprising: ICMP probing also reveals multiple interfaces per hop, but usually fewer than UDP probing. We found that most of per-flow load balancers also use some fields in the ICMP header to define a flow. 1
# interfaces per hop

13. Conclusion
It no longer makes sense to trace a single route from source to destination in the internet
Classic traceroute discovers incomplete paths
Paris traceroute and the MDA trace all paths to a destination with a high level of confidence
This work shows that it no longer makes sense to trace a single route from a source to a destination in the internet. With Paris traceroute in the MDA, we are now able to trace all the load balanced paths to a destination with a high degree of confidence.

14. Perspectives
Measure “native path diversity” in the internet (submitted to IMC2007)
Handle some probing subtleties
Simple extensions to detect:
Per-destination load balancing
Uneven load balancing
Return path diversity
We are currently working on the following things. First, We used the MDA to measure the path diversity provided by load balancers. We performed measurements from 15 sources towards 69 thousands destinations and found that the 39 of the paths were affected by per-flow load balancing.
We also want to improve the probing algorithm to handle some subtleties, for instance in the presence of routers that don’t respond to probes.
Simple extensions to the MDA enables the detection of per-destination load balancers. Instead of varying the port numbers, we vary the destination address in a small prefix block. Our experiments show that per-destination load balancers are even more frequent than per-flow.
We also observed cases where load balancers split traffic unevenly. We believe that it would be easy to detect that with a high degree of confidence.
Finally, we also propose a method to control the flow on the return path.

15. More information

16. Contributions
Identified traceroute deficiencies on load balanced paths
Measured paths are inaccurate [IMC 2006]
Measured paths are incomplete
Built a new traceroute: Paris traceroute
Added a probing algorithm to trace all paths
In a previous work, we identified two severe deficiencies with classic traceroute under load balancing. The first problem was that traceroute measures inaccurate paths. This leads to many measurement artifacts like loops, cycles, false links. We fixed this problem by building a tool called Paris traceroute which we presented in a paper to IMC The second problem, which we had not addressed, was that traceroute also measures incomplete paths, because there is no longer a single path from source to destination. In this work, we present a probing strategy to detect all paths under load balancing.

17. Traceroute under normal case
Actual topology:
Src A B
Dst
TTL = 1
TTL = 2
TTL = 3
Inferred topology:
Let us now look at how traceroute works under a normal case, with a source on the left, a destination on the right, and a single path with two routers between them. Traceroute sends a few probes with a low TTL value to discover intermediate hops. It begins with a TTL of 1, which is dropped by router A. Thus traceroute discovers A. It then increments the TTL to discover B, and so on until it reaches the destination.
This is the normal case with a single path, and here traceroute behaves correctly.
Src A B
Dst

18. Tracing all the paths Multipath Detection Algorithm (MDA)
Change the probing strategy:
Adapt the number of probes at each hop
Enumerate the nexthops of load balancers with a high degree of confidence
Classify load balancers
That’s why we place our work in the context of tracing from a single source to a single destination. We suggest a new goal for route tracing in this context and propose an adaptive probing strategy. Instead of sending a fixed number of probes per hop as does classic traceroute, we adapt this number according what has been discovered just before. Instead of expecting a single path, we explore the set of possible flow identifiers to enumerate the nexthops of load balancers with a high degree of confidence. Finally, we enrich the output with valuable information on the type of load balancing encountered. Let us now describe how this algorithm works with a simple example.

19. Tracing a single path [IMC 2006]
Solves the problem with per-flow load balancing
Probes have the same flow identifier
Add another dest to show that we discover the other path ?
Port 1
Port 1 A C
Port 1
We addressed the problem of inaccuracy in a paper in IMC2006. Paris traceroute maintains the flow identifier of the probes, in order to take the same path under per-flow load balancing. There is still the problem of per-packet load balancers, but we found that they are infrequent in our traces. Now we can trace a clean path, but we can only trace one path at a time. Multiple traceroutes towards different destinations, but all traversing the same load balancer, will eventually discover all the paths. This would be the case for Internet cartography applications, which trace towards thousands, or hundreds of thousands destinations. So why would be need a tool to enumerate the paths? Simply because Internet cartography is NOT the most common way people use traceroute. Usually people use it to trace to a single destination. If you allow to trace a single path one at a time, you may diagnose the wrong path, for instance. E
Dst
Src L
TTL = 2
TTL = 3 B D
TTL = 1
Show a problem on link B – D. can’t see it if trace only upper path

20. The effects of load balancing
1000
UDP probing
ICMP probing
100
Most hops have a small number of reachable interfaces
ICMP probing usually discovers fewer interfaces
# distinct hops
Some hops have up to 16 interfaces 10 1
# interfaces per hop

21. Paris traceroute performance20
Discovered (+) + 15
Missed (-) * 10
Paris traceroute finds up to 16 more interfaces
201 cases where Paris traceroute missed 1 or more interfaces
Number of interfaces 5
Paris traceroute finds one more interface than classic traceroute
This plot shows the number of interfaces discovered or missed by Paris traceroute, in comparison with classic traceroute.
The X axis represents each of the 1700 destinations where we detected load balancing. In the Y axis, in red we plot the number of interfaces that Paris traceroute discovered, but that classic traceroute missed. In the majority of cases classic traceroute misses a single interface. We also found cases where it misses much more interfaces, up to 16.
- 5
-10
Destination