1. Detection of Routing Loops and Analysis of Its Causes
Sue Moon
Dept. of Computer Science
KAIST
Joint work with Urs Hengartner, Ashwin Sridharan,
Richard Mortier, Christophe Diot

2. Link Utilization Internet backbone link Routing loop causes
increase by 25%!
Why do we know it is a loop?
What exactly causes increase?
Why is this bad? (useful traffic might be dropped)

3. Overview
Routing protocols have much impact on the performance of the network
How do we detect them?
How often do loops occur?
How do they impact loss and delay?
Analyze causes of loops
What causes them?

4. Possible Causes of Routing Loops
Persistent routing loops
E.g., due to misconfiguration.
Loops can last hours if undetected.
Transient routing loops
Routing state is dynamic.
Inconsistencies in routing state can cause loops.
Inconsistencies should disappear within seconds/minutes.
Expectation: Loops last seconds/minutes.

5. How Can Transient Routing Loop Occur?

6. Detection of “Loops” in Packet Traces
Detect replicas in a packet trace
Packets with exact same header but for TTL,CRC
TTL difference: 2 or larger
Set of replicas = Packet Loop
Set of packet loops associated with a routing event = Routing Loop

7. Traces Backbone traces NYC and SJ links from Nov. 8th, 2001
NYC links from Oct. 9th, 2002

8. Packet Traces Trace Length Avg BW Packets Looped Total (106)
(hours)
(Mbps)
Total (106)
Backbone 1 24 1 50
4.839%
Backbone 2
7.5
243
1 677
0.118%
Backbone 3 11
2.2 20
1.687%
Backbone 4
107
1350
0.026%
…loops occur in bursts and can affect up to 25% of packets!
Highlight that we measured looped packets with our algorithm
On average, loops do not affect much traffic, but…

9. Observations about Packet Loops
General Observations
Loop size: # of nodes involved in packet loop
Number of replicas in packet loop
Properties of packet loops
Packet types
Duration
Of packet loops in packets

10. Loop Size
Loop size: value by which TTL field in packet loops gets decremented.
Figure 2

11. Packet Loop Length How often does a packet show up before it expires?
Figure 3

12. Traffic Types
Different types of Internet traffic.
Routers are oblivious to type of traffic.
Expectation: Traffic types of packet loops streams are distributed similarly as traffic types of overall traffic.

13. Traffic Types (Backbone 2)
By protocol
TCP: 10% (93%)
UDP: 16% (6%)
ICMP 77% (0.3%)
TCP Flags
SYN: 51% (5%)
ACK: 73% (97%)
RST: 13% (1.5%)
FIN: 8% (4%)
Most traffic has ack bit set
TCP does not add up to 100%
After first figure: mention that reasons will be discussed soon

14. Reasons for Increases TCP SYN traffic. UDP traffic. ICMP traffic.
TCP is connection oriented.
End point tries to open connection, sends SYN packet.
SYN packet loops and expires, no other packets are sent.
UDP traffic.
UDP is connectionless, no feedback from receiver.
Sending application is oblivious of loop.
ICMP traffic.
Caused by traceroute/ping applications.
People are exploring loop.
Observations confirm presence of loops!

15. Out-Of-Order Delivery

16. Causes of Packet Loops: BGP
customer
AS 2 C
AS 1 A B D

17. Matching BGP Updates
Any advertisement of the longest prefix?
Temporal vicinity of 2 minutes to packet loops?
Change in next hop or AS path?

18. Causes of Loops: ISIS R1 1 1 R2 1 R3 1 1 R4 4 R5

19. Time-Line at Nodes R2 and R3
Failure Detection
LSP generation
Shortest Path
Computation
LSP Flooding
FIB Update
LSP Arrival
Shortest Path
Computation
FIB Update

20. If forwarding path changed and it is within temporal vicinity of loop
Matching ISIS Updates
Upon receipt of an LSP, compute the shortest path from the observation node to the egress router
If forwarding path changed and it is within temporal vicinity of loop
see if the observation node lies on the shortest path before or after the change

21. BGP Update Matches Trace % transient % persistent (BGP)
% persistent (no BGP)
Total
NYC-20
40.1
50.8
90.8
NYC-21
80.2
7.5
87.9
NYC-23
3.3
NYC-22
18.8
80.6
99.4
NYC-24
70.0
NYC-25
43.7
15.5
59.2

22. Factors to Varying Success
Persistent Loops
Events occurred before trace collection
BGP changes external to Sprint
Comparison with RouteView updates: increase in matches
Geographical distribution of loop destinations
Measurement PoP not involved in route changes
Avg # of ASes traversed: longest for NYC-23

23. Conclusions
Loops can be detected and analyzed
Loops are not uncommon
Most are due to BGP updates
BGP changes farther away from the observations point may not be identified

24. BACKUP SLIDE

25. CDF of Number of Replicas

26. CDF of Inter-Replica Spacing Time

27. Packet Types of All Traffic

28. Packet Types of Loops

29. Destination Addresses of Loops
Backbone 1
Regional 2

30. CDF of Replica Stream Duration in Time

31. CDF of Routing Loop Duration in Time

32. Overview Types and causes behind routing loops
Transient - part of normal routing protocol operation
Persistent - “long-lasting”, manual intervention required
Detection of routing loops in packet traces
Detection algorithm
Observations about the routing loops
Analysis of performance impact
Loss, delay, out-of-order delivery
On-line detection algorithm
Summary

33. Fraction of Packets in Loops
Backbone 1
Backbone 4

34. Construction of a Typical End-To-End Path
10 hops
in the Backbone
Regional to Backbone
DSL/LAN/Cable/Phone

35. Estimate of End-to-End Loss
Assume:
No loss on the access link due to routing loops
Losses are independence between links
Estimate:
Lr: from Regional traces
Lb: from Backbone traces but for Backbone 4
1 - (1- Lr)2(1- Lb)10 = ~ 0.025
Implications on SLA??

36. Delay Due to Routing Loops
Average delays range from 50 to 500ms.

37. Out-Of-Order Delivery

38. Causes of Loop
Backbone 2,3, and 6 possibly had configuration problems.
BGP updates - transient
IGP - transient and persistent

39. Overview Types and causes behind routing loops
Transient - part of normal routing protocol operation
Persistent - “long-lasting”, manual intervention required
Detection of routing loops in packet traces
Detection algorithm
Observations about the routing loops
Analysis of performance impact
Loss, delay, out-of-order delivery
On-line detection algorithm
Summary & Future Work

40. To Detect a Loop On-line
Focus on persistent loops
Questions:
More focus on persistent loops
How much traffic is affected? -> alarm
What prefix is affected? -> warning

41. On-Line Detection Algorithm
How many packets to /24 get looped? 100
WARNING
How many looped packets / million? 5%
How long (in millions) did it last? 10 millions
ALARM
By the time an alarm is raised, warnings are raised and help debugging the system
Fixed memory and computation complexity

42. Validation of On-Line Algorithm

43. Summary
Impact of routing on performance has been analyzed in terms of loss and delay.
Per-link loss varies greatly.
Excluding “outliers”, end-to-end loss of 0.3% is unavoidable.
For a small number of packets that escape the loops, 50 ~ 500 msec delay is added on the average.
On-line detection algorithm
In conjunction with routing protocol monitoring, it will help detect and fix persistent loops.

44. More work needed to determined causes behind routing loops
Future Work
More work needed to determined causes behind routing loops
Correlate with BGP/IS-IS updates
Address hijacking
Wrong aggregation
Origin misconfiguration
Export misconfiguration
Integration with existing monitoring tools

45. Backup Slides

46. Superbowl Sunday, 2/3/2002
New Traces, not yet analyzed in detail. Only the on-line detection algorithm was used to collect alarms and warnings.

47. Superbowl Sunday, 2/3/2002
New Traces, not yet analyzed in detail. Only the on-line detection algorithm was used to collect alarms and warnings.

48. What Next? Alarms and warnings
How to extract just enough info to be useful
How to relate it with BGP/IS-IS update info
How to integrate with management/monitoring infrastructure