Detection of Routing Loops and Analysis of Its Causes Sue Moon Dept. of Computer Science KAIST Joint work with Urs Hengartner, Ashwin Sridharan, Richard Mortier, Christophe Diot

Link Utilization Internet backbone link Routing loop causes increase by 25%! Why do we know it is a loop? What exactly causes increase? Why is this bad? (useful traffic might be dropped)

Overview Routing protocols have much impact on the performance of the network How do we detect them? How often do loops occur? How do they impact loss and delay? Analyze causes of loops What causes them?

Possible Causes of Routing Loops Persistent routing loops E.g., due to misconfiguration. Loops can last hours if undetected. Transient routing loops Routing state is dynamic. Inconsistencies in routing state can cause loops. Inconsistencies should disappear within seconds/minutes. Expectation: Loops last seconds/minutes.

How Can Transient Routing Loop Occur?

Detection of “Loops” in Packet Traces Detect replicas in a packet trace Packets with exact same header but for TTL,CRC TTL difference: 2 or larger Set of replicas = Packet Loop Set of packet loops associated with a routing event = Routing Loop

Traces Backbone traces NYC and SJ links from Nov. 8th, 2001 NYC links from Oct. 9th, 2002

Packet Traces Trace Length Avg BW Packets Looped Total (106) (hours) (Mbps) Total (106) Backbone 1 24 1 50 4.839% Backbone 2 7.5 243 1 677 0.118% Backbone 3 11 2.2 20 1.687% Backbone 4 107 1350 0.026% …loops occur in bursts and can affect up to 25% of packets! Highlight that we measured looped packets with our algorithm On average, loops do not affect much traffic, but…

Observations about Packet Loops General Observations Loop size: # of nodes involved in packet loop Number of replicas in packet loop Properties of packet loops Packet types Duration Of packet loops in packets

Loop Size Loop size: value by which TTL field in packet loops gets decremented. Figure 2

Packet Loop Length How often does a packet show up before it expires? Figure 3

Traffic Types Different types of Internet traffic. Routers are oblivious to type of traffic. Expectation: Traffic types of packet loops streams are distributed similarly as traffic types of overall traffic.

Traffic Types (Backbone 2) By protocol TCP: 10% (93%) UDP: 16% (6%) ICMP 77% (0.3%) TCP Flags SYN: 51% (5%) ACK: 73% (97%) RST: 13% (1.5%) FIN: 8% (4%) Most traffic has ack bit set TCP does not add up to 100% After first figure: mention that reasons will be discussed soon

Reasons for Increases TCP SYN traffic. UDP traffic. ICMP traffic. TCP is connection oriented. End point tries to open connection, sends SYN packet. SYN packet loops and expires, no other packets are sent. UDP traffic. UDP is connectionless, no feedback from receiver. Sending application is oblivious of loop. ICMP traffic. Caused by traceroute/ping applications. People are exploring loop. Observations confirm presence of loops!

Out-Of-Order Delivery

Causes of Packet Loops: BGP customer AS 2 C AS 1 A B D

Matching BGP Updates Any advertisement of the longest prefix? Temporal vicinity of 2 minutes to packet loops? Change in next hop or AS path?

Causes of Loops: ISIS R1 1 1 R2 1 R3 1 1 R4 4 R5

Time-Line at Nodes R2 and R3 Failure Detection LSP generation Shortest Path Computation LSP Flooding FIB Update LSP Arrival Shortest Path Computation FIB Update

If forwarding path changed and it is within temporal vicinity of loop Matching ISIS Updates Upon receipt of an LSP, compute the shortest path from the observation node to the egress router If forwarding path changed and it is within temporal vicinity of loop see if the observation node lies on the shortest path before or after the change

BGP Update Matches Trace % transient % persistent (BGP) % persistent (no BGP) Total NYC-20 40.1 50.8 90.8 NYC-21 80.2 7.5 87.9 NYC-23 3.3 NYC-22 18.8 80.6 99.4 NYC-24 70.0 NYC-25 43.7 15.5 59.2

Factors to Varying Success Persistent Loops Events occurred before trace collection BGP changes external to Sprint Comparison with RouteView updates: increase in matches Geographical distribution of loop destinations Measurement PoP not involved in route changes Avg # of ASes traversed: longest for NYC-23

Conclusions Loops can be detected and analyzed Loops are not uncommon Most are due to BGP updates BGP changes farther away from the observations point may not be identified

BACKUP SLIDE

CDF of Number of Replicas

CDF of Inter-Replica Spacing Time

Packet Types of All Traffic

Packet Types of Loops

Destination Addresses of Loops Backbone 1 Regional 2

CDF of Replica Stream Duration in Time

CDF of Routing Loop Duration in Time

Overview Types and causes behind routing loops Transient - part of normal routing protocol operation Persistent - “long-lasting”, manual intervention required Detection of routing loops in packet traces Detection algorithm Observations about the routing loops Analysis of performance impact Loss, delay, out-of-order delivery On-line detection algorithm Summary

Fraction of Packets in Loops Backbone 1 Backbone 4

Construction of a Typical End-To-End Path 10 hops in the Backbone Regional to Backbone DSL/LAN/Cable/Phone

Estimate of End-to-End Loss Assume: No loss on the access link due to routing loops Losses are independence between links Estimate: Lr: from Regional traces Lb: from Backbone traces but for Backbone 4 1 - (1- Lr)2(1- Lb)10 = 0.003 ~ 0.025 Implications on SLA??

Delay Due to Routing Loops Average delays range from 50 to 500ms.

Out-Of-Order Delivery

Causes of Loop Backbone 2,3, and 6 possibly had configuration problems. BGP updates - transient IGP - transient and persistent

Overview Types and causes behind routing loops Transient - part of normal routing protocol operation Persistent - “long-lasting”, manual intervention required Detection of routing loops in packet traces Detection algorithm Observations about the routing loops Analysis of performance impact Loss, delay, out-of-order delivery On-line detection algorithm Summary & Future Work

To Detect a Loop On-line Focus on persistent loops Questions: More focus on persistent loops How much traffic is affected? -> alarm What prefix is affected? -> warning

On-Line Detection Algorithm How many packets to /24 get looped? 100 WARNING How many looped packets / million? 5% How long (in millions) did it last? 10 millions ALARM By the time an alarm is raised, warnings are raised and help debugging the system Fixed memory and computation complexity

Validation of On-Line Algorithm

Summary Impact of routing on performance has been analyzed in terms of loss and delay. Per-link loss varies greatly. Excluding “outliers”, end-to-end loss of 0.3% is unavoidable. For a small number of packets that escape the loops, 50 ~ 500 msec delay is added on the average. On-line detection algorithm In conjunction with routing protocol monitoring, it will help detect and fix persistent loops.

More work needed to determined causes behind routing loops Future Work More work needed to determined causes behind routing loops Correlate with BGP/IS-IS updates Address hijacking Wrong aggregation Origin misconfiguration Export misconfiguration Integration with existing monitoring tools

Backup Slides

Superbowl Sunday, 2/3/2002 New Traces, not yet analyzed in detail. Only the on-line detection algorithm was used to collect alarms and warnings.

Superbowl Sunday, 2/3/2002 New Traces, not yet analyzed in detail. Only the on-line detection algorithm was used to collect alarms and warnings.

What Next? Alarms and warnings How to extract just enough info to be useful How to relate it with BGP/IS-IS update info How to integrate with management/monitoring infrastructure