1. Internet Routing Instability
Craig Labovitz
G. Robert Malan
Farnam Jahanian
Appeared: SIGCOMM ‘97
Presenters:
Supranamaya Ranjan
Mohammed Ahamed

2. Internet Structure Many small ISP’s at lowest level Small number of
big ISP’s at core

3. The Core of the Internet
Sprint
Verio
UUNet
rice.edu
Routing done using BGP at core
Inter-domain routing could be RIP/OSPF etc

4. BGP Overview Sprint Verio UUNet 92.92.x.x 128.42.x.x 196.29.x.x

5. BGP Overview (contd.) Path Vector protocol
Similar to Distance Vector routing
Loop detection done using AS_PATH field R1 R2
Peering session (TCP)
Exchange full routing table at start
Updates sent incrementally

6. The volume of BGP messages exchanged is
Key Point
The volume of BGP messages exchanged is
abnormally high
Most messages are redundant / unnecessary and do not
correspond to and topology or policy changes

7. Consequence: Instability
Normal data packets handled by dedicated hardware
BGP packet processing consumes CPU time
Severe CPU processing overhead takes the router offline
Route Flap Storm: B
Router A temporarily fails
When A becomes alive B & C
send full routing tables A
B & C fail…cascading effect C
How do we avoid /lessen the impact of these problems?

8. Route Dampening Router does not accept frequent route updates to a
destination
Might signal that network has erratic connectivity
Increment counter for destination when route changes
Counter exceeds threshold stop accepting updates
Decrement counter with time
Impediment-legitimate updates delayed
Problem:
Future legitimate announcements are accepted only
after a delay

9. Prefix Aggregation/Super-netting
Core router advertises a less specific network prefix
Reduces size of routing tables exchanged
Problems:
Prefix aggregation is not effective because:
- Internet addresses largely non-hierarchically assigned
Impediment: multihoming
- Domain renumbering not done when changing ISP’s
- 25% of prefixes multi-homed
- Multi-homed prefixes should be exposed at the core

10. Route Servers O(N) peering sessions per Router
1 peering session per router
Route
Server
In spite of this data exposes instability-due to non-hierarchical nature of network
In-spite of all these measures the BGP message overhead
is unexpectedly high

11. Evaluation Methodology
Data from Route Server at M.A.E west (D.C) peering point
Peering point for more than 60 major ISP’s
Nine month log
Time series analysis of message exchange events

12. Observation: Lot’s of redundant updates
Duplicate route with-drawls
ISP
Number of With-drawls
Unique
Ratio A
23276
4344 5 F
86417
12435 7 I
14112
175
One Reason:
- Stateless BGP
- No state of previous with-drawls maintained

13. Observation: Instability Proportional to Activity
After removing duplicate messages:
Lesser
messages
10:00 AM
ISP infrastructure up-grade
Instability density with time
6:00
12:00
18:00
24:00
Time of day
Logarithmic Z-axis

14. Evidence from Fine Grained Structure
Number of instability events
7 days
24 hours
Power spectral density
Frequency (1/hour)
Conjecture:
BGP packets are competing with data packets during
high bandwidth activity.

15. Observation: Instability & size uncorrelated
WADiff
Proportion of routing table
Proportion of announcements
AADiff
Proportion of routing table
Proportion of announcements
WADup
Proportion of routing table
Proportion of announcements
ISP’s serving more network prefixes
may not contribute more to instability

16. Observation: Instability distributed over routes
75% median
Cumulative proportion 10
# of announcements per prefix+AS
20% to 90% of routes change 10 times or less
No single route contributes significantly to instability

17. Observation: Synchronized updates
AADiff
Inter Arrival Time distribution for
AADiff’s
Proportion
30s
1min
Inter-arrival times of
updates shows periodicity
30 s and 1 minute patterns
Some routers collect and send
Updates once every 30 s
Possible reasons:
Routers get synchronized
Border router- Internal router: interaction misconfigured??

18. End-to-end Perspective
Chinoy: “Dynamics of Internet routing information” (SIGCOMM 93)
Measurements on NSFNET showed:
- Processing and forwarding latency of BDP update
is 3 orders of magnitude more than the latency incurred in
forwarding data packets
- Will lead to packet drops during the intervening period
Paxson: “End-to-End routing behavior in the internet” (SIGCOMM 96)
Routing loops introduce loops into other router’s routing tables
An end-to-end route changes every 1.5 hours on an average

19. End-to-End perspective (Paxson)
Pathology
type
Probability
in 1995
Probability
in 1996
same
Long-lived
Routing loops
0.14% ~
Short-lived
Routing loops
0.065% ~
same
Outage>30s
0.96%
2.2%
Total
1.5%
3.4%

20. Summary and Conclusions
Redundant routing information flows in core
Instability distributed across autonomous systems
Possible reasons for instability:
Stateless BGP updates
Misconfigured routers
Synchronization
Clocks driving the links not synchronized (link “flaps”)

21. Follow-up work & impact
“Origins of Internet Routing Instability”-1999
Migration from stateless to stateful BGP decreased duplicate withdrawals
by an order of magnitude
But Duplicate Announcements (AADup) doubled
Reason: Non-transitive attribute filtering not implemented
- BGP specification: “never propagate non-transitive attributes”..
- ASPATH is transitive attribute
- MED (Multi Exit Discriminator) is NOT transitive

22. Propagating MED’s Causes Oscillations