Internet Routing Instability Craig Labovitz G. Robert Malan Farnam Jahanian Appeared: SIGCOMM ‘97 Presenters: Supranamaya Ranjan Mohammed Ahamed

Internet Structure Many small ISP’s at lowest level Small number of big ISP’s at core

The Core of the Internet Sprint Verio UUNet rice.edu Routing done using BGP at core Inter-domain routing could be RIP/OSPF etc

BGP Overview Sprint Verio UUNet 92.92.x.x 128.42.x.x 196.29.x.x

BGP Overview (contd.) Path Vector protocol Similar to Distance Vector routing Loop detection done using AS_PATH field R1 R2 Peering session (TCP) Exchange full routing table at start Updates sent incrementally

The volume of BGP messages exchanged is Key Point The volume of BGP messages exchanged is abnormally high Most messages are redundant / unnecessary and do not correspond to and topology or policy changes

Consequence: Instability Normal data packets handled by dedicated hardware BGP packet processing consumes CPU time Severe CPU processing overhead takes the router offline Route Flap Storm: B Router A temporarily fails When A becomes alive B & C send full routing tables A B & C fail…cascading effect C How do we avoid /lessen the impact of these problems?

Route Dampening Router does not accept frequent route updates to a destination Might signal that network has erratic connectivity Increment counter for destination when route changes Counter exceeds threshold stop accepting updates Decrement counter with time Impediment-legitimate updates delayed Problem: Future legitimate announcements are accepted only after a delay

Prefix Aggregation/Super-netting Core router advertises a less specific network prefix Reduces size of routing tables exchanged Problems: Prefix aggregation is not effective because: - Internet addresses largely non-hierarchically assigned Impediment: multihoming - Domain renumbering not done when changing ISP’s - 25% of prefixes multi-homed - Multi-homed prefixes should be exposed at the core

Route Servers O(N) peering sessions per Router 1 peering session per router Route Server In spite of this data exposes instability-due to non-hierarchical nature of network In-spite of all these measures the BGP message overhead is unexpectedly high

Evaluation Methodology Data from Route Server at M.A.E west (D.C) peering point Peering point for more than 60 major ISP’s Nine month log Time series analysis of message exchange events

Observation: Lot’s of redundant updates Duplicate route with-drawls ISP Number of With-drawls Unique Ratio A 23276 4344 5 F 86417 12435 7 I 2479023 14112 175 One Reason: - Stateless BGP - No state of previous with-drawls maintained

Observation: Instability Proportional to Activity After removing duplicate messages: Lesser messages 10:00 AM ISP infrastructure up-grade Instability density with time 6:00 12:00 18:00 24:00 Time of day Logarithmic Z-axis

Evidence from Fine Grained Structure Number of instability events 7 days 24 hours Power spectral density Frequency (1/hour) Conjecture: BGP packets are competing with data packets during high bandwidth activity.

Observation: Instability & size uncorrelated WADiff Proportion of routing table Proportion of announcements AADiff Proportion of routing table Proportion of announcements WADup Proportion of routing table Proportion of announcements ISP’s serving more network prefixes may not contribute more to instability

Observation: Instability distributed over routes 75% median Cumulative proportion 10 # of announcements per prefix+AS 20% to 90% of routes change 10 times or less No single route contributes significantly to instability

Observation: Synchronized updates AADiff Inter Arrival Time distribution for AADiff’s Proportion 30s 1min Inter-arrival times of updates shows periodicity 30 s and 1 minute patterns Some routers collect and send Updates once every 30 s Possible reasons: Routers get synchronized Border router- Internal router: interaction misconfigured??

End-to-end Perspective Chinoy: “Dynamics of Internet routing information” (SIGCOMM 93) Measurements on NSFNET showed: - Processing and forwarding latency of BDP update is 3 orders of magnitude more than the latency incurred in forwarding data packets - Will lead to packet drops during the intervening period Paxson: “End-to-End routing behavior in the internet” (SIGCOMM 96) Routing loops introduce loops into other router’s routing tables An end-to-end route changes every 1.5 hours on an average

End-to-End perspective (Paxson) Pathology type Probability in 1995 Probability in 1996 same Long-lived Routing loops 0.14% ~ Short-lived Routing loops 0.065% ~ same Outage>30s 0.96% 2.2% Total 1.5% 3.4%

Summary and Conclusions Redundant routing information flows in core Instability distributed across autonomous systems Possible reasons for instability: Stateless BGP updates Misconfigured routers Synchronization Clocks driving the links not synchronized (link “flaps”)

Follow-up work & impact “Origins of Internet Routing Instability”-1999 Migration from stateless to stateful BGP decreased duplicate withdrawals by an order of magnitude But Duplicate Announcements (AADup) doubled Reason: Non-transitive attribute filtering not implemented - BGP specification: “never propagate non-transitive attributes”.. - ASPATH is transitive attribute - MED (Multi Exit Discriminator) is NOT transitive

Propagating MED’s Causes Oscillations