1 Traffic Engineering for ISP Networks Jennifer Rexford IP Network Management and Performance AT&T Labs - Research; Florham Park, NJ

2 Outline  Internet routing protocols  Traffic engineering using traditional protocols –Optimizing network configuration to prevailing traffic –Requirements for topology, routing, and traffic info  Traffic demands –Volume of load between edges of the network –Technique for measuring the traffic demands  Route optimization –Tuning the link weights to the offered traffic –Incorporating various operational constraints  Conclusions

3 Autonomous Systems (ASes)  Internet divided into ASes –Distinct regions of administrative control (~14,000) –Routers and links managed by a single institution –Service provider, company, university, …  Hierarchy of ASes –Large, tier-1 provider with a nationwide backbone –Medium-sized regional provider with smaller backbone –Small network run by a single company or university  Interaction between ASes –Internal topology is not shared between ASes –… but, neighbors interact to coordinate routing

4 Path Traversing Multiple ASes Client Web server Path: 6, 5, 4, 3, 2, 1

5 Interdomain Routing: Border Gateway Protocol  ASes exchange info about who they can reach –IP prefix: block of destination IP addresses –AS path: sequence of ASes along the path  Policies configured by the AS’s network operator –Path selection: which of the paths to use? –Path export: which neighbors to tell? Client ( ) “I can reach /24” “I can reach /24 via AS 1”

6 Intradomain Routing: OSPF or IS-IS  Shortest path routing based on link weights –Routers flood the link-state information to each other –Routers compute the “next hop” to reach other routers  Weights configured by the AS’s network operator –Simple heuristics: link capacity or physical distance –Traffic engineering: tuning the link weights to the traffic

7 Motivating Problem: Congested Link  Detecting that a link is congested –Utilization statistics reported every five minutes –Sample probe traffic suffers degraded performance –Customers complain (via the telephone network?)  Reasons why the link might be congested –Increase in demand between some set of src-dest pairs –Failed router/link within the AS causes routing change –Failure/reconfiguration in another AS changes routes  Challenges –Know the cause, not just the manifestations –Predict the effects of possible changes to link weights

8 Traffic Engineering in an ISP Backbone  Topology of the ISP backbone –Connectivity and capacity of routers and links  Traffic demands –Offered load between points in the network  Routing configuration –Link weights for selecting paths  Performance objective –Balanced load, low latency, …  Question: Given the topology and traffic demands in an IP network, what link weights should be used?

9 Modeling Traffic Demands  Volume of traffic V(s,d,t) –From a particular source s –To a particular destination d –Over a particular time period t  Time period –Performance debugging -- minutes or tens of minutes –Time-of-day traffic engineering -- hours or days –Network design -- days to weeks  Sources and destinations –Individual hosts -- interesting, but huge! –Individual prefixes -- still big; not seen by any one AS! –Individual edge links in an ISP backbone -- hmmm….

10 Traffic Matrix in out Traffic matrix: V(in,out,t) for all pairs (in,out)

11 Problem: Hot Potato Routing  ISP is in the middle of the Internet –Multiple connections to multiple other ASes –Egress point depends on intradomain routing  Problem with point-to-point models –Want to predict impact of changing intradomain routing –But, a change in weights may change the egress point!

12 Traffic Demand: Multiple Egress Points  Definition: V(in, {out}, t) –Entry link (in) –Set of possible egress links ({out}) –Time period (t) –Volume of traffic (V(in,{out},t))  Computing the traffic demands –Measure the traffic where it enters the ISP backbone –Identify the set of egress links where traffic could leave –Sum over all traffic with same in, {out}, and t

13 Traffic Mapping: Ingress Measurement  Packet measurement (e.g., Netflow, sampling) –Ingress point i –Destination prefix d –Traffic volume V id i d ingress destination

14 Traffic Mapping: Egress Point(s)  Routing data (e.g., forwarding tables) –Destination prefix d –Set of egress points e d d destination

15 Traffic Mapping: Combining the Data  Combining multiple types of data –Traffic: V id (ingress i, destination prefix d) –Routing: e d (set e d of egress links toward d) –Combining: sum over V id with same e d i ingress egress set

16 Application on the AT&T Backbone  Measurement data –Netflow data (ingress traffic) –Forwarding tables (sets of egress points) –Configuration files (topology and link weights)  Effectiveness –Ingress traffic could be matched with egress sets –Simulated flow of traffic consistent with link loads  Challenges –Loss of Netflow records during delivery (can correct for it!) –Egress set changes between table dumps (not very many) –Topology changes between configuration dumps (just one!)

17 Traffic Analysis Results  Small number of demands contribute most traffic –Optimize routes for just the heavy hitters –Measure a small fraction of the traffic –Must watch out for changes in load and set of exit links  Time-of-day fluctuations –Reoptimize routes a few times a day (three?)  Traffic (in)stability –Select routes that are good for different demand sets –Reoptimize routes after sudden changes in load

18 Three Traffic Demands in San Francisco

19 Underpinnings of the Optimization  Route prediction engine (“what-if” tool) –Model the influence of link weights on traffic flow »Select a closest exit point based on link weights »Compute shortest path(s) based on link weights »Capture splitting over multiple shortest paths –Sum the traffic volume traversing each link  Objective function –Rate the “goodness” of a setting of the link weights –E.g., “max link utilization” or “sum of exp(utilization)”

20 Weight Optimization  Local search –Generate a candidate setting of the weights –Predict the resulting load on the network links –Compute the value of the objective function –Repeat, and select solution with lowest objective function  Efficient computation –Explore the “neighborhood” around good solutions –Exploit efficient incremental graph algorithms  Performance results on AT&T’s network –Much better than simple heuristics (link capacity, distance) –Quite competitive with multi-commodity flow solution

21 Incorporating Operational Realities  Minimize changes to the network –Changing just one or two link weights is often enough  Tolerate failure of network equipment –Weights settings usually remain good after failure –… or can be fixed by changing one or two weights  Limit the number of distinct weight values –Small number of integer values is sufficient  Limit dependence on accuracy of traffic demands –Good weights remain good after introducing random noise  Limit frequency of changes to the weights –Joint optimization for day and night traffic matrices

22 Conclusions  Our approach –Measure: network-wide view of traffic and routing –Model: data representations and “what-if” tools –Control: intelligent changes to operational network  Application in AT&T’s network –Capacity planning –Customer acquisition –Preparing for maintenance activities –Comparing different routing protocols  Ongoing work: interdomain traffic engineering

23 To Learn More…  Overview papers –“Traffic engineering for IP networks” ( –“Traffic engineering with traditional IP routing protocols” (  Traffic measurement –"Measurement and analysis of IP network usage and behavior” ( –“Deriving traffic demands for operational IP networks” (  Topology and configuration –“IP network configuration for intradomain traffic engineering” (  Intradomain route optimization –“Internet traffic engineering by optimizing OSPF weights” ( –“Optimizing OSPF/IS-IS weights in a changing world” (