An Approach to IP Network Traffic Engineering NANOG Miami, FL Chris Liljenstolpe Cable & Wireless

Scope and Purpose  Describes C&W’s Traffic Engineering methodology as well as some of the reasoning behind it.  Not “The One True Way,” but a method that works for us.

Scale of the Previous Design  Originally a flat network - one layer of routers interconnected over a complete PVC mesh.  A Network “event” in 1998 on AS3561 “educated” the engineering staff on IGP scaling issues.  This “event” lead to a week of network instability as it was re-engineered.  At one time there were 380+ routers in the direct mesh, accounting for 30k PVCs in the network & 760+ direct IGP associations per router.

Hierarchy  At one time there were 380+ routers in the direct mesh, accounting for 30k PVCs & IGP associations.  Currently there are no more than 80 routers in any one mesh due to the addition of hierarchy.  Due to the shrinking mesh sizes, and code optimization efforts, calculation times have dropped from 4 hours to 20 minutes.

Online vs. Offline  We like to always know where our traffic is and where it is routed.  Calculating optimal routing takes time on dedicated compute platforms…

Layer 2 vs. Layer 3  Utilizing IGP metrics to adjust traffic flows on an IP network leads to network-wide (and sometimes/usually, unplanned) effects in a large network, due to flooding.  This can lead to the network equivalent of the midway game “Hit the groundhog”

IGP Use  The IGP (in our case 2 level IS-IS) is only used for link state signaling in normal and most failure mode conditions.  In the worst case dual failure mode condition, the IGP does provide real next- hop calculations.

IGP Metrics  Because of the direct router-router adjacencies provided by the underlying network, a large set of IGP metrics are not needed.  The set in use is small, and only used to select primary vs. secondary path, and discourage “expensive” link utilization in a multi-point failure that leads to multi-hop routing.

ATM to MPLS for TE  ATM w/ PVC’s worked quite nicely Except for ATM overhead And lack of high-speed router interfaces  For our traffic engineering network, we are treating MPLS as an IP friendly ATM (actually more like Frame Relay, but never mind)

Will ’s Replace MPLS?  Only when the bandwidth required for any router- router pair approaches the bandwidth available from a single on the DWDM plant AND the cost of a port on an OXC is significantly cheaper than an equivalent bandwidth port on an MPLS switch.  When that occurs, the ’s will be provisioned just as the MPLS LSP’s are – statically with resilience.  GMPLS may be the technology used to signal the path over the OXC, just as MPLS is used for the LSP’s today.

Tools  Currently the tools that compute the paths, and configure the layer 2 and layer 3 equipment with those paths are all developed and maintained in-house.  Some have been in continual development and “tweak” mode for 6 years.

Futures  Most link failures will be detected and handled at the layer 2 traffic engineering layer, instead of at layer 3.  Path redundancy will grow from 2 to 4 paths per router-router pair.  Developments optimization mathematics originally researched for circuit path layout and analog circuit design will be utilized in the path layout tools.  Networks other than the IP backbone will utilize the traffic engineering core.