1. OLD DOG CONSULTING www.mpls2007.com MPLS-TE Doesn’t Scale Adrian Farrel Old Dog Consulting adrian@olddog.co.uk

2. OLD DOG CONSULTING 2 Is the Sky Falling? The only way to get your attention is to be alarmist MPLS-TE is perfectly functional in today’s networks But: MPLS-TE will not scale indefinitely The problem is the well-known “full mesh” or “n-squared” problem The number of LSPs scales as the square of the number of PEs

3. OLD DOG CONSULTING 3 What Do We Want to Achieve? MPLS-TE is an important feature for many SPs Allow traffic to be groomed Optimize use of network resources Provide quality of service guaranties Carriers look to provide edge-to-edge tunnels across their core networks Differentiated Services VPNs VLANS and pseudowires Multimedia content distribution Normal IP traffic

4. OLD DOG CONSULTING 4 What is the Scope of the Problem? Consider a service provider network with 1000 PEs This is not outrageously large Such a network may be broken into areas or ASes Consider a full mesh of PE-PE TE-LSPs Consider parallel tunnels for different services, QoS levels, and for protection May give rise to multiples of 999,000 LSPs in the core What is the capacity of a core LSR? What is the capacity of a management system?

5. OLD DOG CONSULTING 5 What Are the Scaling Limits? Management NMS How many LSPs can the NMS process Management protocols Reporting on large numbers of LSPs may overload the management network LSR issues Memory capacity Per LSP data requirements CPU capacity – largely an RSVP-TE protocol issue Degradation of LSP setup times Soft state addressed by Refresh Reduction MPLS forwarding plane Number of labels (Only 1048559 per interface)

6. OLD DOG CONSULTING 6 The Snowflake Topology Example network for analysis Meshed core of P nodes Called P 1 nodes Each P i+1 node connected to just one P i node PE nodes connected to just one P n node Well-defined connectivity and symmetry allows many important metrics to be computed Number of levels & number of nodes per level may be varied We can vary the number of P 1 nodes We can vary the ratio of P i+1 to P i We can vary the value n We can vary the number of PE nodes per P n node PE P1P1 P2P2

7. OLD DOG CONSULTING 7 Analysing the Snowflake Topology Define P n a node at the nth level (level 1 is core) S n the number of nodes at the nth level M n the multiplier at the nth level (how many P n+1 nodes are connected to a P n node) L n number of LSPs seen by a P n node Discover L PE = 2*(S PE - 1) L 2 = M 2 *(2*S PE - M 2 - 1) L 1 = M 1 *M 2 *(2*S PE – M 2 *(M 1 + 1)) Practical numbers S 1 = 10, M 1 = 10, and M 2 = 20 S PE = 2000 L PE = 3998 L 2 = 79580 L 1 = 756000

8. OLD DOG CONSULTING 8 The Ladder Topology Example network for analysis Core of P 1 nodes looks like a ladder Similar to many national networks Symmetrical trees subtended to core Each P i+1 node connected to just one P i node Each PE node connected to just one P node Again: Well-defined connectivity and symmetry allows many important metrics to be computed Number of levels & number of nodes per level may be varied

9. OLD DOG CONSULTING 9 Analysing the Ladder Topology Same definitions as for snowflake network E the number of subtended edge nodes (PEs) to each spar-node (E = M 1 *M 2 ) Discover L PE = 2*(S PE - 1) L 2 = 2*M 2 *(S PE - 1) - M 2 *(M 2 - 1) L 1 ≈ E*E*S 1 *S 1 /2 + E*E*S 1 + 3*E*E - E*M 2 Practical numbers S 1 = 10, M 1 = 10, and M 2 = 20 E = 200 S PE = 2000 L PE = 3998 L 2 = 79580 L 1 = 2516000

10. OLD DOG CONSULTING 10 Option 1 – Solve a Different Problem! If a full mesh of PE-PE LSPs is too big, don’t build it! This is the bottom line if we don’t fix the problem The suggestion is to build a full mesh of P n -to-P n LSPs, and perform routing or routing-based MPLS between P n and PE Scaling improves from O(1000 2 ) to O(100 2 ) But we lose functionality Why did we want a PE-PE mesh? How do we handle private address spaces? What if the traffic is not routable? This may simply not be good enough to provide the function

11. OLD DOG CONSULTING 11 Option 2 – LSP Hierarchies Well-known, core MPLS function Label stacks Forwarding Adjacencies (RFC 4206) Configured or automatic grooming Possible to build a full or partial mesh of hierarchical tunnels For example connect all P 2 nodes Each P 2 node must encapsulate each PE-PE LSP in the correct tunnel Each P 1 node only sees the P 2 -P 2 tunnels

12. OLD DOG CONSULTING 12 Scaling Properties of Hierarchies - Snowflake Note that PE-PE tunnels don’t help P 1 -P 1 tunnels are also no benefit (core is fully meshed) P 2 nodes see all PE-PE LSPs and new tunnels L 2 = M 2 *(2*S PE - M 2 - 1) + 2*(S 2 - 1) Situation at P 1 nodes is much better L 1 = M 1 *(2*S 2 - M 1 - 1) Numbers (S 1 = 10, M 1 = 10, and M 2 = 20) Flat2-Level Hierarchy S PE 20002000 L PE 39983998 L 2 7958079778 L 1 7560001890 Maybe insert another layer (P 3 ) to increase the scaling? L 3 remains high

13. OLD DOG CONSULTING 13 Scaling Properties of Hierarchies - Ladder Note that PE-PE tunnels don’t help But P 1 -P 1 tunnels are good because core is not fully-meshed L 1 ≈ S 1 *S 1 /2 + 2*S 1 + 2*E*E*(S 1 - 1) - E*M 2 - 2 Another level of hierarchy is also possible Add a mesh of P 2 -P 2 tunnels L 1 = S 1 *S 1 /2 + 2*S 1 + 2*M 1 *M 1 *S 1 - M 1 (M 1 + 1) – 2 L 2 = 2*M 2 *(S(PE) - 1) - M 2 *(M 2 - 1) + 2*(S(1)*M(1) - 1) Numbers (S 1 = 10, M 1 = 10, and M 2 = 20) Flat2-Level 3-LevelHierarchy S PE 200020002000 L PE 399839983998 L 2 795807958079778 L 1 2516000 7160601958

14. OLD DOG CONSULTING 14 Issues and Drawbacks for Hierarchies Scaling is not good enough! Impact on layer adjacent to PEs is negligible Actually impact is slightly negative Management burden Plan and operate a secondary mesh Effectively the same burden as managing PEs or a layered network Possible to consider auto-mesh techniques Fast Reroute protection is a problem FRR struggles to protect tunnel end-points Not obvious how to arrange the hierarchy when the network is not symmetrical E.g., some PEs closer to the core

15. OLD DOG CONSULTING 15 Option 3 – Multipoint-to-Point LSPs LSPs merge automatically as they converge on the destination Reduces the number of LSPs toward the egress Other LSP properties (e.g., bandwidth) must be cumulative TE is still possible, but de-merge is not considered Should count “LSP state” not number of LSPs New definition X n the amount of LSP state held at each P n node For flat and hierarchical networks: Each LSP adds one state at ingress or egress Each LSP adds two states at each transit node

16. OLD DOG CONSULTING 16 Scaling Properties of MP2P LSPs - Snowflake X PE = 2*(S PE - 1) X 2 = S PE *(M 2 + 1) X 1 = M 1 *M 2 *(S 1 - 2) + S PE *(M 1 + 1) Numbers (S 1 = 10, M 1 = 10, and M 2 = 20) Flat2-Level HierarchyP2MP S PE 200020002000 X PE 399839983998 X 2 15916015935842000 X 1 1512000378023600

17. OLD DOG CONSULTING 17 Scaling Properties of MP2P LSPs - Ladder X PE = 2*(S PE - 1) X 2 = (M 2 + 1)*S 1 *E X 1 ≤ (4 + M 1 )*S 1 *E - M 1 *E Numbers (S 1 = 10, M 1 = 10, and M 2 = 20) Flat2-Level 3-LevelP2MPHierarchy S PE 200020002000 2000 X PE 3998399839983998 X 2 15916015916015935842000 X 1 50320001433998389826000

18. OLD DOG CONSULTING 18 Issues and Drawbacks for MP2P LSPs Clear scaling benefits Better than flat networks Only thing that improves the situation adjacent to PEs But… Data plane support This will only ever be a packet/frame/cell technology Control plane support RSVP does have MP2P support RSVP-TE features not yet specified or implemented De-aggregation and disambiguation May be necessary to use label stack so that egress can detect sender of data OAM may be more complex and require source labels New management applications needed FRR still to be designed

19. OLD DOG CONSULTING 19 Other Topics for Investigation Cost-effectiveness of the network Revenue only generated by PEs K = S(PE)/(S(1)+S(2) +... + S(n)) Many ways to improve scaling reduce cost-effectiveness Fast Reroute What are the implications of FRR to scaling? Can scaling contributions be designed that can be protected by FRR? Point-to-multipoint What are the scaling properties of P2MP MPLS-TE? Domain boundaries (in particular AS boundaries) Boundaries such as at area and AS borders cause constrictions How can we reduce the number of LSPs seen by ABRs and ASBRs?

20. OLD DOG CONSULTING 20 Conclusions, Next Steps, and References MPLS-TE is not a scaling issue today But it won’t scale arbitrarily We need to plan now for tomorrow’s scalability Hierarchical LSPs are not as good as expected MP2P LSPs may offer a better solution More research and implementation is needed draft-ietf-mpls-te-scaling-analysis-01.txt Seisho Yaukawa (NTT) Adrian Farrel (Old Dog Consulting) Olufemi Komolafe (Cisco Systems)

21. OLD DOG CONSULTING 21 Questions? adrian@olddog.co.uk