1 BGRP (Border Gateway Reservation Protocol) A Tree-Based Aggregation Protocol for Inter-Domain Reservations Ping Pan Ellen Hahne Henning Schulzrinne Ping Pan Ellen Hahne Henning Schulzrinne Bell Labs + Columbia Bell Labs Columbia

2 Outline Resource ReservationResource Reservation –Applications –Architectures –Challenges Protocol Scaling IssuesProtocol Scaling Issues BGRP ProtocolBGRP Protocol –Major Messages –Performance Conclusions & Future WorkConclusions & Future Work

3 Resource Reservation ApplicationsApplications Old Architecture: Int Serv + RSVPOld Architecture: Int Serv + RSVP ChallengesChallenges New Architecture: Diff Serv + BGRPNew Architecture: Diff Serv + BGRP

4 Reservation Applications Real-Time QoSReal-Time QoS –Voice over IP –Video Virtual Private NetworksVirtual Private Networks Differentiated ServicesDifferentiated Services –Better than Best Effort Traffic EngineeringTraffic Engineering –Offload congested routes –Integration of ATM, Optical & IP (MPLS) –Inter-Domain Agreements

5 Reservation Architectures Old Solution: Int Serv + RSVPOld Solution: Int Serv + RSVP –End-to-end –Per-flow ChallengesChallenges –Data Forwarding Costs –Protocol Overhead –Inter-Domain Administration New Solution: Diff Serv + BGRPNew Solution: Diff Serv + BGRP –Aggregated –Scalable –Manageable

6 Two Scaling Challenges Data Forwarding CostsData Forwarding Costs –Int Serv: per micro-flow –Diff Serv: ~32 AF/EF Code Points –Solves that problem ! Control Protocol OverheadControl Protocol Overhead –RSVP: O(N 2 ), N = # hosts –BGRP: O(N), N = something much smaller –Much more to say about this !

7 Protocol Scaling Issues Network StructureNetwork Structure Network SizeNetwork Size How much Aggregation?How much Aggregation? How to Aggregate?How to Aggregate?

8 R1 S1 S2 R8 R7 R5 R6 R4R3 R2 h1, h2, h3 Single-homed Stub Domain Multi-homed Transit Domain h1, h2, h3 S3 H2 H1 H5 AS1 AS2 AS3 AS4 AS5 AS6 Network Structure: Multiple Domains (AS)

9 Current Network Size 10 8 ( 60,000,000) Hosts10 8 ( 60,000,000) Hosts 10 5 (60,000) Networks10 5 (60,000) Networks 10 4 (6,000) Domains10 4 (6,000) Domains

10 Traffic Trace ( 90-sec trace, 3 million IP packet headers, at MAE-West, June 1, 1999)

11 Traffic Trace Over 1-month span (May 1999) at MAE-West:Over 1-month span (May 1999) at MAE-West: – 4,908 Source AS seen – 5,001 Destination AS seen –7,900,362 Source-Destination pairs seen!

12 How many Reservations? (How much aggregation?) 1 reserv’n per source-dest’n pair?1 reserv’n per source-dest’n pair? –10 16 host pairs –10 10 network pairs –10 8 domain pairs 1 reserv’n per source OR 1 reserv’n per dest’n?1 reserv’n per source OR 1 reserv’n per dest’n? –10 8 hosts –10 5 networks –10 4 domains Router capacity: 10 4 < # Reserv’ns < 10 6Router capacity: 10 4 < # Reserv’ns < 10 6 Conclusion: 1 reserv’n per Network or Domain for each Diff Serv traffic classConclusion: 1 reserv’n per Network or Domain for each Diff Serv traffic class

13 Network Growth ( )

14 Growth Rates Graph has a Log ScaleGraph has a Log Scale H (# Hosts) : Exponential growthH (# Hosts) : Exponential growth D (# Domains) : Exponential growthD (# Domains) : Exponential growth Moore’s Law can barely keep up!Moore’s Law can barely keep up! Overhead of control protocols?Overhead of control protocols? –O(H) or O(D), May be OK –O(H 2 ) or O(D 2 ), Not OK !

15 How to Aggregate? Combine Reserv’ns from all Sources to 1 Dest’n for 1 Diff Serv classCombine Reserv’ns from all Sources to 1 Dest’n for 1 Diff Serv class Data & Reserv’ns take BGP route to Dest’nData & Reserv’ns take BGP route to Dest’n BGP routes form Sink Tree rooted at Dest’n domain (no load balancing)BGP routes form Sink Tree rooted at Dest’n domain (no load balancing) Aggregated Reserv’ns form Sink TreeAggregated Reserv’ns form Sink Tree Where 2 branches meet, Sum Reserv’nsWhere 2 branches meet, Sum Reserv’ns

16 H2 S2 R8 R7 R4R3 h1, h2, h3 H1 H5 AS1 AS2 AS3AS4 AS5 AS6 S R1 R2 R5 R6 S3 A Sink Tree rooted at S3

17 AS 200 C D AS 400 G AS 300 E F AS 100 B AH Source Destination User-Edge Routers Border Routers end-to-end reservation request aggregated reservation request How to handle end-user reservation?

18 BGRP Protocol Basic OperationBasic Operation Comparison with RSVPComparison with RSVP EnhancementsEnhancements Performance EvaluationPerformance Evaluation

19 BGRP Basics Inter-Domain onlyInter-Domain only Runs between Border RoutersRuns between Border Routers Follows BGP RoutesFollows BGP Routes Reserves for Unicast FlowsReserves for Unicast Flows Aggregates Reserv’ns into Sink TreesAggregates Reserv’ns into Sink Trees Delivers its Messages ReliablyDelivers its Messages Reliably 3 Major Messages3 Major Messages –Probe: source to dest’n; reserv’n path discovery –Graft: dest’n to source; reserv’n establishm’t & aggreg’n –Refresh: adjacent routers; reserv’n maintenance

20 Tree Construction: 1st Branch R8 R7 R4 R3 h1, h2, h3 H2 H1 H5 AS1 AS2 AS3AS4 AS5 AS R5 R6 R2 R S3 S1 S2

21 Tree Construction: 2nd Branch S1 R8 R7 R6 R4 R3 h1, h2, h3 S3 H2 H1 H5 AS1 AS2 AS3AS4 AS5 AS R5 R2 R1 S2

22 Tree Construction: Complete R1 S1 S2 R8 R7 R6 R4 R3 R2 h1, h2, h3 S3 H2 H1 H5 AS1 AS2 AS3AS4 AS5 AS R5

23 PROBE Message Source (leaf) toward Destination (root)Source (leaf) toward Destination (root) Finds reservation pathFinds reservation path Constructs Route Record:Constructs Route Record: –Piggybacks Route Record in message –Checks for loops –Checks resource availability Does not store path (breadcrumb) stateDoes not store path (breadcrumb) state Does not make reservationDoes not make reservation

24 GRAFT Message Destination (root) toward Source (leaf)Destination (root) toward Source (leaf) Uses path from PROBE’s Route RecordUses path from PROBE’s Route Record Establishes reservations at each hopEstablishes reservations at each hop Aggregates reservations into sink treeAggregates reservations into sink tree Stores reservation state per-sink treeStores reservation state per-sink tree

25 REFRESH Message Sent periodicallySent periodically Between adjacent BGRP hopsBetween adjacent BGRP hops Bi-directionalBi-directional Updates all reserv’n state in 1 messageUpdates all reserv’n state in 1 message

26 Comparison of BGRP vs. RSVP Probing:Probing: –BGRP PROBE vs. RSVP PATH –Stateless vs. Stateful [O(N 2 )] Reserving:Reserving: –BGRP GRAFT vs. RSVP RESV –State-light [O(N)] vs. Stateful [O(N 2 )] –Aggregated vs. Shared Refreshing:Refreshing: –Explicit vs. Implicit –Bundled vs. Unbundled

27 BGRP Enhancements Flapping leavesFlapping leaves Rushing sapRushing sap Broken branchesBroken branches Keeping Our Reservation Tree Beautiful Despite:

28 Problem: Flapping Leaves Over-reservationOver-reservation QuantizationQuantization HysteresisHysteresis

29 Problem: Rushing Sap CIDR LabelingCIDR Labeling Quiet GraftingQuiet Grafting

30 Quiet Grafting: 1st Branch S1 S2 R8 R7 R4 R3 h1, h2, h3 S3 H2 H1 H5 AS1 AS2 AS3AS4 AS5 AS R5 R6 R2 R

31 Quiet Grafting: 2nd Branch S1 S2 R8 R7 R6 R4 R3 h1, h2, h3 S3 H2 H1 H5 AS1 AS2 AS3AS4 AS5 AS R5 R2 R1

32 Quiet Grafting: Complete S1 S2 R8 R7 R6 R4 R3 R2 h1, h2, h3 S3 H2 H1 H5 AS1 AS2 AS3AS4 AS5 AS R5 R1

33 Problem: Broken Branches Self-HealingSelf-Healing Filtering Route ChangesFiltering Route Changes

34 Performance Evaluation Region SizeRegion Size TopologyTopology Traffic LoadTraffic Load Refresh RateRefresh Rate Quantum SizeQuantum Size Show BGRP benefits as function of:

35 Flow Counts vs. Region Size

36 Flow Counts vs. Region Size Assume reserv’n is popular.Assume reserv’n is popular. Aggregation is needed !Aggregation is needed ! Region-based aggregation works.Region-based aggregation works. BGRP helps most when:BGRP helps most when: –Aggregating Region is Large. –Reserv’n Holding Time is Long. Theoretical “N vs. N 2 ” problem is real !Theoretical “N vs. N 2 ” problem is real !

37 Number of Flows (broken down by BW)

38 BGRP / RSVP Gain for each BW Class

39 n0n0 n1n1 nDnD nini l0l0 l1l1 lili s0s0 s1s1 sisi d1d1 didi dDdD d0d0 sDsD 3 distributions: Flat, Hierarchical, Selected Source Modeling the Topological Distribution of Demand

40 Reservation Count vs. Link Number

41 Reservation Count vs. Node Number

42 Gain: N rsvp / N bgrp

43 Reservation Count vs. Traffic Load Model for given hop H:Model for given hop H: – P paths thru H – T sink trees thru H –  path (Poisson  ) # RSVP reserv’ns =# RSVP reserv’ns = # BGRP reserv’ns =# BGRP reserv’ns = BGRP helps most for large BGRP helps most for large  –Gain ~ P / T Graph: P =100000, T =1000Graph: P =100000, T =1000   ()1eP () / 1   eT FT 

44 Reservation Count vs. Traffic Load

45 Message Rate vs. Refresh Rate Model for given hop H:Model for given hop H: – P paths thru H – T sink trees thru H –  path (Poisson  ) –  refresh rate RSVP msg rate =RSVP msg rate = BGRP msg rate =BGRP msg rate = BGRP helps most for , BGRP helps most for ,  –Gain ~ P / T Graph: P =100K, T =1000,  =10, =.001Graph: P =100K, T =1000,  =10, =     PPe() 321     PTe PT () /

46 Message Rate vs. Refresh Rate

47 Message Reduction vs. Quantum Size Single hop H (tree leaf)Single hop H (tree leaf)  micro-flows on H (birth/death, Poisson)  micro-flows on H (birth/death, Poisson) Each micro-flow needs 1 unit of BWEach micro-flow needs 1 unit of BW H manages aggregate BW reserv’nH manages aggregate BW reserv’n Quantization: Reserv’n must beQuantization: Reserv’n must be Hysteresis: Descent lags byHysteresis: Descent lags by kQ  Q

48    3  2,6 6,6 4  3,6 5  4,6 6  5,6 0, 0  0,3 22 1,3 33 2,3 3,3 6  5,9 9,9 7  6,9 8  7,9 9  8,9 99 8,1212,12 10  9,12 11  10,12 12  11,12 Quantization with Hysteresis State Transition Diagram for Q=3

49 Message Reduction vs. Quantum Size Closed-form expression for state probabilitiesClosed-form expression for state probabilities Quantization & Hysteresis cut message rate by:Quantization & Hysteresis cut message rate by: E.g.,  =100 & Q=10, message rate cut by 100E.g.,  =100 & Q=10, message rate cut by 100 Multi-hop model with Quiet Grafting:Multi-hop model with Quiet Grafting: –Further improvement –Approximate analysis –Simulation ekQi kQi i Q k          (/) () [()!] 0 1 1

50 Message Reduction vs. Load & Quantum Size

51 Conclusions Scalable Protocol StateScalable Protocol State Scalable Protocol ProcessingScalable Protocol Processing Scalable Protocol BandwidthScalable Protocol Bandwidth Scalable Data ForwardingScalable Data Forwarding Inter-Domain AdministrationInter-Domain Administration BGRP meets Challenges

52 Future Work Detailed Protocol SpecificationDetailed Protocol Specification SimulationSimulation Reference ImplementationReference Implementation MPLSMPLS Lucent productsLucent products Internet 2 (Q-bone)Internet 2 (Q-bone) IETF: Draft; BOF; Working GroupIETF: Draft; BOF; Working Group

53 Future Work: Bandwidth Broker Model DiffServ Trunk BB BR RT Backbone Network BB Bandwidth Broker RT First-hop Router BR Border Router End User BB RT Regional ISP-B RT Regional ISP-C Regional ISP-A BR BB