Dynamic Scheduling of Network Updates Based on the slides by Xin Jin Hongqiang Harry Liu, Rohan Gandhi, Srikanth Kandula, Ratul Mahajan, Ming Zhang, Jennifer Rexford, Roger Wattenhofer Presented by Sergio Rivera

SDN: Paradigm Shift in Networking Many benefits – Traffic engineering [B4, SWAN] – Flow scheduling [Hedera, DevoFlow] – Access control [Ethane, vCRIB] – Device power management [ElasticTree] Systems operate by updating the data plane state of the network (periodically or by triggers) 1 Controller Direct and centralized updates of forwarding rules in switches

Motivation Time to update switch rules varies and leads to slow network updates: – Switch hardware capabilities – Control load on the switch – Nature of updates – RPC delays (not that much…) Controlled experiments explore the impact of four factors: – # rules to be updated – Priorities of the rules – Type of rules (insertion vs modification) – Control load 2

Observations 3 Single-Priority rules 3.3 ms per-rule update time Random-priority rules Reaches 18 ms per-rule update time when inserting 600 rules

Observations (cont’d). 4 Modifying rules (TCAM starts with 600 single-priority rules) 11 ms per-rule modification time (insert & delete). Control activities were performed while updating 100 rules Time to update highly varies. (e.g. 99 th percentile vs. 50 th percentile) >10x Even in controlled conditions, switch update time varies significantly!

Network Update is Challenging Requirement 1: fast – Slow updates imply longer periods where congestion or packet loss occur Requirement 2: consistent – No congestion, no blackhole, no loop, etc. 5 Network Controller

What is Consistent Network Update 6 Current State A D C B E F1: 5 F4: 5 F2: 5 F3: 10 Target State A D C B E F1: 5 F4: 5 F2: 5 F3: 10

What is Consistent Network Update Asynchronous updates can cause congestion Need to carefully order update operations 7 Current State A D C B E F1: 5 F4: 5 F2: 5 F3: 10 Target State A D C B E F1: 5 F4: 5 F2: 5 F3: 10 A D C B E F4: 5 F1: 5 F3: 10 F2: 5 Transient State

Existing Solutions are Slow Existing solutions are static [ConsistentUpdate’12, SWAN’13, zUpdate’13] – Pre-compute an order for update operations 8 F3 F4 F2 F1 Static Plan A Target State Current State A D C B E F4: 5 F2: 5 F1: 5 F3: 10 A D C B E F1: 5 F4: 5 F2: 5 F3: 10

Existing Solutions are Slow Existing solutions are static [ConsistentUpdate’12, SWAN’13, zUpdate’13] – Pre-compute an order for update operations Downside: Do not adapt to runtime conditions – Slow in face of highly variable operation completion time 9 F3 F4 F2 F1 Static Plan A

Static Schedules can be Slow 10 F3 F4 F2 F1 Static Plan B F3 F4 F2 F1 Static Plan A F1 F2 F3 F4 Time F1 F2 F3 F4 Time F1 F2 F3 F4 Time F1 F2 F3 F4 Time Target State Current State A D C B E F4: 5 F2: 5 F1: 5 F3: 10 A D C B E F1: 5 F4: 5 F2: 5 F3: 10 No static schedule is a clear winner under all conditions!

Dynamic Schedules are Adaptive and Fast 11 F3 F4 F2 F1 Static Plan B F3 F4 F2 F1 Static Plan A Target State Current State A D C B E F4: 5 F2: 5 F1: 5 F3: 10 A D C B E F1: 5 F4: 5 F2: 5 F3: 10 No static schedule is a clear winner under all conditions! F3 F4 F2 F1 Dynamic Plan Adapts to actual conditions!

Challenges of Dynamic Update Scheduling Exponential number of orderings Cannot completely avoid planning 12 Current State A D C B E F3: 5 F2: 5 F1: 5 F4: 5 F5: 10 Target State A D C B E F1: 5 F3: 5 F4: 5 F5: 10 F2: 5

Challenges of Dynamic Update Scheduling Exponential number of orderings Cannot completely avoid planning 13 Current State A D C B E F3: 5 F1: 5 F4: 5 F5: 10 Target State A D C B E F1: 5 F3: 5 F4: 5 F5: 10 F2: 5 F2 Deadlock

Challenges of Dynamic Update Scheduling Exponential number of orderings Cannot completely avoid planning 14 Current State A D C B E F3: 5 F2: 5 F4: 5 F5: 10 Target State A D C B E F1: 5 F3: 5 F4: 5 F5: 10 F2: 5 F1 F1: 5

Challenges of Dynamic Update Scheduling Exponential number of orderings Cannot completely avoid planning 15 Current State A D C B E F2: 5 F4: 5 F5: 10 Target State A D C B E F1: 5 F3: 5 F4: 5 F5: 10 F2: 5 F1F3 F1: 5 F3: 5

Challenges of Dynamic Update Scheduling Exponential number of orderings Cannot completely avoid planning 16 Current State A D C B E F4: 5 F5: 10 Target State A D C B E F1: 5 F3: 5 F4: 5 F5: 10 F2: 5 F1F3F2 Consistent update plan F2: 5 F1: 5 F3: 5

Dionysus Pipeline 17 Dependency Graph Generator Update Scheduler Network Current State Target State Consistency Property Encode valid orderings Determine a fast order based on constraints of CP Balance planning and opportunism using a two-stage approach Scheduler is independent of CP

Dependency Graph Generation 18 Dependency Graph Elements Operation node Resource node (link capacity, switch table size) Node Edge Dependencies between nodes Path node Current State A D C B E F3: 5 F2: 5 F1: 5 F4: 5 F5: 10 Target State A D C B E F1: 5 F3: 5 F4: 5 F5: 10 F2: 5 Group operations and link capacity resources

Dependency Graph Generation 19 Current State A D C B E F3: 5 F2: 5 F1: 5 F4: 5 F5: 10 Target State A D C B E F1: 5 F3: 5 F4: 5 F5: 10 F2: 5 Move F3 Move F1 Move F2

Dependency Graph Generation 20 Current State A D C B E F3: 5 F2: 5 F1: 5 F4: 5 F5: 10 Target State A D C B E F1: 5 F3: 5 F4: 5 F5: 10 F2: 5 A-E: 5 Move F3 Move F1 Move F2

Dependency Graph Generation 21 Current State A D C B E F1: 5 F4: 5 F5: 10 A-E: 5 5 Move F3 Move F1 Move F2 Target State A D C B E F1: 5 F3: 5 F4: 5 F5: 10 F2: 5 F3: 5 F2: 5

Dependency Graph Generation 22 A-E: Move F3 Move F1 Move F2 Target State A D C B E F1: 5 F3: 5 F4: 5 F5: 10 F2: 5 Current State A D C B E F3: 5 F2: 5 F1: 5 F4: 5 F5: 10

Dependency Graph Generation 23 A-E: Move F3 Move F1 Move F2 Target State A D C B E F1: 5 F3: 5 F4: 5 F5: 10 F2: 5 Current State A D C B E F3: 5 F2: 5 F1: 5 F4: 5 F5: 10

Dependency Graph Generation 24 A-E: 5 5 D-E: Move F3 Move F1 Move F2 Target State A D C B E F1: 5 F3: 5 F4: 5 F5: 10 F2: 5 Current State A D C B E F3: 5 F2: 5 F1: 5 F4: 5 F5: 10

Dependency Graph Generation DG captures dependencies but still leaves scheduling flexibility. Two challenges in dynamically scheduling updates: – Cycle resolution in the DG – At any given time, decide which subset of rules will be issued first 25

Dependency Graph Generation Supported scenarios – Tunnel-based forwarding: WANs – WCMP forwarding: data center networks Supported consistency properties – Loop freedom – Blackhole freedom – Packet coherence – Congestion freedom 26

Tunnel-Based Forwarding A flow is forwarded along one or more tunnels Ingress switch tags packet with the tunnel identifier Ingress switch splits incoming traffic accross the tunnels based on configured weights. Offers loop freedom and packet coherence by design Blackhole freedom is guaranteed if: – A tunnel is fully established before the ingress switch puts any traffic on it – All traffic is removed from the tunnel before the tunnel is deleted 27

Tunnel-Based Dependency Graph 28 S4-S5: 10 5 D A p3 p2 5 C S1-S4: 10 S5: 50 S1: 50 S4: E F G S2: 50 S1-S2: 0 S2-S5: B 1

WCMP Forwarding Switches at every hop match on packet headers and split flows over multiple next hops with configured weights Packet coherence is based on version numbers: – Ingress switch tags each packet with a version number – Downstream switches handle packets based on the embedded version number – Tagging ensures packets never mix configurations Blackhole- and Loop-Freedom is possible: – Do not require versioning – Can be encoded in the dependency graph – (authors omit description of graph construction for such conditions) 29

WCMP Dependency Graph 30 S4-S5: 10 Y p3 p2 5 S1-S4: 10 S2: S1-S2: 0 S2-S5: [(S2, 1), (S4,0)] -> [(S2, 0.5), (S4, [(S3, 0.5), (S5, 0.5)] -> [(S3, 1), (S5, 0)] X Has only one next-hop. No change needed

Dionysus Pipeline 31 Dependency Graph Generator Update Scheduler Network Current State Target State Consistency Property Encode valid orderings Determine a fast order

Dionysus Scheduling Scheduling as a resource allocation problem 32 A-E: 5 5 D-E: Move F3 Move F1 Move F2 How to allocate available resources to operations to minimize the update time?

Dionysus Scheduling Scheduling as a resource allocation problem 33 A-E: 0 D-E: Move F3 Move F1 Move F2 Deadlock!

Dionysus Scheduling Scheduling as a resource allocation problem 34 A-E: 0 5 D-E: Move F3 Move F1 Move F2

Dionysus Scheduling Scheduling as a resource allocation problem 35 A-E: 0 5 D-E: Move F3 Move F2

Dionysus Scheduling Scheduling as a resource allocation problem 36 A-E: Move F3 Move F2

Dionysus Scheduling Scheduling as a resource allocation problem 37 A-E: 5 5 Move F2

Dionysus Scheduling Scheduling as a resource allocation problem 38 Move F2 Done!

Dionysus Scheduling Scheduling as a resource allocation problem NP-complete problems under link capacity and switch table size constraints Approach – DAG: always feasible, critical-path scheduling to solve ties – Convert to a virtual DAG (general case include cycles) – Rate limit flows to resolve deadlocks 39

Critical-Path Scheduling Calculate critical-path length (CPL) for each node – Extension: assign larger weight to operation nodes if we know in advance the switch is slow Resource allocated to operation nodes with larger CPLs 40 A-B:5 CPL=3 C-D:0 5 CPL= CPL=2 CPL=1 Move F4 Move F3 Move F2 Move F1

Critical-Path Scheduling Calculate critical-path length (CPL) for each node – Extension: assign larger weight to operation nodes if we know in advance the switch is slow Resource allocated to operation nodes with larger CPLs 41 A-B:0 CPL=3 C-D:0 5 CPL= CPL=2 CPL=1 Move F4 Move F3 Move F2 Move F1

Handling Cycles Convert to virtual DAG – Consider each strongly connected component (SCC) as a virtual node 42 CPL=1 CPL=3 weight = 1weight = 2 Critical-path scheduling on virtual DAG – Weight w i of SCC: number of operation nodes A-E: 5 5 D-E: Move F3 Move F1 Move F2 Move F2

Handling Cycles Convert to virtual DAG – Consider each strongly connected component (SCC) as a virtual node 43 CPL=1 CPL=3 weight = 1weight = 2 Critical-path scheduling on virtual DAG – Weight w i of SCC: number of operation nodes A-E: 0 5 D-E: Move F3 Move F1 Move F2 Move F2

Deadlocks may still happen 44 The algorithm does not find feasible solution among all possible orderings (problem hardness) No feasible solution exists even if the target state is valid Resolve deadlocks reducing flow rates (informing rate limiters)

Resolving Deadlocks 45 K* = Max number of flows rate limited each time (affects throughput). Centrality to decide the order of iteration. Resource consumption by path-nodes or tunnel-add operations must happen within the same SCC.

Resolving Deadlocks (cont’d) 46 K* = 1 R3:0 P5 8 A P6 R1:0 8 R2:0 P1 C P4 P2 P7 B P By centrality, node A is selected. Reduces 4 units of traffic on both P6 and P7.

Resolving Deadlocks (cont’d) 47 K* = 1 R3:0 P5 8 A P6 R1:0 R2:0 P1 C P4 P2 P7 B P Which releases 4 units of free capacity in R1 and R2 8

Resolving Deadlocks (cont’d) 48 K* = 1 R3:0 P5 8 A R1:4 R2:4 P1 C P4 P2 B P A has no children. It does not belong to the SCC any more C is scheduled and partially schedules B 8

Resolving Deadlocks (cont’d) 49 K* = 1 R3:0 P5 0/8 A R1:0 0/8 R2:0 C P4 P2 B P3 4/8 4 4 Moves 4 units from P3 to P4 After C finishes, it schedules the remainded operation B 4/8

Resolving Deadlocks (cont’d) 50 K* = 1 R3:4 P5 0/8 A R1:0 0/8 R2:4 P4 B P3 0/4 For node A, increase its rate on P5 as long as R3 receives free capacity released by P4 0/4

Resolving Deadlocks (cont’d) 51 K* = 1 R3:0 P5 4/8 A R1:0 4/8 R2:0 P4 B For node A, increase its rate on P5 as long as R3 receives free capacity released by P4 4/4

Resolving Deadlocks (cont’d) 52 K* = 1 R3:4 P5 4/8 A R1:0 4/8 R2:0 For node A, increase its rate on P5 as long as R3 receives free capacity released by P4

Resolving Deadlocks (cont’d) 53 K* = 1 R3:0 R1:0 R2:0 NO DEADLOCK!

Evaluation: Testbed – TE Case Straggler switches (+500 ms latency) Install Tunnel Remove Tunnel

Evaluation: Testbed – Failure Recovery Case Straggler switches (+500 ms latency) Install Tunnel Remove Tunnel

Evaluation: Large-Scale Simulations Focus on congestion freedom as consistency property WAN: Dataset from a large WAN that interconnects O(50) sites. o Collect traffic logs on routers every 5-minute intervals o 288 traffic matrices collected. (each is a different state) o Tunnel-based routing is used DC: Dataset from a large data center network with several hundred switches. o Collect traffic traces by logging the socket events on all servers and aggregate into ToR-to-ToR flows over 5-minutes intervals o 288 traffic matrices collected per day o 1500 largest flows are chosen (40-60% of all traffic) o 1500 as switch rule memory size Alternative approaches: OneShot (opportunistic) and SWAN (static)

Evaluation: Update Time 57 99th 90th 50th Dionysus outperforms SWAN in both configurations. For instance, in WAN TE: Normal: 57% - 49% - 52% faster Straggler: 88% - 84% - 81% faster

Evaluation: Link Oversubscription (WAN Failure Recovery – Random Link) 58 Least oversubscription among the three Reduces oversubscription by 41% (N) - 42% (S) < SWAN Update time: 45% (N) - 82% (S) faster in the 99th percentile

Evaluation: Deadlocks Planning-based approaches lead to no deadlocks Opportunistic approach deadlocks 90% (WAN) and 70% (DC) of the time WAN Topology is less regular than DC topology (i.e., more complex dependencies)

Evaluation: Deadlocks (cont’d) 60 10% memory slack = Memory size as 1100 when switch is loaded with 1000 rules K*=5 in RateLimit algorithm

Conclusion Dionysus provides fast, consistent network updates through dynamic scheduling – Dependency graph: compactly encodes orderings – Scheduling: dynamically schedules operations 61 Dionysus enables more agile SDN control loops

Thanks! 62