1. DARD: Distributed Adaptive Routing for Datacenter Networks Xin Wu, Xiaowei Yang

2. Multiple equal cost paths in DCN Scale-out topology -> Horizontal expansion -> More paths srcdst core Agg ToR pod

3. Suboptimal scheduling -> hot spot src 1 src 2 dst 1 dst 2 Unavoidable intra-datacenter traffic Common services: DNS, search, storage Auto-scaling: dynamic application instances

4. To prevent hot spots Distributed – ECMP & VL2: flow-level hashing in switches Centralized – Hedera: compute optimal scheduling in ONE server Centralized: Efficient but Not Robust Distributed: Robust but Not Efficient Design Space

5. Goal: practical, efficient, robust Practical – Using well-proven technologies Efficient – Close to optimal traffic scheduling Robust – No single point failure Centralized: Efficient but Not Robust Distributed: Robust but Not Efficient Design Space Distributed: Robust and Efficient

6. Contributions Explore the possibility of distributed yet close- to-optimal flow scheduling in DCNs. A working implementation in testbed. Proven convergence upper bound.

7. Intuition: minimize the maximum number of flows via a link src 1 dst 1 src 3 dst 3 Step 0: maximum # of flows via a link = 3

8. src 1 dst 1 src 3 dst 3 Step 1: maximum # of flows via a link = 2 Intuition: minimize the maximum number of flows via a link

9. src 1 dst 1 src 3 dst 3 Step 2: maximum # of flows via a link = 1

10. Architecture Monitor network states Compute next scheduling Change flow’s path Control loop runs on every server independently

11. Monitor network states src asks switches for the #_of_flows and bandwidth of each link to dst. src dst src assemblies the link states to identify the most and least congested paths to dst.

12. Distributed computation Runs on every server 1. for each dst 2. { 3. P busy : the most congested path from src to dst; 4. P free : the least congested path from src to dst; 5. if (moving one flow from p busy to p free won’t cause a more congested path than p busy ) 6. Move one flow from p busy to p free ; 7. } Steps to convergence is bounded

13. Change path: using different src-dst pair core 1 core 2 1.0.0.0/8 2.0.0.0/8 3.0.0.0/84.0.0.0/8 core 3 core 4 src 1.1.1.2 2.1.1.2 3.1.1.2 4.1.1.2 srcdst 1.2.1.2 2.2.1.2 3.2.1.2 4.2.1.2 src-dst address pair uniquely encodes a path Static forwarding table tor 1 1.1.1.0/24 2.1.1.0/24 3.1.1.0/24 4.1.1.0/24 tor 2 agg 1 ’s down-hill table dst next hop 1.1.1.0/24 tor 1 1.1.2.0/24 tor 2 2.1.1.0/24 tor 1 2.1.2.0/24 tor 2 agg 1 1.1.0.0/16 2.1.0.0/16 agg 1 agg 2 agg 1 ’s up-hill table src next hop 1.0.0.0/8 core1 2.0.0.0/8 core2

14. Forwarding example: E 2 ->E 1 core 1 tor 1 tor 2 agg 1 agg 2 E1E1 E2E2 agg 1 ’s down-hill table dst next hop 1.1.1.0/24 tor 1 1.1.2.0/24 tor 2 2.1.1.0/24 tor 1 2.1.2.0/24 tor 2 agg 1 ’s up-hill table src next hop 1.0.0.0/8 core1 2.0.0.0/8 core2 1.0.0.0/8 2.0.0.0/8 1.1.1.21.2.1.2 src: 1.2.1.2, dst: 1.1.1.2 Packet header:

15. Forwarding example: E 1 ->E 2 core 1 tor 1 tor 2 agg 1 agg 2 E1E1 E2E2 agg 1 ’s down-hill table dst next hop 1.1.1.0/24 tor 1 1.1.2.0/24 tor 2 2.1.1.0/24 tor 1 2.1.2.0/24 tor 2 agg 1 ’s up-hill table src next hop 1.0.0.0/8 core1 2.0.0.0/8 core2 1.0.0.0/8 2.0.0.0/8 1.1.1.21.2.1.2 src: 1.1.1.2, dst: 1.2.1.2 Packet header:

16. Randomness: prevent path oscillation Add a random time interval to the control cycle

17. Implementation DeterLab testbed – 16-end-hosts fattree – Monitoring: OpenFlow API – Computation: daemon on end hosts – One NIC multiple addresses: IP alias – Static routes: OpenFlow forwarding table – Multipath: IP-in-IP encapsulation ns-2 simulator – For different & larger topologies

18. DARD fully utilizes the bisection bandwidth Traffic Patterns Bisection bandwidth (Gbps) Simulation, 1024-end-host fattree pVLB: periodical flow-level VLB

19. DARD improves large file transfer time Inter-pod dominant Intra-pod dominant random # of new files per secondDARD vs. ECMP improvement Testbed, 16-end-host fattree

20. Convergence time (seconds) Inter-pod dominant random Intra-pod dominant DARD converges in 2~3 control cycles Simulation, 1024-end-host fattree, static traffic patterns One control cycle ≈ 10 seconds

21. Inter-pod dominant random Intro-pod dominant Times a flow switches its paths Randomness prevents path oscillation Simulation, 128-end-host fattree

22. DARD’s control overhead is bounded by the topology control_traffic = #_of_servers x #_of_switches. Simulation, 128-end-host fattree DARD Hedera # of simultaneous flows Control traffic (MB/s)

23. Conclusion DARD: Distributed Adaptive Routing for Datacenters – Practical: well-proven end-host-based technologies – Efficient: close to optimal traffic scheduling – Robust: no single point failure Monitor network states Compute next scheduling Change flow’s path

24. Thank You! Questions and comments: xinwu@cs.duke.edu