1. Data Center Architectures
CIS 700/005 – Lecture 2
Includes material from lectures by Hakim Weatherspoon and Jennifer Rexford

2. Traditional Data Centers
Internet
Data Center
Layer-3 router
Core
Layer-2/3 switch
Aggregation
Layer-2 switch
Access
Servers

3. Limitation 1: Cost (Oversub)
Ratio of the worst-case achievable aggregate bandwidth among the end hosts to the total bisection bandwidth of a particular communication topology
Lower the total cost of the design
Typical designs: factor of 2:5:1 to 8:1
Often much higher! [VL2]

4. Limitation 2: Fault tolerance
Oversubscription + Bigger routers
 less routers at the top of the tree
 a core router failure has high blast radius
Most data centers used 1+1 redundancy
Dedicated backup switch and links

5. Limitation 3: Multi-path routing
Traditional data centers use static load balancing like ECMP
Can use bandwidth inefficiently
Limited ECMP group size

6. A Scalable, Commodity Data Center Network Architecture
Mohammad Al-Fares, Alexander Loukissas, Amin Vahdat
Scalable interconnection bandwidth
1:1 oversubscription
Economies of scale
Backwards compatibility

7. History Lesson: Clos Networks (1953)
Emulate a single huge switch with many smaller switches
Add more layers to scale out

8. History Lesson: Clos Networks (1953)
Emulate a single huge switch with many smaller switches
Add more layers to scale out

9. History Lesson: Clos Networks (1953)
Emulate a single huge switch with many smaller switches
Add more layers to scale out

10. Fat-tree Architecture
K-ary fat tree: three-layer topology (edge, aggregation and core)
each pod consists of (k/2)2 servers & 2 layers of k/2 k-port switches
each edge switch connects to k/2 servers & k/2 aggr. switches
each aggr. switch connects to k/2 edge & k/2 core switches
(k/2)2 core switches: each connects to k pods

11. Obligatory Network Questions
How do I address destinations?
Hierarchical IP addresses for scalability
[PodNumber].[SwitchNumber].[Endhost]
How does a switch route packets?
Assumption: every routing table entry has 1 output
Route downward using prefix (for scalability)
Route upward using suffix (for load balancing)

12. Routing Optimizations
Flow classification
Classify flows (e.g., src, dest, port #s)
Move around a small set of flows as needed
Flow scheduling
Keep track of large, long-lived flows at the edge switches
Assign them to different links

13. FatTree Summary Motivation: Data center networks are expensive
Limitation 1: Cost (oversub)
Use commodity hardware to keep costs down
Use Clos Networks
Limitation 2: Fault tolerance
Stop caring about individual components
Limitation 3: Multi-path routing
Schedule everything

14. Data centers today

15. Things they didn’t think about

16. Things they didn’t think about

17. Things they didn’t think about

18. What did they get right?
Motivation: Data center networks are expensive
Limitation 1: Cost (oversub)
Use commodity hardware to keep costs down
Use Clos Networks
Limitation 2: Fault tolerance
Stop caring about individual components
Limitation 3: Multi-path routing
Schedule everything

19. What did they get wrong?
Motivation: Data center networks are expensive
Limitation 1: Cost (oversub)
Use commodity hardware to keep costs down
Use Clos Networks
Limitation 2: Fault tolerance
Stop caring about individual components
Limitation 3: Multi-path routing
Schedule everything