CQRD: A Switch-based Approach to Flow Interference in Data Center Networks Guo Chen Dan Pei, Youjian Zhao Tsinghua University, Beijing, China

The Problem Flow interference dramatically increases the flow completion time (FCT) of short delay-sensitive flows in data center networks (DCN) 2

Flow Interference Short delay-sensitive flows (majority in DCN) have to wait a long time at switches for buffer and bandwidth resources occupied by a few of long bandwidth-greedy flows (e.g., backup, replication)Short delay-sensitive flows (majority in DCN) have to wait a long time at switches for buffer and bandwidth resources occupied by a few of long bandwidth-greedy flows (e.g., backup, replication) 3

Flow Interference Short delay-sensitive flows (majority in DCN) have to wait a long time at switches for buffer and bandwidth resources occupied by a few of long bandwidth-greedy flows (e.g., backup, replication)Short delay-sensitive flows (majority in DCN) have to wait a long time at switches for buffer and bandwidth resources occupied by a few of long bandwidth-greedy flows (e.g., backup, replication) Caused by coarse Output Queue (OQ) switch queue management schemeCaused by coarse Output Queue (OQ) switch queue management scheme 4

Transport Layer Rate Control:Transport Layer Rate Control: DCTCP [SIGCOMM’10]DCTCP [SIGCOMM’10] HULL [NSDI’12]HULL [NSDI’12] D 2 TCP [SIGCOMM’12]D 2 TCP [SIGCOMM’12] D 3 [SIGCOMM’11]D 3 [SIGCOMM’11] Preemptive Flow Scheduling:Preemptive Flow Scheduling: PDQ [SIGCOMM’12]PDQ [SIGCOMM’12] pFabric [SIGCOMM’13]pFabric [SIGCOMM’13] Prior solutions Modification to end host and/or switch hardware New protocol stack and switch hardware 5

Intuition of CQRD Tackling the root cause of flow interference: Need a more fine-grained queue management scheme 6

The Goal Goal:Goal: Alleviate flow interferenceAlleviate flow interference Reduce FCT of short delay-sensitive flowsReduce FCT of short delay-sensitive flows Maintain high goodput of long bandwidth-greedy flowsMaintain high goodput of long bandwidth-greedy flows Objectives:Objectives: Transparent to end hostTransparent to end host No modification to protocol stackNo modification to protocol stack Based on underlying techniques available in commodity productionsBased on underlying techniques available in commodity productions 7

Our Solution CQRD: A fine-grained switch queue management scheme to flow interference 8

A GENDA 9

Toy Example: Flow Interference in OQ Switch NS2 simulation parameters:NS2 simulation parameters: Link capacity=10Gbps, Link delay=4us, Total buffer size=288KB, TCP initial window size=4, TCP initial RTO=200us.Link capacity=10Gbps, Link delay=4us, Total buffer size=288KB, TCP initial window size=4, TCP initial RTO=200us. 8x8 switch connected to host 1-8, Host 1-5 sending 10KB TCP flow to host 8, Host 6-7 sending 100MB TCP flow to host 88x8 switch connected to host 1-8, Host 1-5 sending 10KB TCP flow to host 8, Host 6-7 sending 100MB TCP flow to host 8 10

Toy Example: Flow Interference in OQ Switch FCT Goodput Short flows completed in ~100ms Goodput of short flows collapse 11

Toy Example: Flow Interference in OQ Switch FCT Goodput Short flows completed in ~100ms Goodput of short flows collapse Interfered by these 2 long flows 12

Toy Example: Flow Interference in OQ Switch FCT Goodput Short flows completed in ~100ms Goodput of short flows collapse Interfered by these 2 long flows Unfairly served 13

A GENDA 14

CQRD Design Crosspoint-QueueCrosspoint-Queue 15

CQRD Design Crosspoint-QueueCrosspoint-Queue Eliminating interference between flows on different switch pathsEliminating interference between flows on different switch paths (Output-Contending but not Path-Contending, OC-PC) 16

CQRD Design Crosspoint-QueueCrosspoint-Queue Eliminating interference between flows on different switch pathsEliminating interference between flows on different switch paths (Output-Contending but not Path-Contending, OC-PC) Separate buffer & Fair scheduling 17

CQRD Design Crosspoint-QueueCrosspoint-Queue Eliminating interference between flows on different switch pathsEliminating interference between flows on different switch paths (Output-Contending but not Path-Contending, OC-PC) Random-DropRandom-Drop Alleviate the flow interference within the same switch path (Path-Contending, PC)Alleviate the flow interference within the same switch path (Path-Contending, PC) 18

CQRD Design Crosspoint-QueueCrosspoint-Queue Eliminating interference between flows on different switch pathsEliminating interference between flows on different switch paths (Output-Contending but not Path-Contending, OC-PC) Random-DropRandom-Drop Alleviate the flow interference within the same switch path (Path-Contending, PC)Alleviate the flow interference within the same switch path (Path-Contending, PC) Occupy more buffer, more likely to be dropped 19

CQRD Design Crosspoint-QueueCrosspoint-Queue Eliminating interference between flows on different switch pathsEliminating interference between flows on different switch paths (Output-Contending but not Path-Contending, OC-PC) Random-DropRandom-Drop Alleviate the flow interference within the same switch path (Path-Contending, PC)Alleviate the flow interference within the same switch path (Path-Contending, PC) Occupy more buffer, more likely to be dropped 20

Toy Example: Flow Interference FCT Goodput 3 orders shorter FCT 3 orders higher goodput 21

Toy Example: Flow Interference FCT Goodput 3 orders shorter FCT 3 orders higher goodput Fairly served Almost no cost of goodput 22

Toy Example: Flow Interference FCT Goodput 3 orders shorter FCT 3 orders higher goodput Fairly served Almost no cost of goodput 23

A GENDA 24

Evaluation 1. How much FCT of short delay-sensitive flows is reduced in CQRD?1. How much FCT of short delay-sensitive flows is reduced in CQRD? 2. How much goodput of long bandwidth-greedy flows is sacrificed in CQRD?2. How much goodput of long bandwidth-greedy flows is sacrificed in CQRD? 25

Experiment 1 Single aggregation/core switch (ns2 simulations)Single aggregation/core switch (ns2 simulations) Simulation parameters:Simulation parameters: Link capacity=10Gbps, Link delay=4us, Total buffer size=5MB, TCP initial window size=4, TCP initial RTO=200us.Link capacity=10Gbps, Link delay=4us, Total buffer size=5MB, TCP initial window size=4, TCP initial RTO=200us. Traffic:Traffic: 1200 TCP flows, Flow size & inter-arrival time from realistic distributions, Random source & destination port1200 TCP flows, Flow size & inter-arrival time from realistic distributions, Random source & destination port 26

Single aggregation/core switch FCT of all short flows ( 100KB) interfered by the giant flows (> 1MB, included by large flows) at moderate load (0.1). 27

Single aggregation/core switch FCT of all short flows ( 100KB) interfered by the giant flows (> 1MB, included by large flows) at moderate load (0.1). ~36% lower ~7% lower ~28% lower ~4% lower 28

Experiment 2 Multi-stage DCN switching fabric (ns2 simulations)Multi-stage DCN switching fabric (ns2 simulations) Simulation parameters:Simulation parameters: Link delay=2us, Agg switch buffer size=5MB, ToR switch buffer size=4MB, TCP initial window size=4, TCP initial RTO=200us.Link delay=2us, Agg switch buffer size=5MB, ToR switch buffer size=4MB, TCP initial window size=4, TCP initial RTO=200us. Traffic:Traffic: 2000 TCP flows, realistic distributions; ECMP load-balancing schemes2000 TCP flows, realistic distributions; ECMP load-balancing schemes 29

Single aggregation/core switch ~14% lower ~30% lower ~2.5% lower ~same FCT of all short flows ( 100KB) interfered by the giant flows (> 1MB, included by large flows) at moderate load (0.1). 30

A GENDA 31

Conclusion Tackling the root cause of flow interference:Tackling the root cause of flow interference: Need a more fine-grained queue management schemeNeed a more fine-grained queue management scheme Simple solution: CQRD—switch queue management schemeSimple solution: CQRD—switch queue management scheme Transparent to end hostTransparent to end host No modification to protocol stackNo modification to protocol stack Based on underlying techniques available in commodity productionsBased on underlying techniques available in commodity productions Reduces the FCT of short flows by 20-44% in a single switch and 8-30% in a multi-stage data center switch networkReduces the FCT of short flows by 20-44% in a single switch and 8-30% in a multi-stage data center switch network At the cost of a minor goodput decrease for large flowsAt the cost of a minor goodput decrease for large flows 32

T HANK Y OU