1. IBM Research GmbH, Zürich Research Laboratory R 3 C 2 : Reactive Route & Rate Control for CEE Mitch Gusat, Daniel Crisan, Cyriel Minkenberg, and Casimer DeCusatis

2. IBM Research GmbH, ZRL 2 Outline Introduction Background R 3 C 2 Evaluation Conclusions

3. IBM Research GmbH, ZRL 3 Introduction & Motivation Context: Congestion management in L2 datacenter networks Existing congestion mgmt schemes for CEE –Rate-only: Quantized Congestion Notification Rate/window reduction don’t benefit trading, HPC, BA apps –Route-only: Switch Adaptive Routing Based on QCN’s load sensor Exploits path diversity Proposal: R 3 C 2 –Dual Route & Rate control –Source-driven

4. IBM Research GmbH, ZRL 4 Outline Introduction Background R 3 C 2 Evaluation Conclusions

5. IBM Research GmbH, ZRL 5 QCN 802.1Qau switch NIC RL switch NIC end node end node CNM end node

6. IBM Research GmbH, ZRL 6 R 3 C 2 Concept Take advantage of CNMs at the source for adaptive load-balancing Congestion Point issues CNMs –Where is the hotspot? –How severe is the hotspot? Source receives the CNMs –Identifies the most severe hotspots –Reroutes traffic around the hotspots –Splits flows and rate-limits subflows

7. IBM Research GmbH, ZRL 7 Source Routing in CEE: VLAN Plain Ethernet is not source-routed Solution: VLAN –One tree per VLAN Source –Set VLAN# at injection  path selection P0P1P2P3P4P5P6P7 S1_0S1_1S1_2S1_3 S2_0S2_1S2_2S2_3 S3_0S3_1S3_2S3_3 VLAN0 VLAN1 VLAN2 VLAN3

8. IBM Research GmbH, ZRL 8 R 3 C 2 Algorithm P0 P7 S1_0 S1_3 S2_0 S2_1 S2_2 S2_3 S3_0 S3_1 S3_2 S3_3 No overload: Deterministic single path Congestion: Activate additional paths Path activation: avoid hotspots Use RL along each path

9. IBM Research GmbH, ZRL 9 R 3 C 2 Reaction Point Packet assigned the VLAN# of the 1 st eligible Rate Limiter RL flow 1 RL flow 2 RL flow 3 Packets from upper layers for a given destination D TX queue Reaction Point for D To network Assign VLAN1 Assign VLAN2 Assign VLAN3 MUX

10. IBM Research GmbH, ZRL 10 Outline Introduction Background R 3 C 2 Evaluation Conclusions

11. IBM Research GmbH, ZRL 11 Evaluation Methodology Venus + Dimemas simulator Traffic –Synthetic: permutations + hotspot –HPC Traces: NAS: BT, CG, FT, IS, MG WRF, NAMD, Liso, Airbus Model parameters –10Gbps CEE with MTU = 1500B –QCN and PFC: 802 DCB settings Topology: 2-ary n-tree

12. IBM Research GmbH, ZRL 12 Permutation Traffic

13. IBM Research GmbH, ZRL 13 Hotspot Traffic Scenario P0P7 S1_0S1_3 S2_0 S2_1 S2_2 S2_3 S3_0 S3_1 S3_2 S3_3 C=25% C=10% C=50% Single 95% flow

14. IBM Research GmbH, ZRL 14 Hotspot Traffic

15. IBM Research GmbH, ZRL 15 HPC Traces: Hotspot

16. IBM Research GmbH, ZRL 16 Conclusions Introduced R 3 C 2 –Source-driven adaptive routing scheme –QCN and VLAN are key Dual Route & Rate control –Improved stability and performance Performance benefits –80% over Deterministic –40% over Random

17. IBM Research GmbH, ZRL 17 Backup

18. IBM Research GmbH, ZRL 18 Routing Schemes

19. IBM Research GmbH, ZRL 19 Routing Schemes

20. IBM Research GmbH, ZRL 20 Routing Schemes

21. IBM Research GmbH, ZRL 21 Routing Schemes

22. IBM Research GmbH, ZRL 22 Routing Schemes

23. IBM Research GmbH, ZRL 23 Routing Schemes

24. IBM Research GmbH, ZRL 24 Routing Schemes

25. IBM Research GmbH, ZRL 25 HPC Traces: No Hotspot

26. IBM Research GmbH, ZRL 26 Hotspot Traffic (1)

27. IBM Research GmbH, ZRL 27 Hotspot Traffic (2)

28. IBM Research GmbH, ZRL 28 Hotspot Traffic (3)