T3: TCP-based High-Performance and Congestion-aware Tunneling Protocol for Cloud Networking Satoshi Ogawa† Kazuki Yamazaki† Ryota Kawashima† Hiroshi Matsuo† † Nagoya Institute of Technology, Japan.

Network Virtualization 1 VM Many data centers use Edge-Overlay model  Multiple-tenant Data Center Multiple-tents share the physical network resources Traditional Data Center Network

The Edge-Overlay Model 2 Sender virtual switch VM Receiver virtual switch VM IP Tunnel Tenant1 Tenant2Tenant1 Tenant2 Traditional Data Center Network Physical Server VM packet Tunnel Header VM packet

 VXLAN(RFC7348)  NVGRE(RFC7637)  GENEVE(RFC draft)  STT(RFC draft) Existing Tunneling Protocols 3 packets discarding No congestion control mechanism Poor throughput VM packet Pseudo TCP IP Ethernet STT

Why Congestion Control is Important  Datacenter Network Environment Huge amount of traffic Small buffer size in physical switches 4 Congestion frequently occurs A queue of switch Data Center Network Lost!! Front Unstable performance

Our Proposal  Transparent Transport Tunneling (T3) 5 Congestion-aware No congestion control Authentic TCP-based packets discarding Effective use of Offloading Poor throughput

Overview of T3 6 Physical Server Tenant 1 VM Tenant 2 Tenant 1 Tenant 2 Virtual Switch Traditional Data Center Network Physical Switch VM1 Tunnel Header VM2 TCP UDP ARP

1. Hybrid Tunnels 7 Physical network TCP UDP Virtual Switch VM Tenant1 Tenant2 VM packet Traditional Data Center Network A Physical Server T3 VXLAN Tunneling protocol Network Stack Packet processing flow in the switch VM’s TCP packet High throughput Using various offloading features VM’s Non TCP packet Low-latency connectionless

2. Packet Aggregation 8 Virtual Switch VM Tenant1 Tenant2 VM packet Effective use of offloading Traditional Data Center Network A Physical Server VM packet aggregate packets VM packet Tunnel Header Making a large packet Packet processing flow in the switch

Application How Offloading Features Work  Offloading features reduce the load of kernel processing e.g.) TCP Segmentation Offloading (TS0) 9 NIC Kernel TSO DisabledTSO Enabled Segmentation Larger segment, more effective High load Low load

3. Suppression of Duplicated Retransmission  TCP-based tunneling has duplicated retransmission problems 10 VM TCP TCP VM TCP lost Increase of number of transmissions Retransmission VM TCP TCP Retransmission Unnecessary reduction of congestion control window Sender VM T3 Virtual switch Receiver VM

Our Approach to Duplicated Retransmission 11 Sender Side Section Tunnel Section Receiver Side Section VM T3 Virtual switch VM The TCP connection is transparently split into three sections TCP Communication  TCP-based tunneling has duplicated retransmission problems

Sender Side Section Receiver Side Section Tunnel Section How to Suppress Duplicated Retransmission 12 Seq.0 Ack.100 time Seq.0 LOST T3 (encapsulated Seq.0) LOST T3 (encapsulated Seq.0) Sender Side Section Tunnel Section Receiver Side Section Sender Side vSwitch Sender VM Receiver Side vSwitch Receiver VM TCP Communication T3 Tunnel (TCP) Virtual switch returns an Ack. Retrans. Seq.0 Ack.100 Virtual switch holds the VM packets for the retransmission.

4. Integration of Congestion Control  Existing tunneling protocol There are various congestion control algorithm in the physical network 13 In-host communication s Physical network In-host communication s Congestion control by T3  T3 Integration of congestion control algorithms CUBIC RENO CTCP CUBIC mixed Unstable performance

Congestion Control for Data Center  TCP congestion control CUBIC  Default in Linux CTCP  Default in windows DCTCP  For data center 14 Throughout oriented High performance and congestion avoidance T3 adopts DCTCP for the physical network Existing tunneling protocols can not use

Implementation  Platform Linux Open vSwitch  Linux TCP stack available congestion control  DCTCP  CUBIC  RENO 15 TCP Linux kernel …VXLANSTTT3 RENOCUBICDCTCP datapath Openflow match vport IP Open vSwitch Ethernet VM

Evaluation Fundamental throughput of T3 with no congestion 2. Performance of T3 (DCTCP) with congestion

Evaluation 1's Environment 17 Physical Server 1Physical Server 2VM CPU Core i5 3.20GHz (4cores, 4threads) Core i7 3.40GHz (4cores, 8threads) vCPU 2 cores Memory16GB4GB Network40Gbps NIC (40GBASE-SR4)Virtio vNIC MTU1500 bytes1420 bytes OSCentOS 6.5 (2.6.32) Virtual SwitchOpen vSwitch VM Virtual Switch Physical Server 1 Virtual Switch Physical Server 2 40Gbps Ethernet VM T3 STT VXLAN Iperf Server Iperf Server Iperf Client Iperf Client

Evaluation 1's Results 18

Evaluation 2's Environment 19 Virtual Switch Physical Server 2 Physical Server 3 VM Iperf Server Iperf Server Virtual Switch VM Iperf Client Iperf Client Physical Server 1 Virtual Switch Middle switch T3 STT Physical Server 1,3Physical Server 2VM CPUCore i5 3.20GHzCore i5 3.40GHzvCPU 1 core Memory8GB1GB Network10GBASE-TVirtio vNIC MTU1500 bytes1420 bytes OSFedora 22 (4.0.8)CentOS 6.5 (2.6.32) Virtual SwitchOpen vSwitch Congestion ControlSTT: CUBIC, T3: DCTCP(threshold 65)CUBIC Background traffic 10Gbps Observation point

Evaluation 2's Results (Queue Length) 20 Virtual Switch Physical Server 2 Physical Server 3 VM Iperf Server Iperf Server Virtual Switch VM Iperf Client Iperf Client Physical Server 1 Virtual Switch Middle switch T3 STT † † M. Alizadeh, A. Greenberg, D. Maltz, J. Padhye, P. Patel, B. Prabhakar, S. Sengupta, and M. Sridharan “Data Center TCP (DCTCP) ” SIGCOMM 2010.

Evaluation 2’s Results (Throughput) 21 Virtual Switch Physical Server 2 Physical Server 3 VM Iperf Server Iperf Server Virtual Switch VM Iperf Client Iperf Client Physical Server 1 Virtual Switch T3 STT 10Gbps

Conclusions and Future work  Transparent Transport Tunneling (T3) Hybrid Tunnel Aggregation of packets Compatibility with existing network devices Congestion-aware  Future work Evaluation of transfer delay 22