1. Virtual Network Diagnosis as a Service Wenfei Wu (UW-Madison) Guohui Wang (Facebook) Aditya Akella (UW-Madison) Anees Shaikh (IBM System Networking)

2. Virtual Networks in the Cloud Application Traffic GW Middlebox Subnet 1 VM VS MB R R Subnet 2 VM VS VM The cloud provider maintains the data center infrastructure The tenant request/configure virtual networks for their apps The virtual network is mapped to the physical infrastructure Network services are provided as virtual middleboxes Multiple tenants share the infrastructure Tenant 2 VM VS VM

3. Virtual Network Problems Multiple layers may have various problems o Virtual network connectivity issues o Tenants application performance may be degraded Isolation and abstraction prevent tenants from diagnosing their virtual networks

4. Existing Solutions Infrastructure-layer tools may expose the physical network to tenants sFlow, NetFlow OFRewind, NDB, etc. Tracing tools in the virtual network are difficult to deploy in some virtual components (e.g., middleboxes) Tcpdump, Xtrace, SNAP

5. VND Proposal The cloud provider should offer a virtual network diagnosis service (VND) to the tenants 1.Diagnostic request 2.Trace collection 3.Trace parse 4.Data query Path: n 1 … n k Pattern: IP, port, … n 1 : Table Server 1 n 2 : Table Server 2 … Diagnosis Policy Table Server Tenant allocated resources & Physical Topology Path: n 1 … n k Pattern: IP, port n 1 : collector 1 n 2 : collector 2 … Diagnosis Policy Diagnosis Request Tenant Control Server

6. VND Challenges Preserve isolation and abstractions Low overhead Scalability

7. Contents Motivation VND Design & Implementation Evaluation Conclusions

8. VND Architecture Cloud Controller Collection Policy tsid… tsid… Raw Data Trace Collector Raw Data Trace Collector Flow Trace Collect Config Collect Config & Topology Policy Manager Tenant Query Execution Analysis Manager Table Server Control Server Query Executor Trace Table Query Executor Trace Table Trace Parser Parse Config Parse Config

9. Data Collection (1) The tenant submits a Data Collection Configuration Load Balancer Front End Server 1 Server 2 Server 3 Cap: Input srcIP=10.0.0.6 dstIP=10.0.0.8 tcp dstPort=80 dualDirection Cap: output … Virtual Appliance Link : l1 Capture Point node1 Flow Pattern field = value,... Capture Point node2... Virtual Appliance Node: n1 Capture Point input, [output]...

10. Data Collection (2) Policy Manager generates a Data Collection Policy srcIP=10.0.0.6, …, etc. Collector (c1) tr_id1 Trace ID tr_id1 Physical Cap Point vs1 Collector c1 at h1 Pattern field = value,... Path vs1,..., h1, c1 Hypervisor vSwitch (vs1) Load Balancer NIC

11. Data Parse The tenant submits a Data Parse Configuration Table ID tab_id1 Filter exp Fields field_list exp = exp and exp | exp or exp | not exp | (exp) | prim prim = field in value_set field_list = field (as name) (, field (as name))* Trace ID all Filter: ip.proto = tcp or ip.proto = udp Fields: timestamp as ts, ip.src as src_ip, ip.dst as dst_ip, ip.proto as proto, tcp.src as src_port, tcp.dst as dst_port, udp.src as src_port, udp.dst as dst_port tssrc_IPdst_IPprotosrc_portdst_port ………………

12. Data Analysis tsid… Trace_IDVnet_AppCap_LocPattern Trace_ID… Analysis Manager Collection ViewParse View tsid…tsid… Trace Table Query Executor Trace Table Query Executor Trace Table Query Executor SQL Interface Diagnostic Schema Trace Tables form a distributed database Operations Diagnostic Applications Tenants

13. Data Analysis Examples RTT 1.create view T1_f as select * from T1 where srcIP=IP1 and dstIP=IP2 2.create view T1_b as select * from T1 where dstIP=IP1 and srcIP=IP2 3.create view RTT as select f.ts as t1, b.ts as t2 from T1_f as f, T1_b as b where f.seq + f.payload_length = b.ack 4.select avg(t2-t1) from RTT throughput, loss, delay, statistics, etc. tssrc_IPdst_IPseqackPayload_length ………………

14. Optimizations Local Table Server placement Place the collector locally with the capture points Avoid the trace collection traffic traversing the network Move data only when queries need it Avoid interference with existing rules using the multi-table feature on OVS

15. Contents Motivation VND Design & Implementation Evaluation Conclusions

16. Trace Collection Overhead (1) We transfer data between VMs in 2 physical servers o 10Gbps NIC o 8 pairs of VMs, each pair transfer 1Gbps traffic Each minute, we capture one VM pairs traffic The duplicated traffic increases as we capture more traffic The original application traffic is not impacted

17. Trace Collection Overhead (2) VMs perform memory copy and data transfer simultaneously We measure memory and network throughput Each 1Gbps network traffic duplication causes 59 MB/s memory overhead

18. Data Query Overhead We use RTT monitoring as an example o 200 Mbps traffic o Calculate average RTT periodically Network overhead is negligible Execution time scales linearly with the data size VND can process 2-3 Gbps traffic in real time RTT

19. Scalability (1) Control Server is a simple web server can be scaled up easily Data collection distributed locally to avoid being the bottleneck

20. Scalability (2) Data Query simulation A data center with 10,000 servers, each has a 10Gbps NIC Virtual network size [2, 20] Query executors can process 3 Gbps traffic in real time Total link utilization [0.1, 0.9] Results 30% of total link capacity can be queried in real time

21. Conclusions The cloud provider should offer a virtual network diagnosis service to the tenants We design VND Architecture, interfaces and operations Address several important challenges Our experiments and simulation demonstrate the feasibility of VND

22. END Q&A Please contact with WISDOM http://wisdom.cs.wisc.edu

23. Functional Validation Middlebox Bottleneck Detection VND use throughput and RTT time to find abnormity REIDS

24. Functional Validation(2) Middlebox scaling REIDS

25. VND Architecture Cloud Controller Trace Parser Trace Table Table Server Policy Manager Analysis Manager Collect Config Parse Config Collect Config & Topology Collection Policy Flow Trace Parse Config Query Execution Tenant tsid… Raw Data Query Executor Trace Collector Trace Parser Trace Table Table Server tsid… Raw Data Query Executor Trace Collector Control Server