1. 1 VTL: A Transparent Network Service Framework John R. Lange and Peter A. Dinda Prescience Lab Department of Electrical Engineering and Computer Science Northwestern University http://plab.cs.northwestern.edu

2. Transparent Network Services Manipulate data and signaling of flows/connections to add services to existing unmodified applications and OSes –High Level transformations of Low Level traffic –Transparency: Manipulations invisible to guest environment VTL (Virtual Traffic Layer) –A framework for creating Transparent Network Services Wide range of possible services –Many useful for HPDC

3. Outline Defining Transparent Network Services Motivation VTL Framework –Architecture –Performance Example Transparent Network Services –Protocol Transformations –Anonymous Networking –… Conclusion and Future Work

4. Transparency Improving Existing Unmodified Applications –Invisible to connection end points –No changes to guest environment –Seamless integration of networking techniques Transparency readily available with VMS –Provide transparent bridge –Service integration below virtual hardware

5. Network Services Implement high level functions Operate on low level network traffic –Monitor –Control –Manipulate Traffic Data Signaling Unique challenges in Virtual Environments –E.g. Migration

6. Motivation HPDC 2005 -- VRESERVE –Automatic Optical Network Reservations for unmodified applications –Demonstrated performance gains over standard internet routes Performance Issues –TCP applications ill suited for optical networks J. Lange, A. Sundararaj, and P. Dinda, Automatic Dynamic Run-time Optical Network Reservations, Proceedings of the 14th IEEE International Symposium on High Performance Distributed Computing, (HPDC 2005)

7. TCP over Optical Networks Optical Networks have high BDPs –Bandwidth Delay Products –Very High bandwidth –Long distance High relative latency –TCP breaks down D. Petravick, Fermilab

8. Typical BDP values Assume endpoints are on opposite ends of the earth –Real world example: CERN and StarLight –Latency lower bound is ~60ms Half circumference of earth / Speed of light CERN FNAL has a measured ~60 ms delay –D. Petravick, Fermilab –Optical Networks currently operate at 10 Gbps But 1 GigE NICs are most common –TCP Window Size (BDP): 10 Gbps ~= 70 MB 1 Gbps ~= 7MB SACK lookups cause TCP timeouts –Window size  1

9. Transparently Optimize high BDP flows High performance protocols exist –UDT/SABUL, RBUDP, etc… –But applications must be configured for them Need method of transforming TCP to UDT –Opens UDT connections based on SYNs –Transmits data segments over UDT

10. VTL Transparent Network Service Framework –Network device interface –Packet modification and creation –Rapid prototyping and evaluation Capabilities –Virtual TCP endpoint –Transparent packet generator Acks, keep-alive –Packet header and content modifications –Not confined to virtual machines

11. VTL Components Network Interface API –Reads/Writes packets to/from network interfaces Packet Access API –Reading and writing packet data State Models –Maintain state of connection endpoints

12. Network Interface API Common interface for packet capture and injection –Virtual or Real devices –Unix or Windows Built on PCAP and libnet Operations –Connect/Disconnect –Read/Write –Packet notifications

13. Packet Access API Packet inspection and modification –Primitives to access standard fields Higher level functions built on primitives –Packet class queries –Field swapping –Header calculations –Derivative packet creation

14. Connection State Models Maintain and manipulate protocol state –Layered architecture Create packets belonging to a connection State kept for both connection endpoints –Generate packets from either endpoint API operation –Manual or packet based Model Initialization State Updates Packet Creation

15. Module VM Hosting Server VMM (VMWare, Xen, etc) VNET Host-only interface VNET Overlay VTL UDT Flow Over Optical Network VTL Configuration Physical interface Sundararaj, A., Gupta, A.,, and Dinda, P. Increasing application performance in virtual environments through run-time inference and adaptation. In Proc. of the 14th IEEE International Symposium on High Performance Distributed Computing (HPDC) (July 2005) (Windows or Unix)

16. Baseline Performance Limited by Network Interface API –Implemented in user space PCAP + libnet Experimental setup –Simple interface bridge (virtual->real) Xen bridge Single process (half duplex) Two processes (full duplex)

17. Baseline Performance Bandwidth (MB/s) Xen BridgeOne VTL Process Two VTL Processes Overhead Measurements

18. Protocol Transformation for High BDP networks Addresses performance of TCP over optical VTL allows transformation of TCP flows to other transport protocols VTL module acts as virtual TCP endpoint –Implements TCP states SYN sequence (open) FIN sequence (close) Data Transfer over new protocol (established)

19. Code Example – Creating Packets int create_data_pkt(vtl_model_t * model, char * data, int data_len) { RawEthernetPacket data_pkt; create_empty_pkt(model, &data_pkt, INBOUND_PKT); memcpy(TCP_DATA(data_pkt), data, data_len); compute_ip_len(&data_pkt, data_len); compute_ip_checksum(&data_pkt); compute_tcp_checksum(&data_pkt); sync_model(model, &data_pkt); queue_pkt(&data_pkt); }

20. Performance Evaluation Setup Comparing TCP vs. VTL + UDT Added artificial latency to gigabit switch –Linux iproute2 + tc netem TTCP benchmark –Standard TCP (Host to host) –TCP with intelligent socket buffers (Host to host) –VTL + UDT (Xen VM to Xen VM) Note: No virtualization present for TCP tests –Same hardware

21. Performance Bandwidth (MB/s) Latency (ms)

22. More Transparent Network Services Socks (TOR) Subnet Tunneling VM Migration Support (TCP keep alive) Stateful Firewall Performance Enhancing Proxies –RFC 3135 –Local acknowledgements

23. TOR NETWORK Tor Server VM Hosting Service VMM (VMWare, Xen, etc) VTL Host-only interface VTL Interface SOCKS Connection TCP Connections + DNS lookups Anonymous Networking for Any Application Tor Anonymous Network (http://tor.eff.org) –Anonymizes source of any TCP connection –Functions as a SOCKS proxy –Requires SOCKS application support

24. Tor + VTL VTL implements transparent SOCKS interface –VTL simulates a TCP endpoint –Extracts data segment from TCP packet and transmits it over SOCKS tunnel –Data from SOCKS is encapsulated into TCP packets and delivered to VM Gotchas –DNS is UDP based VTL handles DNS case for UDP –ARPs VTL answers ARPs with a fake MAC address All tcp connections from a VM are anonymized –No modification to OS or applications –User not restricted to applications implementing socks

25. Transparent Security Iptables and Windows Firewall are now ubiquitous –Not perfect Successful attacker can alter rules Only as strong as the weakest link VTL rules are not accessible by VM –Even if VM is compromised firewall rules are safe

26. VNET Overlay (Internet) VNET Proxy (PROXY2) VNET Proxy (PROXY1) Gateway (GW2) Gateway (GW1) Internet LAN connection 234.234.1.0/24 123.123.1.0/24 VM1 VM2 123.123.1.50 234.234.1.50 234.234.1.1 123.123.1.1 10.10.0.0/16 Gateway Router Subnet Tunneling MAC Address Mismatch!

27. Subnet Tunneling Two VMs on different subnets communicating –Fast Path link is available between them Bypasses routers VMs use subnet gateway –Set gateway MAC as destination VTL rewrites destination MAC addresses –Route packets on fast path link

28. Network Suspension during VM Migrations A VM is suspended for a long duration –i.e. VM is migrating over WAN –Open TCP connections begin to timeout In order to maintain connections VTL generates keep-alive packets Secondary service must handle routing –i.e. VNET

29. Cooperative Selective Wormholing Distributed traffic aggregation for Network Intrusion Detection Systems Wormhole –Tunnel traffic from a remote sensor to backend NIDS –VTL mechanisms for packet capture and injection Cooperative –Volunteer machines aggregate traffic –VTL implementation cross platform Selective –Aggregates traffic that Volunteer client is not interested in –VTL mechanisms for packet inspection J. Lange, P. Dinda, and F. Bustamante, Vortex: Enabling Cooperative Selective Wormholing for Network Security Systems, Proceedings of the 10th International Symposium on Recent Advances in Intrusion Detection (To Appear)

30. Future Work Generalizable to complete IO framework Performance –VMM based implementation Automatic Service Adaptation

31. Conclusion Transparent Network Services allow high level transformations of low level network traffic VTL –A framework for creating Transparent Network Services Wide range of potential services –Many useful for HPDC

32. Prescience Lab –http://plab.cs.northwestern.edu Virtuoso –http://virtuoso.cs.northwestern.edu John Lange –http://www.artifex.org/~jarusl

33. Vortex Cooperative Selective Wormhole implementation VTL –Traffic capture and injection –Packet modifications Rewrite addresses Anonymize packets –Cross platform functionality

34. Vortex VTL PCAPlibnet Firewall NIC VNET Proxy Apps IDS Analysis Backend VNET Overlay Windows/UNIX Commodity PC Operating System Physical Honeypot VM Based Honeypot VM Backend Network Vortex Architecture

35. VNET Overlay VNET Proxy (PROXY2) VNET Proxy (PROXY1) Gateway (GW2) Gateway (GW1) Internet LAN connection 234.234.1.0/24 123.123.1.0/24 VM1 VM2 123.123.1.50 234.234.1.50 234.234.1.1 123.123.1.1 10.10.0.0/16 Gateway Router Subnet Tunneling