VINI: Virtual Network Infrastructure Nick Feamster Georgia Tech Andy Bavier, Mark Huang, Larry Peterson, Jennifer Rexford Princeton University

VINI Overview Runs real routing software Bridge the gap between “lab experiments” and live experiments at scale. ? Emulation VINI Simulation Small-scale experiment Live deployment Runs real routing software Exposes realistic network conditions Gives control over network events Carries traffic on behalf of real users Is shared among many experiments

Goal: Control and Realism Traffic Real clients, servers Synthetic or traces Goal: Control and Realism Control Reproduce results Methodically change or relax constraints Realism Long-running services attract real users Connectivity to real Internet Forward high traffic volumes (Gb/s) Handle unexpected events Topology Arbitrary, emulated Actual network Traffic Synthetic or traces Real clients, servers Scientifically, what’s the point of such an infrastructure? Network Events Inject faults, anomalies Observed in operational network

Overview VINI characteristics PL-VINI: prototype on PlanetLab Fixed, shared infrastructure Flexible network topology Expose/inject network events External connectivity and routing adjacencies PL-VINI: prototype on PlanetLab Preliminary Experiments Ongoing work

Fixed Infrastructure Concretely, what is VINI Overlays! Aggregate Gb/s Topology, IP address, BGP peering

Shared Infrastructure Virtual nodes Isolation between experiments (e.g., CPU reservation)

Arbitrary Virtual Topologies P2P connectivity Unique interfaces visible to each virtual node

Exposing and Injecting Failures Concretely, what is VINI Overlays! Aggregate Gb/s Topology, IP address, BGP peering

Carry Traffic for Real End Users Talk about flexible routing and forwarding here s

Participate in Internet Routing BGP c Find out about external address blocks In future: advertise address blocks s

PL-VINI: Prototype on PlanetLab First experiment: Internet In A Slice XORP open-source routing protocol suite (NSDI ’05) Click modular router (TOCS ’00, SOSP ’99) Clarify issues that VINI must address Unmodified routing software on a virtual topology Forwarding packets at line speed Illusion of dedicated hardware Injection of faults and other events

PL-VINI: Prototype on PlanetLab PlanetLab: testbed for planetary-scale services Simultaneous experiments in separate VMs Each has “root” in its own VM, can customize Can reserve CPU, network capacity per VM Virtual Machine Monitor (VMM) (Linux++) Node Mgr Local Admin VM1 VM2 VMn … PlanetLab node

XORP: Control Plane BGP, OSPF, RIP, PIM-SM, IGMP/MLD XORP Goal: run real routing protocols on virtual network topologies XORP (routing protocols)

User-Mode Linux: Environment Interface ≈ network PlanetLab limitation: Slice cannot create new interfaces Run routing software in UML environment Create virtual network interfaces in UML UML XORP (routing protocols) eth0 eth1 eth2 eth3

Click: Data Plane Performance XORP Interfaces  tunnels Filters Click Avoid UML overhead Move to kernel, FPGA Interfaces  tunnels Click UDP tunnels correspond to UML network interfaces Filters “Fail a link” by blocking packets at tunnel UML XORP (routing protocols) eth0 eth1 eth2 eth3 Control Data Packet Forward Engine UmlSwitch element Tunnel table Click Filters

Intra-domain Route Changes 2095 856 700 260 233 1295 c 639 366 548 587 846 Talk about flexible routing and forwarding here 902 1893 1176

Ping During Link Failure Routes converging Link down Link up

Close-Up of TCP Transfer PL-VINI enables a user-space virtual network to behave like a real network on PlanetLab Slow start Retransmit lost packet

Challenge: Attracting Real Users Could have run experiments on Emulab Goal: Operate our own virtual network Carrying traffic for actual users We can tinker with routing protocols Attracting real users

Conclusion http://www.vini-veritas.net/ VINI: Controlled, Realistic Experimentation Installing VINI nodes in NLR, Abilene Download and run Internet In A Slice http://www.vini-veritas.net/

TCP Throughput Link up Link down Zoom in

Ongoing Work Improving realism Improving control Exposing network failures and changes in the underlying topology Participating in routing with neighboring networks Improving control Better isolation Experiment specification

Resource Isolation Issue: Forwarding packets in user space PlanetLab sees heavy use CPU load affects virtual network performance Property Depends On Solution Throughput CPU% received PlanetLab provides CPU reservations Latency CPU scheduling delay PL-VINI: boost priority of packet forward process

Performance is bad User-space Click: ~200Mb/s forwarding

VINI should use Xen

Experimental Results Is a VINI feasible? Click in user-space: 200Mb/s forwarded Latency and jitter comparable between network and IIAS on PL-VINI. Say something about running on just PlanetLab? Don’t spend much time talking about CPU scheduling…

Low latency for everyone? PL-VINI provided IIAS with low latency by giving it high CPU scheduling priority

Internet In A Slice XORP Run OSPF Configure FIB Click FIB Tunnels Inject faults OpenVPN & NAT Connect clients and servers S S C C Talk about flexible routing and forwarding here S C

PL-VINI / IIAS Router Blue: topology XORP Red: routing and forwarding Virtual net devices Tunnels Red: routing and forwarding Data traffic does not enter UML Green: enter & exit IIAS overlay UML XORP eth1 eth3 eth2 UmlSwitch element FIB Encapsulation table eth0 Control Data Click tap0

PL-VINI Summary Flexible Network Topology Virtual point-to-point connectivity Tunnels in Click Unique interfaces per experiment Virtual network devices in UML Exposure of topology changes Upcalls of layer-3 alarms Flexible Routing and Forwarding Per-node forwarding table Separate Click per virtual node Per-node routing process Separate XORP per virtual node Connectivity to External Hosts End-hosts can direct traffic through VINI Connect to OpenVPN server Return traffic flows through VINI NAT in Click on egress node Support for Simultaneous Experiments Isolation between experiments PlanetLab VMs and network isolation CPU reservations and priorities Distinct external routing adjacencies BGP multiplexer for external sessions

PL-VINI / IIAS Router XORP: control plane UML: environment XORP Virtual interfaces Click: data plane Performance Avoid UML overhead Move to kernel, FPGA Interfaces  tunnels “Fail a link” UML XORP (routing protocols) eth0 eth1 eth2 eth3 Control Data Packet Forward Engine UmlSwitch element Tunnel table Click