VROOM: Virtual ROuters On the Move Jennifer Rexford Joint work with Yi Wang, Eric Keller, Brian Biskeborn, and Kobus van der Merwe (AT&T)

Virtual ROuters On the Move Key idea – Routers should be free to roam around Useful for many different applications – Simplify network maintenance – Simplify service deployment and evolution – Reduce power consumption –…–… Feasible in practice – No performance impact on data traffic – No visible impact on routing protocols 2

The Two Notions of “Router” IP-layer logical functionality, and physical equipment 3 Logical (IP layer) Physical

Tight Coupling of Physical & Logical Root of many network-management challenges (and “point solutions”) 4 Logical (IP layer) Physical

VROOM: Breaking the Coupling Re-mapping logical node to another physical node 5 Logical (IP layer) Physical VROOM enables this re-mapping of logical to physical through virtual router migration.

Case 1: Planned Maintenance NO reconfiguration of VRs, NO reconvergence 6 A B VR-1

Case 1: Planned Maintenance NO reconfiguration of VRs, NO reconvergence 7 A B VR-1

Case 1: Planned Maintenance NO reconfiguration of VRs, NO reconvergence 8 A B VR-1

Case 2: Service Deployment/Evolution Move (logical) router to more powerful hardware 9

Case 2: Service Deployment/Evolution VROOM guarantees seamless service to existing customers during the migration 10

Case 3: Power Savings 11 $ Hundreds of millions/year of electricity bills

Case 3: Power Savings 12 Contract and expand the physical network according to the traffic volume

Case 3: Power Savings 13 Contract and expand the physical network according to the traffic volume

Case 3: Power Savings 14 Contract and expand the physical network according to the traffic volume

Virtual Router Migration: Challenges 15 1.Migrate an entire virtual router instance All control plane & data plane processes / states

Virtual Router Migration: Challenges 16 1.Migrate an entire virtual router instance 2.Minimize disruption Data plane: millions of packets/sec on a 10Gbps link Control plane: less strict (with routing message retrans.)

Virtual Router Migration: Challenges 17 1.Migrating an entire virtual router instance 2.Minimize disruption 3.Link migration

Virtual Router Migration: Challenges 18 1.Migrating an entire virtual router instance 2.Minimize disruption 3.Link migration

VROOM Architecture 19 Dynamic Interface Binding Data-Plane Hypervisor

Key idea: separate the migration of control and data planes 1.Migrate the control plane 2.Clone the data plane 3.Migrate the links 20 VROOM’s Migration Process

Leverage virtual server migration techniques Router image – Binaries, configuration files, etc. 21 Control-Plane Migration

Leverage virtual server migration techniques Router image Memory – 1 st stage: iterative pre-copy – 2 nd stage: stall-and-copy (when the control plane is “frozen”) 22 Control-Plane Migration

Leverage virtual server migration techniques Router image Memory 23 Control-Plane Migration Physical router A Physical router B DP CP

Clone the data plane by repopulation – Enable migration across different data planes – Avoid copying duplicate information 24 Data-Plane Cloning Physical router A Physical router B CP DP-old DP-new

Data-plane cloning takes time – Installing 250k routes may take several seconds Control & old data planes need to be kept “online” Solution: redirect routing messages through tunnels 25 Remote Control Plane Physical router A Physical router B CP DP-old DP-new

Data-plane cloning takes time – Installing 250k routes takes over 20 seconds Control & old data planes need to be kept “online” Solution: redirect routing messages through tunnels 26 Remote Control Plane Physical router A Physical router B CP DP-old DP-new

Data-plane cloning takes time – Installing 250k routes takes over 20 seconds Control & old data planes need to be kept “online” Solution: redirect routing messages through tunnels 27 Remote Control Plane Physical router A Physical router B CP DP-old DP-new

At the end of data-plane cloning, both data planes are ready to forward traffic 28 Double Data Planes CP DP-old DP-new

With the double data planes, links can be migrated independently 29 Asynchronous Link Migration A CP DP-old DP-new B

Virtualized operating system – OpenVZ, supports VM migration Routing protocols – Quagga software suite Packet forwarding – Linux kernel (software), NetFPGA (hardware) Router hypervisor – Our extensions for repopulating data plane, remote control plane, double data planes, … 30 Prototype Implementation

Experiments in Emulab – On realistic Abilene Internet2 topology 31 Experimental Evaluation

Data traffic – Linux: modest packet delay due to CPU load – NetFPGA: no packet loss or extra delay Routing-protocol messages – Core router migration (OSPF only) Inject an unplanned link failure at another router At most one retransmission of an OSPF message – Edge router migration (OSPF + BGP) Control-plane downtime: 3.56 seconds Within reasonable keep-alive timer intervals – All routing-protocol adjacencies stay up 32 Experimental Results

Where To Migrate Physical constraints – Latency E.g, NYC to Washington D.C.: 2 msec – Link capacity Enough remaining capacity for extra traffic – Platform compatibility Routers from different vendors – Router capability E.g., number of access control lists (ACLs) supported Constraints simplify the placement problem – By limiting the size of the search space 33

Conclusions on VROOM VROOM: useful network-management primitive – Separate tight coupling between physical and logical – Simplify network management, enable new applications Evaluation of prototype – No disruption in packet forwarding – No noticeable disruption in routing protocols Future work – Migration scheduling as an optimization problem – Extensions to router hypervisor for other applications 34

Other Projects Related to Router Virtualization

Bug-Tolerant Routers Seriousness of routing software bugs – Cause serious outages, misbehavior, vulnerability – Violate protocol semantics, so not handled by traditional failure detection and recovery Handling bugs at run time – Run multiple routing instances in parallel – Use different execution environments, message timings/orderings, or code bases – Vote on “answers” forwarding-table entries and messages to neighboring routers Collaboration with Matt Caesar and Yuanyuan Zhou at UIUC

Virtual Network Infrastructure (VINI) Experimental platform for network research – Evaluating prototypes of network architectures – Supporting multiple experiments in parallel – Carrying real user traffic & connecting to Internet VINI platform ( – Virtual nodes, links, and network stack in Linux – Instantiation of virtual topology for experimenters – VINI nodes deployed in NLR and Internet2 Collaboration with Nick Feamster (GA Tech) and Andy Bavier and Larry Peterson (Princeton)

Concurrent Architectures are Better than One (CABO) Overcome limitations of today’s Internet – Applications with diverse requirements – Too many (sometimes conflicting) goals – Difficulty of coordinating across domains New architecture based on virtualization – Infrastructure providers: own and manage equipment, and host virtual nodes and links – Service providers: run virtual networks customized to their end-to-end services Collaboration with Nick Feamster (GA Tech) and Lixin Gao (UMass)

Dynamically Adaptive Virtual Networks for a Customized Internet (DaVinci) Multiple applications with different goals – E.g., throughput-sensitive and delay-sensitive – Want to operate customized network protocols How to allocate bandwidth between classes? – Static is inefficient, but dynamic may be risky Theoretical foundation based on optimization – Customized protocol for each traffic class – Dynamic bandwidth allocation rule for each link – Provably maximizes aggregate performance Collaboration with Mung Chiang (Princeton)

Conclusions Router virtualization is exciting – Enables wide variety of new networking techniques – … for network management & service deployment – … and even rethinking the Internet architecture Fascinating space of open questions – What is the right interface to router hardware? – What is the right programming environment for customized protocols on virtual networks? Looking forward to talking more with Juniper!