VROOM: Virtual ROuters On the Move Yi Wang (Princeton) With: Eric Keller (Princeton) Brian Biskeborn (Princeton) Kobus van der Merwe (AT&T Labs - Research) Jennifer Rexford (Princeton)

Virtual ROuters On the Move (VROOM) Key idea Routers should be free to roam around Useful for many different applications Simplify network maintenance Simplify service deployment and evolution Reduce power consumption … Feasible in practice No performance impact on data traffic No visible impact on routing protocols This is the talk-in-one-slide. If you had to leave now, here is all the information I’ll be talking about… The key idea of VROOM is that routers should be free to roam around, instead of being permanently attached to specific piece of hardware. I hope by the end of this talk I’ll convince you that VROOM is useful for many network management tasks. Actually, it can not only save network operators a lot of headache in their daily jobs, it can even save you power! And we show through a prototype implementation that VROOM is actually feasible in practice.

VROOM: The Basic Idea Virtual routers (VRs) form logical topology 1 4 3 2 physical router 5 virtual router logical link Here is the basic idea of VROOM: virtual routers running on top of physical routers form the logical topology of a network. The physical routers only provide shared hardware resource and the necessary virtualization support. It is the virtual routers that run routing protocols and forward actual traffic.

VROOM: The Basic Idea VR migration does not affect the logical topology 2 3 physical router virtual router 1 logical link 4 In VROOM, virtual routers can migrate from one physical router to another, along with all the links attached to it. Therefore, no configurations within the virtual routers need to be changed, and the topology stays intact. For example, the logical topology among the five virtual routers remains the same before and after the migration. 5

The Rest of the Talk is Q&A Why is VROOM a good idea? What are the challenges? Or it is just technically trivial? How does VROOM work? The migration process Is VROOM practical? Prototype system Performance evaluation Where to migrate? The scheduling problem Still have questions? Feel free to ask! 5

The Coupling of Logical and Physical Today, the physical and logical configurations of a router is tightly coupled Physical changes break protocol adjacencies, disrupt traffic Logical configuration as a tool to reduce the disruption E.g., the “cost-out/cost-in” of IGP link weights Cannot eliminate the disruption Account for over 73% of network maintenance events So why is VROOM useful? What’s the rationale behind the idea of migrating routers? Today, the physical and logical configurations of a router is tightly coupled. For example, hardware upgrade to a router requires logical reconfiguration to routing protocols. And moving a customer from one physical router to another also requires re-configuration. But we all know that we are better off with the less re-configurations, because less re-configuration means less protocol reconvergence, less traffic disruption, and less configuration errors and overhead. Remember that operators errors account for more network outages than equipment failures.

VROOM Separates the Logical and Physical Make a logical router instance migratable among physical nodes All logical configurations/states remain the same before/after the migration IP addresses remain the same Routing protocol configurations remain the same Routing-protocol adjacencies stay up No protocol (BGP/IGP) reconvergence Network topology stays intact No disruption to data traffic Given such observations, VROOM separates the tight coupling between logical and physical by supporting live virtual router migration. The protocol configuration, network topology and traffic all stay intact during the process and after the migration. VROOM is useful for many difference applications. Here I’ll talk about two examples. In one example, VROOM greatly simplifies a conventional network management task. In another example, VROOM enables new application that is hard to do today.

Case 1: Planned Maintenance Today’s best practice: “cost-out/cost-in” Router reconfiguration & protocol reconvergence VROOM NO reconfiguration of VRs, NO reconvergence VR-1 PR-A PR-B

Case 1: Planned Maintenance Today’s best practice: “cost-out/cost-in” Router reconfiguration & protocol reconvergence VROOM NO reconfiguration of VRs, NO reconvergence PR-A VR-1 PR-B

Case 1: Planned Maintenance Today’s best practice: “cost-out/cost-in” Router reconfiguration & protocol reconvergence VROOM NO reconfiguration of VRs, NO reconvergence PR-A VR-1 PR-B

Case 2: Service Deployment & Evolution Deploy a new service in a controlled “test network” first CE CE CE Test network Test network Production network Test network

Case 2: Service Deployment & Evolution Roll out the service to the production network after it matures VROOM guarantees seamless service to existing customers during the roll-out and later evolution Test network Test network Production network Test network

Case 3: Power Savings Big power consumption of routers Millions of Routers in the U.S. Electricity bill: $ hundreds of millions/year (Source: National Technical Information Service, Department of Commerce, 2000. Figures for 2005 & 2010 are projections.)

Case 3: Power Savings Observation: the diurnal traffic pattern Idea: contract and expand the physical network according to the traffic demand

Case 3: Power Savings Dynamically contract & expand the physical network in a day - 3PM

Case 3: Power Savings Dynamically contract & expand the physical network in a day - 9PM

Case 3: Power Savings Dynamically contract & expand the physical network in a day - 4AM

Virtual Router Migration: the Challenges Migrate an entire virtual router instance All control plane & data plane processes / states Minimize disruption Data plane: up to millions packets per second Control plane: less stringent (w/ routing message retrans.) Migrate links 18 18

Outline Why is VROOM a good idea? What are the challenges? How does VROOM work? The migration enablers The migration process What to be migrated? How? (in order to minimize disruption) Is VROOM practical? Where to migrate?

VROOM Architecture Three enablers that make VR migration possible Router virtualization Control and data plane separation Dynamic interface binding The VROOM architecture has several key components that enable virtual router migration. First, since virtual routers are not permanently attached to a specific piece of hardware, the substrate layer of the physical router needs to provide dynamic bindings between the logical interfaces of a virtual router and the physical interfaces of underlying physical router.

A Naive Migration Process Freeze the virtual router Copy states Restart Migrate links Practically unacceptable Packet forwarding should not stop during migration Second, to maintain the logical topology, the links attached to a virtual router need to be migrated together with the virtual router to the new physical router. There are two ways to do this. The first approach is to leverage the programmable transport networks, which is commonly available in large backbone ISPs. A programmable transport network is capable of switching an optical path from one port to another in very short time. The reported theoretical bound is sub-nanoseconds. For ISPs that don’t have programmable transport networks, they can use tunnels to connect virtual routers. In this case, migrating a link only requires re-configuring its tunnel end point to the new physical router.

VROOM’s Migration Process Key idea: separate the migration of control and data plane No data-plane interruption Low control-plane interruption Control-plane migration Data-plane cloning Link migration Second, to maintain the logical topology, the links attached to a virtual router need to be migrated together with the virtual router to the new physical router. There are two ways to do this. The first approach is to leverage the programmable transport networks, which is commonly available in large backbone ISPs. A programmable transport network is capable of switching an optical path from one port to another in very short time. The reported theoretical bound is sub-nanoseconds. For ISPs that don’t have programmable transport networks, they can use tunnels to connect virtual routers. In this case, migrating a link only requires re-configuring its tunnel end point to the new physical router. 22 22

Control-Plane Migration Two things to be copied Router image Binaries, configuration files, etc. Memory 1st stage: pre-copy 2nd stage: stall-and-copy (when the control plane is “frozen”) 1 2 Second, to maintain the logical topology, the links attached to a virtual router need to be migrated together with the virtual router to the new physical router. There are two ways to do this. The first approach is to leverage the programmable transport networks, which is commonly available in large backbone ISPs. A programmable transport network is capable of switching an optical path from one port to another in very short time. The reported theoretical bound is sub-nanoseconds. For ISPs that don’t have programmable transport networks, they can use tunnels to connect virtual routers. In this case, migrating a link only requires re-configuring its tunnel end point to the new physical router. time t1 t2 t3 t4 pre-copy stall-and-copy 1: router-image copy 2: memory copy 23 23

Data-Plane Cloning Clone the data plane by repopulation Copying the data plane states is wasteful, and could be hard Instead, repopulate the new data plane using the migrated control plane The old data plane continues working during migration 1 2 3 Second, to maintain the logical topology, the links attached to a virtual router need to be migrated together with the virtual router to the new physical router. There are two ways to do this. The first approach is to leverage the programmable transport networks, which is commonly available in large backbone ISPs. A programmable transport network is capable of switching an optical path from one port to another in very short time. The reported theoretical bound is sub-nanoseconds. For ISPs that don’t have programmable transport networks, they can use tunnels to connect virtual routers. In this case, migrating a link only requires re-configuring its tunnel end point to the new physical router. time t1 t2 t3 t4 t5 1: router-image copy 2: memory copy 3: data-plane cloning 24 24

Remote Control Plane The migrated control plane plays two roles Act as a “remote control plane” for the old data plane Populate the new data plane 1 2 3 time t1 t2 t3 t4 t5 remote control plane control plane old node new node Second, to maintain the logical topology, the links attached to a virtual router need to be migrated together with the virtual router to the new physical router. There are two ways to do this. The first approach is to leverage the programmable transport networks, which is commonly available in large backbone ISPs. A programmable transport network is capable of switching an optical path from one port to another in very short time. The reported theoretical bound is sub-nanoseconds. For ISPs that don’t have programmable transport networks, they can use tunnels to connect virtual routers. In this case, migrating a link only requires re-configuring its tunnel end point to the new physical router. 1: router-image copy 2: memory copy 3: data-plane cloning 25 25

Keep the Control Plane “Online” Data-plane cloning takes time Around 110 us per FIB entry update (for high-end router) * Installing 250k routes could take over 20 seconds The control plane needs connectivity during this period Redirect the routing messages through tunnels Second, to maintain the logical topology, the links attached to a virtual router need to be migrated together with the virtual router to the new physical router. There are two ways to do this. The first approach is to leverage the programmable transport networks, which is commonly available in large backbone ISPs. A programmable transport network is capable of switching an optical path from one port to another in very short time. The reported theoretical bound is sub-nanoseconds. For ISPs that don’t have programmable transport networks, they can use tunnels to connect virtual routers. In this case, migrating a link only requires re-configuring its tunnel end point to the new physical router. *: P. Francios, et. al., Achieving sub-second IGP convergence in large IP networks, ACM SIGCOMM CCR, no. 3, 2005. 26 26

Double Data Planes At the end of data-plane cloning, two data planes are ready to forward traffic (i.e., “double data planes”) 1 2 3 4 time t0 t1 t2 t3 t4 t5 t6 remote control plane control plane old node new node Second, to maintain the logical topology, the links attached to a virtual router need to be migrated together with the virtual router to the new physical router. There are two ways to do this. The first approach is to leverage the programmable transport networks, which is commonly available in large backbone ISPs. A programmable transport network is capable of switching an optical path from one port to another in very short time. The reported theoretical bound is sub-nanoseconds. For ISPs that don’t have programmable transport networks, they can use tunnels to connect virtual routers. In this case, migrating a link only requires re-configuring its tunnel end point to the new physical router. old node data plane new node 0: tunnel setup double data plane 1: router-image copy 2: memory copy 3: data-plane cloning 4: asynchronous link migration 27 27

Asynchronous Link Migration With the double data planes, each link can be migrated independently Eliminate the need for a synchronization system Second, to maintain the logical topology, the links attached to a virtual router need to be migrated together with the virtual router to the new physical router. There are two ways to do this. The first approach is to leverage the programmable transport networks, which is commonly available in large backbone ISPs. A programmable transport network is capable of switching an optical path from one port to another in very short time. The reported theoretical bound is sub-nanoseconds. For ISPs that don’t have programmable transport networks, they can use tunnels to connect virtual routers. In this case, migrating a link only requires re-configuring its tunnel end point to the new physical router. 28 28

Outline Why is VROOM a good idea? What are the challenges? How does VROOM work? Is VROOM practical? Prototype system Performance evaluation Where to migrate?

Prototype Implementation PC + OpenVZ OpenVZ: OS-level virtualization Lighter-weight Supports live migration Two prototypes Software-based data plane (SD): Linux kernel Hardware-based data plane (HD): NetFPGA NetFPGA: 4-port gigabit Ethernet PCI with an FPGA Why two prototypes? To validate the data-plane hypervisor design (e.g., migration between SD and HD) 30 30

The Out-of-box OpenVZ Approach Packets are forwarded inside each VE When a VE is being migrated, packets are dropped 31 31

Control and Data Plane Separation Move the FIBs out of the VEs shadowd in each VE, “pushing down” route updates virtd in VE0, as the “data-plane hypervisor” 32 32

Dynamic Interface Binding bindd provides two types of bindings: Map substrate interfaces to the right FIB Map substrate interfaces to the right virtual interfaces 33 33

Putting It Altogether: Realizing Migration 1. The migration program notifies shadowd about the completion of the control plane migration 34 34

Putting It Altogether: Realizing Migration 2. shadowd requests zebra to resend all the routes, and pushes them down to virtd 35 35

Putting It Altogether: Realizing Migration 3. virtd installs routes the new FIB, while continuing to update the old FIB 36 36

Putting It Altogether: Realizing Migration 4. virtd notifies the migration program to start link migration after finishing populating the new FIB 5. After link migration is completed, the migration program notifies virtd to stop updating the old FIB 37 37

Evaluation Answer three questions Experiments on Emulab Performance of individual migration steps? Impact on data traffic? Impact on routing protocol? Experiments on Emulab 38 38

Performance of Migration Steps Memory copy time With different numbers of routes (dump file sizes) 39 39

Performance of Migration Steps FIB population time Grows linearly w.r.t. the number of route entries Installing a FIB entry into NetFPGA: 7.4 microseconds Installing a FIB entry into Linux kernel: 1.94 milliseconds FIB update time: time for virtd to install entries to FIB Total time: FIB update time + time for shadowd to send routes to virtd 40 40

Data Plane Impact The diamond testbed 64-byte UDP packets, round-trip traffic 41 41

Data Plane Impact HD router with separate migration bandwidth No delay increase or packet loss SD router with separate migration bandwidth Up to 3.7% delay increase at 5k packets/s Less than 0.4% delay increase at 25k packets/s SD, 5k packets/s 42 42

The Importance of Separate Migration Bandwidth The dumbbell testbed 250k routes in the RIB 43 43

Separate Migration Bandwidth is Important Throughput of the migration traffic 44 44

Separate Migration Bandwidth is Important Delay increase of the data traffic 45 45

Separate Migration Bandwidth is Important Loss rate of the data traffic 46 46

Control Plane Impact The Abilene testbed Assume a backbone running MPLS VR5 configured as Core router (running OSPF only) Edge router (running OSPF + BGP) 47 47

Core Router Migration No events during migration Average control plane downtime: 0.972 seconds (0.924 - 1.008 seconds in 10 runs) Support 1-second OSPF hello-interval (with 4-second dead-interval) Miss at most one hello message 48 48

Core Router Migration Events happen during migration Introducing events (LSA) by flapping link VR2-VR3 Miss at most one LSA Get retransmission 5 seconds later (the default LSA retransmission-interval) Can use smaller LSA retransmission-interval (e.g., 1 second) 49 49

Edge Router Migration 255k BGP routes + OSPF Dump file size grows from 3.2MB to 76.0MB Average control plane downtime: 3.560 seconds (3.484 - 3.594 seconds in 10 runs) Support 2-second OSPF hello-interval (with 8-second dead-interval) BGP sessions stay up In practice, ISPs often use the default values 10-second hello-interval 40-second dead interval 50 50

Outline Why is VROOM a good idea? What are the challenges? How does VROOM work? Is VROOM practical? Where to migrate?

Deciding Where To Migrate Physical constraints Latency E.g, NYC to Washington D.C.: 2 msec Link capacity Enough remaining capacity for extra traffic Platform compatibility Routers from different vendors Router capability E.g., number of access control lists (ACLs) supported Good news: these constraints limit the search space

Two Optimization Problems For planned maintenance/service deployment Minimize path stretch With constraints on link capacity, platform compatibility, router capability, etc. For power savings Maximize power savings With different regional electricity prices With constraints on path stretch, link capacity, etc. 53 53

Conclusions VROOM offers a useful network-management primitive separates the tight coupling between physical and logical Simplify network management, enable new applications Live router migration with minimal disruption Data-plane hypervisor enables Data-plane cloning Remote control plane Double data plane and asynchronous link migration No data-plane disruption No visible control-plane disruption

Questions & Comments Please! Thanks! Questions & Comments Please!

Backup Slides

Packet-aware Access Network

Packet-aware Access Network Pseudo-wires (virtual circuits) from CE to PE PE CE P/G-MSS: Packet-aware/Gateway Multi-Service Switch MSE: Multi-Service Edge

Events During Migration Network failure during migration The old VR image is not deleted until the migration is confirmed successful Routing messages arrive during the migration of the control plane BGP: TCP retransmission OSPF: LSA retransmission

Flexible Transport Networks Migrate links affixed to the virtual routers Enabled by: programmable transport networks Long-haul links are reconfigurable Layer 3 point-to-point links are multi-hop at layer 1/2 New York Chicago Programmable Transport Network Washington D.C. : Multi-service optical switch (e.g., Ciena CoreDirector) 60 60

Requirements & Enabling Technologies Migrate links affixed to the virtual routers Enabled by: programmable transport networks Long-haul links are reconfigurable Layer 3 point-to-point links are multi-hop at layer 1/2 New York Chicago Programmable Transport Network Washington D.C. : Multi-service optical switch (e.g., Ciena CoreDirector)

Requirements & Enabling Technologies Enable edge router migration Enabled by: packet-aware access networks Access links are becoming inherently virtualized Customers connects to provider edge (PE) routers via pseudo-wires (virtual circuits) Physical interfaces on PE routers can be shared by multiple customers Dedicated physical interface per customer Shared physical interface

Link Migration in Transport Networks With programmable transport networks, long-haul links are reconfigurable IP-layer point-to-point links are multi-hop at transport layer VROOM leverages this capability in a new way to enable link migration 63 63

Link Migration in Flexible Transport Networks 2. With packet-aware transport networks Logical links share the same physical port Packet-aware access network (pseudo wires) Packet-aware IP transport network (tunnels) 64 64

Power Consumption of Routers Vendor Cisco Juniper Model CRS-1 12416 7613 T1600 T640 M320 Power (watt) 10,920 4,212 4,000 9,100 6,500 3,150 A Synthetic large tier-1 ISP backbone 50 POPs (Point-of-Presence) 20 major POPs, each has: 6 backbone routers, 6 peering routers, 30 access routers 30 smaller POPs, each has: 6 access routers

Future Work Algorithms that solve the constrained optimization problems Control-plane hypervisor to enable cross-vendor migration 66 66