1. Virtual ROuters On the Move (VROOM): Live Router Migration as a Network-Management Primitive
Yi Wang, Eric Keller, Brian Biskeborn,
Kobus van der Merwe, Jennifer Rexford

2. Virtual ROuters On the Move (VROOM)
Key idea
Routers should be free to roam around
Useful for many different applications
Simplify network maintenance
Simplify service deployment and evolution
Reduce power consumption …
Feasible in practice
No performance impact on data traffic
No visible impact on control-plane protocols
The key idea of VROOM is that routers should be free to roam around, instead of being permanently attached to a specific piece of hardware.
In this talk, I’ll first show that VROOM is useful for many different network management applications, such as simplifying network maintenance, simplifying service deployment and evolution. In fact, it can also save us power.
I will then show through our prototype implementation and evaluation that VROOM is actually feasible in practice, with no performance impact on data traffic and no visible impact on routing protocols.

3. The Two Notions of “Router”
The IP-layer logical functionality, and the physical equipment
Logical
(IP layer)
Here is the basic idea of VROOM: virtual router instances running on top of physical routers form the logical topology of a network.
The physical routers only provide shared hardware resource and the necessary virtualization support. It is the virtual routers that run routing protocols and forward actual traffic.
Physical

4. The Tight Coupling of Physical & Logical
Root of many network-management challenges (and “point solutions”)
Logical
(IP layer)
They are basically the same thing in people’s mind. When we think of a node in a topology, we also have the physical box in our mind, and vice versa.
Physical

5. VROOM: Breaking the Coupling
Re-mapping the logical node to another physical node
VROOM enables this re-mapping of logical to physical through virtual router migration.
Logical
(IP layer)
The mapping can change, maintaining the IP layer logical topology and configuration intact.
Each virtual router has its own routing protocol instances and its own forwarding tables. A physical router can support multiple virtual router instances through virtualization.
Physical

6. Case 1: Planned Maintenance
NO reconfiguration of VRs, NO reconvergence
VR-1 A B

7. Case 1: Planned Maintenance
NO reconfiguration of VRs, NO reconvergence A
VR-1 B

8. Case 1: Planned Maintenance
NO reconfiguration of VRs, NO reconvergence A
VR-1 B

9. Case 2: Service Deployment & Evolution
Move a (logical) router to more powerful hardware
In ISPs, as a service grows, it may need to be moved to a more powerful router. Today, this process usually involves certain period of downtime.

10. Case 2: Service Deployment & Evolution
VROOM guarantees seamless service to existing customers during the migration

11. Case 3: Power Savings
$ Hundreds of millions/year of electricity bills

12. Case 3: Power Savings
Contract and expand the physical network according to the traffic volume

13. Case 3: Power Savings
Contract and expand the physical network according to the traffic volume

14. Case 3: Power Savings
Contract and expand the physical network according to the traffic volume

15. Virtual Router Migration: the Challenges
Migrate an entire virtual router instance
All control plane & data plane processes / states

16. Virtual Router Migration: the Challenges
Migrate an entire virtual router instance
Minimize disruption
Data plane: millions of packets/second on a 10Gbps link
Control plane: less strict (with routing message retrans.)

17. Virtual Router Migration: the Challenges
Migrating an entire virtual router instance
Minimize disruption
Link migration
We need to migrate all the links associated with the virtual router as well. To do this in a seamless fashion, we leverage the fact that in ISPs a point-to-point link at the IP layer is usually a multi-hop path in the underlying transport network. The advances of the transport network now offer the capability to dynamically setup a new optical path and switch the old path to the new path virtually instantaneously. This allows us to realize link migration by configuring the optical transport networks.

18. Virtual Router Migration: the Challenges
Migrating an entire virtual router instance
Minimize disruption
Link migration

19. Data-Plane Hypervisor Dynamic Interface Binding
VROOM Architecture
Data-Plane Hypervisor
Dynamic Interface Binding

20. VROOM’s Migration Process
Key idea: separate the migration of control and data planes
Migrate the control plane
Clone the data plane
Migrate the links

21. Control-Plane Migration
Leverage virtual server migration techniques
Router image
Binaries, configuration files, etc.

22. Control-Plane Migration
Leverage virtual migration techniques
Router image
Memory
1st stage: iterative pre-copy
2nd stage: stall-and-copy (when the control plane is “frozen”)
During the first iteration, all pages are transferred from A to B. Subsequent iterations copy only those pages dirtied during the previous transfer phase.

23. Control-Plane Migration
Leverage virtual server migration techniques
Router image
Memory CP
Physical router A DP
Physical router B

24. Data-Plane Cloning Clone the data plane by repopulation
Enable migration across different data planes
Eliminate synchronization issue of control & data planes
Physical router A
DP-old CP
Physical router B
DP-new
DP-new

25. Remote Control Plane Data-plane cloning takes time
Installing 250k routes takes over 20 seconds*
The control & old data planes need to be kept “online”
Solution: redirect routing messages through tunnels
Physical router A
DP-old CP
Physical router B
DP-new
*: P. Francios, et. al., Achieving sub-second IGP convergence in large IP networks,
ACM SIGCOMM CCR, no. 3, 2005.

26. Remote Control Plane Data-plane cloning takes time
Installing 250k routes takes over 20 seconds*
The control & old data planes need to be kept “online”
Solution: redirect routing messages through tunnels
Physical router A
DP-old CP
Physical router B
DP-new
*: P. Francios, et. al., Achieving sub-second IGP convergence in large IP networks,
ACM SIGCOMM CCR, no. 3, 2005.

27. Remote Control Plane Data-plane cloning takes time
Installing 250k routes takes over 20 seconds*
The control & old data planes need to be kept “online”
Solution: redirect routing messages through tunnels
Physical router A
DP-old CP
Physical router B
DP-new
*: P. Francios, et. al., Achieving sub-second IGP convergence in large IP networks,
ACM SIGCOMM CCR, no. 3, 2005.

28. Double Data Planes
At the end of data-plane cloning, both data planes are ready to forward traffic
DP-old CP
DP-new

29. Asynchronous Link Migration
With the double data planes, links can be migrated independently
DP-old A B
The double data planes simplify link migration because they enable the links to be migrated independently. For example, in this simple example, to migrate the links, we can first set up the links from the new physical nodes to the two adjacent nodes. We can then migrate the traffic in each direction separately. CP
DP-new

30. Prototype Implementation
Control plane: OpenVZ + Quagga
Data plane: two prototypes
Software-based data plane (SD): Linux kernel
Hardware-based data plane (HD): NetFPGA
Why two prototypes?
To validate the data-plane hypervisor design (e.g., migration between SD and HD)

31. Evaluation Performance of individual migration steps
Impact on data traffic
Impact on routing protocols
Experiments on Emulab

32. Evaluation Performance of individual migration steps
Impact on data traffic
Impact on routing protocols
Experiments on Emulab

33. Impact on Data Traffic
The diamond testbed VR n1 n0 n3 n2

34. Impact on Data Traffic SD router w/ separate migration bandwidth
Slight delay increase due to CPU contention
HD router w/ separate migration bandwidth
No delay increase or packet loss

35. Impact on Routing Protocols
The Abilene-topology testbed

36. Core Router Migration: OSPF Only
Introduce LSA by flapping link VR2-VR3
Miss at most one LSA
Get retransmission 5 seconds later (the default LSA retransmission timer)
Can use smaller LSA retransmission-interval (e.g., 1 second)

37. Edge Router Migration: OSPF + BGP
Average control-plane downtime: 3.56 seconds
Performance lower bound
OSPF and BGP adjacencies stay up
Default timer values
OSPF hello interval: 10 seconds
BGP keep-alive interval: 60 seconds

38. Where To Migrate Physical constraints
Latency
E.g, NYC to Washington D.C.: 2 msec
Link capacity
Enough remaining capacity for extra traffic
Platform compatibility
Routers from different vendors
Router capability
E.g., number of access control lists (ACLs) supported
The constraints simplify the placement problem

39. Conclusions & Future Work
VROOM: a useful network-management primitive
Separate tight coupling between physical and logical
Simplify network management, enable new applications
No data-plane and control-plane disruption
Future work
Migration scheduling as an optimization problem
Other applications of router migration
Handle unplanned failures
Traffic engineering

40. Thanks!
Questions & Comments?

41. Packet-aware Access Network

42. Packet-aware Access Network
Pseudo-wires (virtual circuits) from CE to PE PE CE
P/G-MSS: Packet-aware/Gateway Multi-Service Switch
MSE: Multi-Service Edge

43. Events During Migration
Network failure during migration
The old VR image is not deleted until the migration is confirmed successful
Routing messages arrive during the migration of the control plane
BGP: TCP retransmission
OSPF: LSA retransmission

44. Flexible Transport Networks
Migrate links affixed to the virtual routers
Enabled by: programmable transport networks
Long-haul links are reconfigurable
Layer 3 point-to-point links are multi-hop at layer 1/2
New York
Chicago
Programmable Transport Network
Washington D.C.
: Multi-service optical switch (e.g., Ciena CoreDirector) 44 44

45. Requirements & Enabling Technologies
Migrate links affixed to the virtual routers
Enabled by: programmable transport networks
Long-haul links are reconfigurable
Layer 3 point-to-point links are multi-hop at layer 1/2
New York
Chicago
Programmable Transport Network
Washington D.C.
: Multi-service optical switch (e.g., Ciena CoreDirector)

46. Requirements & Enabling Technologies
Enable edge router migration
Enabled by: packet-aware access networks
Access links are becoming inherently virtualized
Customers connects to provider edge (PE) routers via pseudo-wires (virtual circuits)
Physical interfaces on PE routers can be shared by multiple customers
Dedicated physical interface
per customer
Shared physical interface

47. Link Migration in Transport Networks
With programmable transport networks, long-haul links are reconfigurable
IP-layer point-to-point links are multi-hop at transport layer
VROOM leverages this capability in a new way to enable link migration 47 47

48. Link Migration in Flexible Transport Networks
2. With packet-aware transport networks
Logical links share the same physical port
Packet-aware access network (pseudo wires)
Packet-aware IP transport network (tunnels) 48 48

49. The Out-of-box OpenVZ Approach
Packets are forwarded inside each VE
When a VE is being migrated, packets are dropped 49 49

50. Putting It Altogether: Realizing Migration
1. The migration program notifies shadowd about the completion of the control plane migration 50 50

51. Putting It Altogether: Realizing Migration
2. shadowd requests zebra to resend all the routes, and pushes them down to virtd 51 51

52. Putting It Altogether: Realizing Migration
3. virtd installs routes the new FIB, while continuing to update the old FIB 52 52

53. Putting It Altogether: Realizing Migration
4. virtd notifies the migration program to start link migration after finishing populating the new FIB
5. After link migration is completed, the migration program notifies virtd to stop updating the old FIB 53 53

54. Power Consumption of Routers
Vendor
Cisco
Juniper
Model
CRS-1
12416
7613
T1600
T640
M320
Power
(watt)
10,920
4,212
4,000
9,100
6,500
3,150
A Synthetic large tier-1 ISP backbone
50 POPs (Point-of-Presence)
20 major POPs, each has:
6 backbone routers, 6 peering routers, 30 access routers
30 smaller POPs, each has:
6 access routers

55. Future Work
Algorithms that solve the constrained optimization problems
Control-plane hypervisor to enable cross-vendor migration 55 55

56. Performance of Migration Steps
Memory copy time
With different numbers of routes (dump file sizes) 56 56

57. Performance of Migration Steps
FIB population time
Grows linearly w.r.t. the number of route entries
Installing a FIB entry into NetFPGA: 7.4 microseconds
Installing a FIB entry into Linux kernel: 1.94 milliseconds
FIB update time: time for virtd to install entries to FIB
Total time: FIB update time + time for shadowd to send routes to virtd 57 57

58. The Importance of Separate Migration Bandwidth
The dumbbell testbed
250k routes in the RIB 58 58

59. Separate Migration Bandwidth is Important
Throughput of the migration traffic 59 59

60. Separate Migration Bandwidth is Important
Delay increase of the data traffic 60 60

61. Separate Migration Bandwidth is Important
Loss rate of the data traffic 61 61