1. April, 2010Oren Laadan, Columbia University 1 Network Migration in Linux Oren Laadan orenl@cs.columbia.edu

2. April, 2010Oren Laadan, Columbia University 2 Network Migration (1) Goal: transparently migrate a live system between hosts Use cases: load balancing fault tolerance green computing Maintenance...

3. April, 2010Oren Laadan, Columbia University 3 Network Migration (2) ”Transparently migrate a live system...” live system: a system with open network connections with other peers migrate: transfer execution state from one host to another host, and continue there transparently: both the system and the network peers should remain unaware

4. April, 2010Oren Laadan, Columbia University 4 Example: Virtual Machine What does it take to migrate a virtual machine while keeping its network connections ? Network IP 1

5. April, 2010Oren Laadan, Columbia University 5 Example: Virtual Machine What does it take to migrate a virtual machine while keeping its network connections ? Network IP 1 IP 2

6. April, 2010Oren Laadan, Columbia University 6 Example: Virtual Machine What does it take to migrate a virtual machine while keeping its network connections ? Network IP 1 IP 2

7. April, 2010Oren Laadan, Columbia University 7 Virtual Machine Migration Preserve IP through migration only within same IP subnet Remap IP ↔ MAC address use gratuitous ARP Lost packets ? UDP - don't care TCP - retransmission after timeout

8. April, 2010Oren Laadan, Columbia University 8 Application Migration What if we don't want to move an entire VM ? What if we want to move specific applications ?

9. April, 2010Oren Laadan, Columbia University 9 Operating System Virtualization Virtual execution environment Encapsulate entire applications Private, virtual namespace: private: isolate and confine depenencies virtual: self containers, decoupled from OS

10. April, 2010Oren Laadan, Columbia University 10 Virtual Execution Environment Hardware Operating System Virtualization Layer Virtual Execution Environment (VEE)

11. April, 2010Oren Laadan, Columbia University 11 getpid() ? Hardware Operating System Virtualization Layer VEE

12. April, 2010Oren Laadan, Columbia University 12 getpid() ? Hardware Operating System Virtualization Layer VEE

13. April, 2010Oren Laadan, Columbia University 13 getpid() ? Hardware Operating System Virtualization Layer VEE

14. April, 2010Oren Laadan, Columbia University 14 getpid() ? Kernel: 4150 (real) Hardware Operating System Virtualization Layer VEE 4150

15. April, 2010Oren Laadan, Columbia University 15 getpid() ? Kernel: 4150 (real) VEE: 305 (virtual) Hardware Operating System Virtualization Layer VEE 4150 305

16. April, 2010Oren Laadan, Columbia University 16 Private Namspace Hardware Operating System Virtualization Layer VEE #1VEE #2VEE #n

17. April, 2010Oren Laadan, Columbia University 17 VM vs Application Migration Virtual Machine entire operating system state includes network stack and drivers Application Container only designated application(s) specific network connections only IP layer and above exclude driver state

18. April, 2010Oren Laadan, Columbia University 18 Application Migration Preserve IP through migration only within same IP subnet Remap IP ↔ MAC address use gratuitous ARP Lost packest ? UDP - don't care TCP - retransmission after timeout But also … ?

19. April, 2010Oren Laadan, Columbia University 19 Application Migration Reconstruct network connections rebuild sockets reconnect to peers transparently

20. April, 2010Oren Laadan, Columbia University 20 Network Migration Source (checkpoint)Target (restart) 1. block networkblock network 2. save state setup network 3. transfer state receive state 4. rebuild state 5. kill application 6. kill network gratuitous ARP 7. unblock networkunblock network

21. April, 2010Oren Laadan, Columbia University 21 Newtork Migration (1) Step 1: Block the network only block for migrating container other containers remain unaffected how do we do that ?

22. April, 2010Oren Laadan, Columbia University 22 Newtork Migration (1) Netfilter and IPTables Block the network drop all packets to/from container IP Unblock the network remove the above rule...

23. April, 2010Oren Laadan, Columbia University 23 TCP Retransmissions What happens if migration takes long enough such that packets are lost ? How to resume full network speed when migration completes ?

24. April, 2010Oren Laadan, Columbia University 24 TCP Retransmissions What happens if migration takes long enough such that packets are lost ? How to resume full network speed when migration completes ? Cheat … Publish 0-window prior to checkpiont Publish (restore) original after restart

25. April, 2010Oren Laadan, Columbia University 25 Newtork Migration (2) Step 2: Save Network State what do we need to save ?

26. April, 2010Oren Laadan, Columbia University 26 Newtork Migration (2) Step 2: Save Network State newtork properties: IPs, netmask, broadcast,... sockets used by applications: type, protocol, … lingering (unreferenced) sockets with buffers half-baked connections (received, not accepted) netfilter and iptables configuration

27. April, 2010Oren Laadan, Columbia University 27 Network Migration (4) Step 4: Restore Network State how do we do this ? hidden caveats ?

28. April, 2010Oren Laadan, Columbia University 28 Network Migration (4) Step 4: Restore Network State Brute force... conceptually simple deep understanding of kernel stack duplicate existing kernel code

29. April, 2010Oren Laadan, Columbia University 29 Restore Socket: Fuerza Bruta Quick code walk-through...

30. April, 2010Oren Laadan, Columbia University 30 Questions ? Thank You ! orenl@cs.columbia.edu