1. Checkpoint-restore in userspace. Are we there yet? Pavel Emelyanov LinuxCon Europe 2012

2. 2 What is C/R and what is it for? C/R is an ability to snapshot an application state and restore it from the state at any time and place later. Usage scenarios: – Live migration – Reboot-less kernel update – Applications start-up boost – Working environment snapshots – HPC load balancing –...

3. 3 Is it possible to do all these nice things now? Yes! Almost. And we're close to it! This talk answers on: ✔ How shall we be able to do it? ✔ How close to it are we? ✔ How far from “impossible to” are we? ✔ What has happened since then?

4. 4 A brief history of C/R in Linux 2005200820102011 Jan OpenVZ project starts with live-migration support all in kernel feature First collaborative attempt to get C/R upstream First more-or-less complete version (over 100 patches) First attempt to do C/R mostly in user-space Linus decided to merge first set of patches upstream CRIU v0.1 CRIU v0.2 + LXC support JulSep 2012

5. 5 CRIU project ultimate goal AP P FSFS M CredsCreds Ti m er s...... AP P share dump Image 0011011001 0010101110 1101011001 1011100111 0001011011 0101011100 1011010110... restore AP P IPC... Network IPC... Network

6. 6 CRIU project concept kernel AP P FDFD open CR IU too l What files are opened? ~A PP open FDFD dum p restore

7. 7 Existing kernel APIs kernel Proc System calls About selfAbout anybody dum p restore Netlink

8. 8 How CRIU grows up kernel AP P FOOFOO Get FOO CR IU too l Info on FOO-s ~A PP Get FOO back FDFD dum p restore ? X Get FOO ++ Info FOO ++

9. 9 CRIU project grow-up concept (Linus vision)... this is a project by various mad Russians to perform c/r mainly from userspace, with various oddball helper code added into the kernel where the need is demonstrated. So rather than some large central lump of code, what we have is little bits and pieces popping up in various places which either expose something new or which permit something which is normally kernel- private to be modified...

10. 1010 Kernel impact ~110 patches merged ~15 patches in flight 9 new features appeared (1 C/R-only) 2 new features to come

11. 1 The most interesting new features in kernel Parasite code injection – Read task states, that are currently retrieved by a task only about himself The kcmp system call – Helps checking which kernel objects are shared between processes Sockets information dumping via netlink ( sock_diag ) – Extendable sockets state retrieving engine TCP repair mode – Read intimate state of a TCP connection and reconstructs it from scratch on a freshly created socket

12. 1212 Other new features in kernel Virtual net devices indices – Allows to restore network devices in a namespace Proc map_files directory – Find out what exact file is mapped – Mappings sharing info Socket peeking offset – Allows peeking sockets queues (reading without removing data from queue) More socket get-able sockoptions – Bound device – Packet filter

13. 1313 CRIU features so far IPC... Network X86_64 architecture Process tree linkage Multi-threaded apps UNIX sockets LXC container environment Terminals, groups and sessions Non-posix files (inotify, epoll, etc.) Open files (+ shared and unlinked) Kernel V3.6 Memory mappings of all kinds Established TCP connection

14. 1414 How we test it ZDTM – set of atomic tests for every new piece of functionality Real software Apache MySQL Make and gcc Tar and gzip Sshd with connections Screen with top inside VNC with xscreensaver and client connection NGINX MongoDB tcpdump

15. 1515 Main plans for the nearest future ● Full OS resources coverage ● Merge in-flight patches, so that everything works on vanilla kernel ● Properly integrate crtools with LXC and OpenVZ ● Live-migration script ● Pre-migrate app memory before freeze (speeds things up)

16. 1616 CRIU project resources http://criu.orghttp://criu.org – project news and documentation http://git.criu.orghttp://git.criu.org – git repo with tool sources https://github.com/cyrillos/linux-2.6/https://github.com/cyrillos/linux-2.6/ – kernel with all in-flight patches applied criu@openvz.orgcriu@openvz.org mailing list +CRIU page

17. Parallels – Optimized Computing TM Confidential 17 Pavel Emelyanov xemul@parallels.com