Checkpoint-restore in userspace. Are we there yet? Pavel Emelyanov LinuxCon Europe 2012

2 What is C/R and what is it for? C/R is an ability to snapshot an application state and restore it from the state at any time and place later. Usage scenarios: – Live migration – Reboot-less kernel update – Applications start-up boost – Working environment snapshots – HPC load balancing –...

3 Is it possible to do all these nice things now? Yes! Almost. And we're close to it! This talk answers on: ✔ How shall we be able to do it? ✔ How close to it are we? ✔ How far from “impossible to” are we? ✔ What has happened since then?

4 A brief history of C/R in Linux Jan OpenVZ project starts with live-migration support all in kernel feature First collaborative attempt to get C/R upstream First more-or-less complete version (over 100 patches) First attempt to do C/R mostly in user-space Linus decided to merge first set of patches upstream CRIU v0.1 CRIU v0.2 + LXC support JulSep 2012

5 CRIU project ultimate goal AP P FSFS M CredsCreds Ti m er s AP P share dump Image restore AP P IPC... Network IPC... Network

6 CRIU project concept kernel AP P FDFD open CR IU too l What files are opened? ~A PP open FDFD dum p restore

7 Existing kernel APIs kernel Proc System calls About selfAbout anybody dum p restore Netlink

8 How CRIU grows up kernel AP P FOOFOO Get FOO CR IU too l Info on FOO-s ~A PP Get FOO back FDFD dum p restore ? X Get FOO ++ Info FOO ++

9 CRIU project grow-up concept (Linus vision)... this is a project by various mad Russians to perform c/r mainly from userspace, with various oddball helper code added into the kernel where the need is demonstrated. So rather than some large central lump of code, what we have is little bits and pieces popping up in various places which either expose something new or which permit something which is normally kernel- private to be modified...

1010 Kernel impact ~110 patches merged ~15 patches in flight 9 new features appeared (1 C/R-only) 2 new features to come

1 The most interesting new features in kernel Parasite code injection – Read task states, that are currently retrieved by a task only about himself The kcmp system call – Helps checking which kernel objects are shared between processes Sockets information dumping via netlink ( sock_diag ) – Extendable sockets state retrieving engine TCP repair mode – Read intimate state of a TCP connection and reconstructs it from scratch on a freshly created socket

1212 Other new features in kernel Virtual net devices indices – Allows to restore network devices in a namespace Proc map_files directory – Find out what exact file is mapped – Mappings sharing info Socket peeking offset – Allows peeking sockets queues (reading without removing data from queue) More socket get-able sockoptions – Bound device – Packet filter

1313 CRIU features so far IPC... Network X86_64 architecture Process tree linkage Multi-threaded apps UNIX sockets LXC container environment Terminals, groups and sessions Non-posix files (inotify, epoll, etc.) Open files (+ shared and unlinked) Kernel V3.6 Memory mappings of all kinds Established TCP connection

1414 How we test it ZDTM – set of atomic tests for every new piece of functionality Real software Apache MySQL Make and gcc Tar and gzip Sshd with connections Screen with top inside VNC with xscreensaver and client connection NGINX MongoDB tcpdump

1515 Main plans for the nearest future ● Full OS resources coverage ● Merge in-flight patches, so that everything works on vanilla kernel ● Properly integrate crtools with LXC and OpenVZ ● Live-migration script ● Pre-migrate app memory before freeze (speeds things up)

1616 CRIU project resources – project news and documentation – git repo with tool sources – kernel with all in-flight patches applied mailing list +CRIU page

Parallels – Optimized Computing TM Confidential 17 Pavel Emelyanov