Implementing CVMFS server using an union file system Monday, 12th of September 2011 René Meusel Implementing CVMFS server using an union file system CERN Summer Student Report René Meusel Hasso Plattner Institute Potsdam (Germany) Supervisors: Predrag Buncic, Jakob Blomer Good Morning rene.meusel@student.hpi.uni-potsdam.de

1 2 3 4 Agenda porting CVMFS to Mac OS X Monday, 12th of September 2011 René Meusel Agenda 1 porting CVMFS to Mac OS X changing it‘s build system to CMake replacing the shadow directory with a union file system approach using the low level interface of FUSE for CVMFS 2 3 port to Mac OS, build system, MAIN TASK: union file system, low level API 4

Porting CVMFS to Mac OS X Monday, 12th of September 2011 René Meusel Porting CVMFS to Mac OS X Currenty CVMFS runs exclusively on Linux Porting to Mac OS X was fairly simple a few system calls had to be abstracted e.g. stat64, readdir, spinlock-implementation and Thread Local Storage Mostly implemented in a compatibility-header Just a proof of concept at the moment No repositories with Mac OS X binaries Rewrote build system from automake to CMake Easier and more convenient user interface packaging for different platforms Fully platform independent packaging requested by some users

Kernel module with no real support anymore Monday, 12th of September 2011 René Meusel Synchronisation Release Manager Shadow Directory Kernel module with no real support anymore Redir-FS CernVM Filesystem Repository read only volume Union Filesystem Overlay redir-FS = unsupported on old kernels User Server Client

Union Filesystems Mount more than one file system at once Monday, 12th of September 2011 René Meusel Union Filesystems Mount more than one file system at once Appear layered on top of each other (branches) Usual use case: Linux Live-CDs Allows changes to read only file systems Main features (see next slide): copy on write white-out files opaque directories UnionFS, AUFS, OverlayFS, UnionFS FUSE, ... several implementations no union file system in mainline kernel yet But: SLC 5 ships with AUFS 1 included read only writable overlay usual use case, main features, support guaranteed

Updating CVMFS repo using AUFS Monday, 12th of September 2011 René Meusel Updating CVMFS repo using AUFS repository writable overlay AUFS CernVM Filesystem appears as ordinary writable volume Advanced multi layered Unification FileSystem AUFS makes CVMFS mutable by the release manager Changes show up in the overlay directory We can sync these files into the repository afterwards

Usual synchronisation internals Monday, 12th of September 2011 René Meusel Usual synchronisation internals repository (read only) Directory foo bar file symlink CernVM Filesystem repository writable overlay AUFS appears as ordinary writable volume $ rm -rf Directory symlink $ mkdir Directory $ touch Directory/newFile $ ln file hardlink writable volume Directory foo bar file symlink overlay directory Directory newFile .wh..wh..opq .wh.symlink hardlink extract changes taking all three volumes into account new data in overlay directory simplifies compression writable volume Directory newFile file hardlink

Union Filesystems Quick Performance test: Simulating a usual use case Monday, 12th of September 2011 René Meusel Union Filesystems Quick Performance test: Simulating a usual use case copy on write semantics by union filesystem means Decompressing Linux 3.0 kernel sources into union volume 77 MB | 400 MB decompressed | 39000 files Delete the resulting directory afterwards Decompression Recursive delete Native (no union) 29.8s 0.9s AUFS 31.0s 1.5s UnionFS FUSE * 54.2s 10.3s AUFS directly decided for write branch  no significant overhead Deleting: AUFS has to check for file existance in read only directory  lots of stat calls Underlying branch must be checked for file existence mean of three trials No real overhead: underlying branch is marked read only Performance of the FUSE implementation is not suitable * FUSE implementation was investigated because it wouldn‘t need a specific kernel but obviously is too slow

Problems foo bar file hardlink Monday, 12th of September 2011 René Meusel Problems Shadow directory had normel file system semantics we just „copied“ them into the CVMFS repository storage inode 3 inode 1 inode 2 foo bar file hardlink

Problems Shadow directory had normel file system semantics Monday, 12th of September 2011 René Meusel Problems Shadow directory had normel file system semantics we just „copied“ them into the CVMFS repository AUFS approach consists of three different file systems inodes are not consistent over file system borders mapping of different inodes is not accessible (only present in kernel) storage 42 12 25 .wh.foo bar moep writable overlay cache 3 1 2 foo bar file hardlink read only repository foo bar 5 hardlink file 6 moep 7 union volume

Problems Impossible to distingush „edit“ and „replace“ Monday, 12th of September 2011 René Meusel Problems Shadow directory had normel file system semantics we just „copied“ them into the CVMFS repository AUFS approach consists of three different file systems inodes are not consistent over file system borders mapping of different inodes is not accessible (only present in kernel) Impossible to distingush „edit“ and „replace“ files are copied to the overlay in any case as file systems differ, inode changes cannot be recognized hard links become challenging AUFS supports pseudo hard links over different branches however hard links are not properly supported by FUSE‘s high level API, resulting in a kernel panic in connection with AUFS solution: port CVMFS to low level API of FUSE

FUSE high vs. low level API Monday, 12th of September 2011 René Meusel FUSE high vs. low level API High level API is designed for easy usage simple callback architecture based on file names and paths rather than inodes provided on top of low level API by libfuse in user space Low level API provides more control usual VFS layer functions passed to user space as callbacks based on inodes  therefore full control over hard links maybe better performance (to be investigated) Possibility to export CVMFS via NFS using low level API requested by Gird sites with limited local disk cache on worker notes

Chance for better performance? Monday, 12th of September 2011 René Meusel Chance for better performance? libfuse CVMFS High Level API CVMFS optimised caching general purpose caching inode translation Low Level API Kernel Space User Space Character Device FUSE Kernel Module Virtual File System Layer

Conclusion Ported CVMFS to Mac OS X Monday, 12th of September 2011 René Meusel Conclusion Ported CVMFS to Mac OS X Converted the build system to CMake Implemented repository updates by union file system means CVMFS on top of the low level FUSE API as a prototype

Future work Integrate CVMFS with FUSE low level API Monday, 12th of September 2011 René Meusel Future work Integrate CVMFS with FUSE low level API optimise internal caching infrastructure implement a proper inode allocation method think about backward compatibility Overall system hardening beta test build system integration tests for newly implemented synchronisation bring low level FUSE API interface to a stable implementation Mac OS X could become interesting over time set up a test repository with Mac binaries