Advancing CernVM-FS and its Development Infrastructure José Molina Colmenero CERN EP-SFT

Contents New features in CernVM-FS – Automatic File Catalog Splitting – Cache preloader – CernVM-FS python library improvements Tests and Benchmarking – HTTP/2 protocol support – Coverity integration with Jenkins – Integration tests on OS X – Travis-CI integration – Integration tests for Parrot Connector Infrastructure – Debian repository – Squid server docker image – Web-based repository browser Extras – CernVM-FS Java client library – Android app 2

New features 3

New features: Catalog splitting Catalogs in CernVM-FS – File system meta data is stored in a catalog hierarchy (super-imposed on directory hierarchy) – Up to now: manually defined splitting points New way of limiting the catalog size – Optimal static algorithm to balance the catalog tree Surprisingly good results! – The obtained catalog distribution is similar to the manually created one in ALICE Server administrators will no longer have to manually create them 4

New features: Cache preloader Meant to be used by supercomputers to populate the cache in a shared file system Useful if worker nodes have no direct connection to the Internet All libraries are statically linked – No dependencies required – Makes it easy to use and deploy 5

New feature: python library It is a library used for testing purposes, mainly as a helper tool – It is not a production component Improvements in the python librarypython library – Creation of local caches Necessary to efficiently instantiate a repository Compatible with the standard CernVM-FS client cache – Creation and instantiation of historic revisions Possibility to go back in time and fetch the files as they were in a concrete time – Pluggable data object retrieval From local disk or HTTP server Optimized download performance for HTTP 6

Tests and benchmarks 7

Test and benchmarks: HTTP/2 Quick reminder: HTTP/1.1 vs HTTP/2.0 8 HTTP/1.1 One connection per request Text protocol No multiplexing No compression in headers No priority policy HTTP/2.0 One connection Binary protocol Multiplexed streams Compression in headers Priority policy (optional)

Test and benchmarks: HTTP/2 Still under development Uses only one multiplexed connection – Much lower latency penalty Compression of the headers libcurl’s API is highly compatible with HTTP/1.1 Current technologies seem not to be fully ready yet – libcurl for the client – mod_http2 for the Apache server Special testing infrastructure – We acquired access to servers in Taipei (ASGC) and the University of Nebraska 9

Tests and benchmarks: Coverity -Once a week Jenkins runs the project and submits the results to Coverity -Most of the reported issues have already been fixed 10

Tests and benchmarks: Coverity Problems when using Jenkins’ Coverity plug-in – It does not properly detect the compiler export CC=gcc export CXX=g++ – It wraps the whole building process We also build our dependencies, which shouldn’t be uploaded to Coverity – There is a trick to remove them cov-manage-emit --dir "$COV_LOCATION" \ --tu-pattern \ 'file("^.+externals.+$")' delete – The Jenkins chart does not seem to work 11

Tests and benchmarks: OS X 12 Test Steering Machine Start Test Run SLC5 x86SLC5 x64SLC6 x86 SLC6 x64 Ubuntu x86 Ubuntu x64 Spawn VMsReport Status OS X x64 Openstack Vagrant on OS X

Tests and benchmarks: OS X Client integration tests adapted to provide OS X support Necessity of an special OS X VM – Integration tests are forced to run in VMs because they are potentially destructive Entirely new setup of both host and VM – Usage of Vagrant together with VirtualBox to spawn the VM Significant differences between Linux and Mac shells – Many commands exist but differ in the parameters Fully integrated in Jenkins 13

Tests and Benchmarks: Travis -Automatic feedback for both maintainer and contributors -Automatic analysis after pull request submission -Checking syntax, building and testing -Automatic publishing on GitHub 14 Travis GitHub

Tests and Benchmarks: Travis-CI What is it? – Continuous integration as a service – Free for open source projects What is the difference between Travis-CI and Jenkins? – Travis-CI tracks every `git push` to the repository – Jenkins is the full integration build and test after merging How does it work? – Link the project with Travis-CI – Create a.travis.yml file and place it in the root of the repository This file describes the building and testing steps – Travis-CI now checks every pushed commit It automatically modifies the commit and pull request status 15

Tests and benchmarks: Parrot What is Parrot? – Parrot is a tool for attaching existing programs to remote I/O system through the file system interface – It allows to run programs without special privileges or kernel changes – It can be applied to almost any program without re- writing, re-linking or re-installing – Useful to mount file systems where FUSE is not present Adaptation of the integration tests to run on Parrot – Work in progress 16

Infrastructure 17

Infrastructure: Debian repository We had debian packages, but no repository – wget sudo dpkg –i Creation of a debian repository Packages include the corresponding signature All dependencies are installed 18 sudo apt-get install cvmfs

Infrastructure: repository browserrepository browser 19

Infrastructure: squid server Development of a dockerized squid serverdockerized squid server – HTTP proxy server for faster file delivery Includes the necessary configuration to be quickly and easily developed – Including CernVM-FS-related configuration 20

Extras 21

Extras: Java client library Base client library written in Javaclient library – Highly inspired by the python library Support for the most important operations – File system walkthrough – Catalog and file listing – Revision instantiation – Repository key matching Proof of concept availableavailable – Actual CernVM-FS Java client using FUSE – Available for all our platforms 22

Extras: Android app 23

Thanks for your attention! 24

25

Backup Slides 26

Travis-CI required permissions More information herehere 27