CVMFS Alessandro De Salvo Outline  CVMFS architecture  CVMFS usage in the LHC experiments  Performance tests A. De Salvo – 18 Dec 2011

A. De Salvo – CVMFS – CCR Workshop, Isola d’Elba CVMFS basics ATLAS and LHCb moving to a dynamic software distribution model via CVMFS (CernVMFileSystem) Virtual software installation by means of an HTTP File System Distribution of read-only binaries Files and file meta data are downloaded on demand and locally cached Self-contained (e. g. /cvmfs/atlas.cern.ch), does not interfere with the base system

A. De Salvo – CVMFS – CCR Workshop, Isola d’Elba Content addressable storage Data Store Compressed Chunks (Files) Eliminates Duplicates File Catalog Directory Structure Symlinks SHA1 of Regular Files Digitally Signed Time to Live Nested Catalogs Immutable Files, trivial to check for corruption

A. De Salvo – CVMFS – CCR Workshop, Isola d’Elba CVMFS Clients Mounted in the remote nodes via the fuse module Local cache for faster access Benefits of a squid hierarchy to guarantee performance, scalability and reliability Same squid type as the one used for Frontier

A. De Salvo – CVMFS – CCR Workshop, Isola d’Elba CVMFS Backends Mirror servers Web servers listening on port 80, 8000 Proxy servers Local load-balanced Squid forward proxy (SL5 Squid)

A. De Salvo – CVMFS – CCR Workshop, Isola d’Elba Fuse Module Namespace: /cvmfs/ e. g. /cvmfs/atlas.cern.ch One process per repository + watchdog One mount per repository Exclusive cache directory per repository Mount scripts Setup environment (number of file descriptors, access rights,... ) Used by autofs on /cvmfs Used by system mount mount -t cvmfs atlas.cern.ch /cvmfs/atlas.cern.ch Diagnostics service cvmfs probe cvmfs_config chksetup cvmfs_config showconfig cvmfs_fsck cvmfs-talk, connect to running instance CVMFS client tools

A. De Salvo – CVMFS – CCR Workshop, Isola d’Elba Proxies SL5 Squid, Load-Balancing + Fail-Over e. g. CVMFS_HTTP_PROXY="A|B|C" as of next revision: semantics like Frontier/Squid Mirrors Fail-Over, set order manually in /etc/cvmfs/domain.d/cern.ch.local Mirrors at CERN, RAL, BNL operational by the end of the month CVMFS Web Caches

A. De Salvo – CVMFS – CCR Workshop, Isola d’Elba Proxies SL5 Squid, Load-Balancing + Fail-Over e. g. CVMFS_HTTP_PROXY="A|B|C" as of next revision: semantics like Frontier/Squid Mirrors Fail-Over, set order manually in /etc/cvmfs/domain.d/cern.ch.local Mirrors at CERN, RAL, BNL operational by the end of the month CVMFS Web Caches

A. De Salvo – CVMFS – CCR Workshop, Isola d’Elba Release manager machine acts as "template". Maintains shadow tree and CernVM-FS repository Publishing ① Install and test new software ② Synchronize changes with repository ③ Sign file catalogs ④ Create new file system snapshot CVMFS Backend storage Steps 2–4: Taken care of by CernVM-FS server tools

A. De Salvo – CVMFS – CCR Workshop, Isola d’Elba Known users: CernVM, ATLAS U.S. Tier3s, RAL, PIC, NIKHEF, CERN, Wuppertal, QMUL, Munich, Lancester, Dortmund, JINR,... /cvmfs/atlas.cern.ch “Production” Software 5 release managers, 24 releases SLC releases SLC5 590 GB, 11 Million files, 16 Million entries (shadow) 85 GB and 1.5 Million files (repository) New production server with the final setup almost ready Populated, but not yet tested Will expose all the software structure as in current grid sites /cvmfs/atlas-condb.cern.ch — ATLAS Condition Flat Files Release manager machine hosted by CERN IT Automatic update several times a day 30 GB, files, directories, symlinks (shadow tree) 30 GB, files (repository) Only fraction of all conditions data /cvmfs/atlas-nightlies.cern.ch — ATLAS Nightlies Work in progress Integrated with the current Installation System CVMFS sites are used by the installation system transparently, aside of sites using a different FS ATLAS on CVMFS

A. De Salvo – CVMFS – CCR Workshop, Isola d’Elba CVMFS test at CNAF (LHCb)

A. De Salvo – CVMFS – CCR Workshop, Isola d’Elba Tests performed Data Intensive Reading ~40 GB of software libraries (all the LHCb repository) from ~200 Worker Nodes concurrently find /opt/lhcb/lib/lhcb –name "*.so" -type f -exec cat {} >/dev/null \; Metadata intensive Initialization (SetupProject) and startup of the LHCb analysis program from ~200 Worker Nodes concurrently Very similar to the procedure used by ATLAS Currently implying a huge number of metadata lookup CVMFS tests at CNAF (LHCb): test definition

A. De Salvo – CVMFS – CCR Workshop, Isola d’Elba Data intensive setup With squid caches unloaded As soon as the files are requested they are downloaded synchronously from CERN then served to the Worker Nodes Typical access mode of an access to a new software release With squid caches loaded The files are already present in the disks of the squid servers Measuring the performance of the software distribution from the squid caches to the CVMFS clients CVMFS tests at CNAF (LHCb): test definition [2]

A. De Salvo – CVMFS – CCR Workshop, Isola d’Elba Peak at 240 MB/s (half bandwidth) CVMFS tests at CNAF (LHCb): data-intensive test results Peak at 440 MB/s (full bandwidth)

A. De Salvo – CVMFS – CCR Workshop, Isola d’Elba Metadata intensive setup With squid caches and local disk caches unloaded With squid caches loaded and local disk caches unloaded With squid caches and local disk caches loaded Same test executed for reference on cNFS over GPFS Without memory caching (page cache) in the WNs With memory caching CVMFS tests at CNAF (LHCb): test definition [3]

A. De Salvo – CVMFS – CCR Workshop, Isola d’Elba CVMFS: good results in general, 10 -> 100 seconds on a heavy-duty case CVMFS tests at CNAF (LHCb): metadata-intensive test results cNFS has similar performance to CVMFS But different architecture and features

A. De Salvo – CVMFS – CCR Workshop, Isola d’Elba New CVMFS stable version (2.0) ready by the end of April ATLAS and LHCb are moving to CVMFS as main distribution method Better management More efficiency Good performance Already integrated with the current installation system (ATLAS) Useful links Download Yum Mailing list News Bug Tracker Credits Thanks to V. Vagnoni and the CNAF folks for the LHCb tests Conclusions