1. CVMFS Alessandro De Salvo Alessandro.DeSalvo@roma1.infn.it 18-5-2011 Alessandro.DeSalvo@roma1.infn.it Outline  CVMFS architecture  CVMFS usage in the LHC experiments  Performance tests A. De Salvo – 18 Dec 2011

2. A. De Salvo – CVMFS – CCR Workshop, Isola d’Elba 18-5-2011 CVMFS basics ATLAS and LHCb moving to a dynamic software distribution model via CVMFS (CernVMFileSystem) Virtual software installation by means of an HTTP File System Distribution of read-only binaries Files and file meta data are downloaded on demand and locally cached Self-contained (e. g. /cvmfs/atlas.cern.ch), does not interfere with the base system

3. A. De Salvo – CVMFS – CCR Workshop, Isola d’Elba 18-5-2011 Content addressable storage Data Store Compressed Chunks (Files) Eliminates Duplicates File Catalog Directory Structure Symlinks SHA1 of Regular Files Digitally Signed Time to Live Nested Catalogs Immutable Files, trivial to check for corruption

4. A. De Salvo – CVMFS – CCR Workshop, Isola d’Elba 18-5-2011 CVMFS Clients Mounted in the remote nodes via the fuse module Local cache for faster access Benefits of a squid hierarchy to guarantee performance, scalability and reliability Same squid type as the one used for Frontier

5. A. De Salvo – CVMFS – CCR Workshop, Isola d’Elba 18-5-2011 CVMFS Backends Mirror servers Web servers listening on port 80, 8000 Proxy servers Local load-balanced Squid forward proxy (SL5 Squid)

6. A. De Salvo – CVMFS – CCR Workshop, Isola d’Elba 18-5-2011 Fuse Module Namespace: /cvmfs/ e. g. /cvmfs/atlas.cern.ch One process per repository + watchdog One mount per repository Exclusive cache directory per repository Mount scripts Setup environment (number of file descriptors, access rights,... ) Used by autofs on /cvmfs Used by system mount mount -t cvmfs atlas.cern.ch /cvmfs/atlas.cern.ch Diagnostics service cvmfs probe cvmfs_config chksetup cvmfs_config showconfig cvmfs_fsck cvmfs-talk, connect to running instance CVMFS client tools

7. A. De Salvo – CVMFS – CCR Workshop, Isola d’Elba 18-5-2011 Proxies SL5 Squid, Load-Balancing + Fail-Over e. g. CVMFS_HTTP_PROXY="A|B|C" as of next revision: semantics like Frontier/Squid Mirrors Fail-Over, set order manually in /etc/cvmfs/domain.d/cern.ch.local Mirrors at CERN, RAL, BNL operational by the end of the month CVMFS Web Caches

8. A. De Salvo – CVMFS – CCR Workshop, Isola d’Elba 18-5-2011 Proxies SL5 Squid, Load-Balancing + Fail-Over e. g. CVMFS_HTTP_PROXY="A|B|C" as of next revision: semantics like Frontier/Squid Mirrors Fail-Over, set order manually in /etc/cvmfs/domain.d/cern.ch.local Mirrors at CERN, RAL, BNL operational by the end of the month CVMFS Web Caches

9. A. De Salvo – CVMFS – CCR Workshop, Isola d’Elba 18-5-2011 Release manager machine acts as "template". Maintains shadow tree and CernVM-FS repository Publishing ① Install and test new software ② Synchronize changes with repository ③ Sign file catalogs ④ Create new file system snapshot CVMFS Backend storage Steps 2–4: Taken care of by CernVM-FS server tools

10. A. De Salvo – CVMFS – CCR Workshop, Isola d’Elba 18-5-2011 Known users: CernVM, ATLAS U.S. Tier3s, RAL, PIC, NIKHEF, CERN, Wuppertal, QMUL, Munich, Lancester, Dortmund, JINR,... /cvmfs/atlas.cern.ch “Production” Software 5 release managers, 24 releases SLC4 + 31 releases SLC5 590 GB, 11 Million files, 16 Million entries (shadow) 85 GB and 1.5 Million files (repository) New production server with the final setup almost ready Populated, but not yet tested Will expose all the software structure as in current grid sites /cvmfs/atlas-condb.cern.ch — ATLAS Condition Flat Files Release manager machine hosted by CERN IT Automatic update several times a day 30 GB, 110 000 files, 7 000 directories, 3 000 symlinks (shadow tree) 30 GB, 70 000 files (repository) Only fraction of all conditions data /cvmfs/atlas-nightlies.cern.ch — ATLAS Nightlies Work in progress Integrated with the current Installation System CVMFS sites are used by the installation system transparently, aside of sites using a different FS ATLAS on CVMFS

11. A. De Salvo – CVMFS – CCR Workshop, Isola d’Elba 18-5-2011 CVMFS test at CNAF (LHCb)

12. A. De Salvo – CVMFS – CCR Workshop, Isola d’Elba 18-5-2011 Tests performed Data Intensive Reading ~40 GB of software libraries (all the LHCb repository) from ~200 Worker Nodes concurrently find /opt/lhcb/lib/lhcb –name "*.so" -type f -exec cat {} >/dev/null \; Metadata intensive Initialization (SetupProject) and startup of the LHCb analysis program from ~200 Worker Nodes concurrently Very similar to the procedure used by ATLAS Currently implying a huge number of metadata lookup CVMFS tests at CNAF (LHCb): test definition

13. A. De Salvo – CVMFS – CCR Workshop, Isola d’Elba 18-5-2011 Data intensive setup With squid caches unloaded As soon as the files are requested they are downloaded synchronously from CERN then served to the Worker Nodes Typical access mode of an access to a new software release With squid caches loaded The files are already present in the disks of the squid servers Measuring the performance of the software distribution from the squid caches to the CVMFS clients CVMFS tests at CNAF (LHCb): test definition [2]

14. A. De Salvo – CVMFS – CCR Workshop, Isola d’Elba 18-5-2011 Peak at 240 MB/s (half bandwidth) CVMFS tests at CNAF (LHCb): data-intensive test results Peak at 440 MB/s (full bandwidth)

15. A. De Salvo – CVMFS – CCR Workshop, Isola d’Elba 18-5-2011 Metadata intensive setup With squid caches and local disk caches unloaded With squid caches loaded and local disk caches unloaded With squid caches and local disk caches loaded Same test executed for reference on cNFS over GPFS Without memory caching (page cache) in the WNs With memory caching CVMFS tests at CNAF (LHCb): test definition [3]

16. A. De Salvo – CVMFS – CCR Workshop, Isola d’Elba 18-5-2011 CVMFS: good results in general, 10 -> 100 seconds on a heavy-duty case CVMFS tests at CNAF (LHCb): metadata-intensive test results cNFS has similar performance to CVMFS But different architecture and features

17. A. De Salvo – CVMFS – CCR Workshop, Isola d’Elba 18-5-2011 New CVMFS stable version (2.0) ready by the end of April ATLAS and LHCb are moving to CVMFS as main distribution method Better management More efficiency Good performance Already integrated with the current installation system (ATLAS) Useful links Download http://cernvm.cern.ch/portal/downloads Yum http://cvmrepo.web.cern.ch/cvmrepo/yum Mailing list cvmfs-talk@cern.ch News http://twitter.com/cvmfs Bug Tracker https://savannah.cern.ch/bugs/?group=cernvmhttps://savannah.cern.ch/bugs/?group=cernvm Credits Thanks to V. Vagnoni and the CNAF folks for the LHCb tests Conclusions