1. Automated software packaging and installation for the ATLAS experiment Simon George Royal Holloway, University of London Christian Arnault, LAL Orsay; Michael Gardner, RHUL; Roger Jones, University of Lancaster; Saul Youssef, Boston University S.George@rhul.ac.uk e-Science All Hands Meeting Nottingham 2-4 September 2003 ATLASexperiment.org

2. Wed 03Sep03Simon George RHUL2 Introduction This talk is about packaging, distribution and installation for a large software project This talk is about packaging, distribution and installation for a large software project It is essential because It is essential because The project computing resources are widely distributed around 140 institutes, who all want to use the software The project computing resources are widely distributed around 140 institutes, who all want to use the software We want to be able to use Grid resources that do not have locally managed installations of the software We want to be able to use Grid resources that do not have locally managed installations of the software Our working model also requires the ability to deploy user code that is not part of an official distribution Our working model also requires the ability to deploy user code that is not part of an official distribution Ill describe the process developed and the tools used. Ill describe the process developed and the tools used.

3. Wed 03Sep03Simon George RHUL3 Contents ATLAS and its software ATLAS and its software Requirements Requirements Tools and formats Tools and formats Meta data Meta data Naming conventions Naming conventions Creating and installing the kits Creating and installing the kits Conclusions and outlook Conclusions and outlook

4. Wed 03Sep03Simon George RHUL4 The ATLAS Experiment A Particle Physics experiment at the Large Hadron Collider, CERN A Particle Physics experiment at the Large Hadron Collider, CERN 1600 physicists, 140 institutes, 6 continents 1600 physicists, 140 institutes, 6 continents Studies include Studies include search for the origin of masssearch for the origin of mass excess of matter over antimatter in the universeexcess of matter over antimatter in the universe evidence for Supersymmetryevidence for Supersymmetry other new physicsother new physics

5. Wed 03Sep03Simon George RHUL5 ATLAS software suite Simulation, data processing and analysis Simulation, data processing and analysis 500 packages, 50 external, inter-dependent. 500 packages, 50 external, inter-dependent. 100s of developers and 1000s of users in 140 institutes 100s of developers and 1000s of users in 140 institutes One release build is 2.5 GB of files One release build is 2.5 GB of files It takes 10 hours to build It takes 10 hours to build Build types and frequencies Build types and frequencies Production release 3-4 times per year Production release 3-4 times per year Developer release every 2-3 weeks Developer release every 2-3 weeks Nightly build of snapshot Nightly build of snapshot Build configuration permutations Build configuration permutations Optimised, debug and sometimes also profile builds. Optimised, debug and sometimes also profile builds. Two platforms (RedHat 7.3 on Intel x86, Solaris 8 on SPARC) Two platforms (RedHat 7.3 on Intel x86, Solaris 8 on SPARC) One or more compilers (gcc 3.2) One or more compilers (gcc 3.2) Config. management, build and install handled by CMT Config. management, build and install handled by CMT So not a trivial task to package, distribute and install So not a trivial task to package, distribute and install

6. Wed 03Sep03Simon George RHUL6 CMT Configuration management tool Configuration management tool Concerned with setting up users environment to build and run software Concerned with setting up users environment to build and run software Needs help of tools for a large project Needs help of tools for a large project CMT helps to define and impose conventions CMT helps to define and impose conventions For naming packages, files, directories For naming packages, files, directories For describing their relationships For describing their relationships In other words, package metadata In other words, package metadata This is the key feature exploited for this project. This is the key feature exploited for this project. Useful features to manage sub projects, dependencies Useful features to manage sub projects, dependencies A broad user base, especially in Particle Physics and Astronomy experiments. A broad user base, especially in Particle Physics and Astronomy experiments. CMT www.cmtsite.org

7. Wed 03Sep03Simon George RHUL7 Packaging Requirements Three types of kit required Three types of kit required Binary kit Binary kit Pre-built executables, libraries and configuration files needed to run the softwarePre-built executables, libraries and configuration files needed to run the software Used for data challenges, production, basic usersUsed for data challenges, production, basic users Developers kit Developers kit Binary kit plusBinary kit plus Headers, libraries and configuration needed to build against itHeaders, libraries and configuration needed to build against it For developers and most usersFor developers and most users Full source kit Full source kit To rebuild from scratch on binary-incompatible platformsTo rebuild from scratch on binary-incompatible platforms When local source code browsing is requiredWhen local source code browsing is required For each permutation of platform, config, compiler For each permutation of platform, config, compiler

8. Wed 03Sep03Simon George RHUL8 Installation requirements For large facilities: unattended, push button deployment For large facilities: unattended, push button deployment For normal user: relocateable, no root access For normal user: relocateable, no root access Automatic configuration Automatic configuration Updates, multiple versions Updates, multiple versions Avoid duplication and unnecessary downloads Avoid duplication and unnecessary downloads Possibility to take subset of software Possibility to take subset of software Self contained, apart from … Self contained, apart from … Prerequisite software: modest list and automatic check Prerequisite software: modest list and automatic check Set up users environment (e.g. LD_LIBRARY_PATH) Set up users environment (e.g. LD_LIBRARY_PATH) Reversible: uninstall Reversible: uninstall Install and work disconnected from network, e.g. install onto a laptop from CDs Install and work disconnected from network, e.g. install onto a laptop from CDs

9. Wed 03Sep03Simon George RHUL9 Constraints ATLAS software is divided into sub-projects ATLAS software is divided into sub-projects Currently ATLAS and Gaudi Currently ATLAS and Gaudi Could be more in the future, e.g. split ATLAS into simulation and reconstruction Could be more in the future, e.g. split ATLAS into simulation and reconstruction Each sub-project consists off many packages Each sub-project consists off many packages External/Internal package distinction External/Internal package distinction Internal packages are developed and managed within the ATLAS software project Internal packages are developed and managed within the ATLAS software project External packages are the opposite, e.g. software from the Particle Physics community, public domain software or commercial products. External packages are the opposite, e.g. software from the Particle Physics community, public domain software or commercial products. Interface packages for externals Interface packages for externals Pure metadata packagePure metadata package Actual external sw can be installed anywhere, any way.Actual external sw can be installed anywhere, any way. Gives it the outside appearance of an internal packageGives it the outside appearance of an internal package

10. Wed 03Sep03Simon George RHUL10 Constraints, continued Existing use of CMT Existing use of CMT Package structure already in place Package structure already in place Meta data provided by packages or implied by default policies is already enough for automated packaging. Meta data provided by packages or implied by default policies is already enough for automated packaging. Problems Problems ATLAS software is written by large communities with a mixed level of experience ATLAS software is written by large communities with a mixed level of experience All such software projects will have small flaws introduced in each release All such software projects will have small flaws introduced in each release These must be worked around when they impact on the packaging. These must be worked around when they impact on the packaging. For example, one problem of particular relevance to packaging & installation is cyclic dependencies For example, one problem of particular relevance to packaging & installation is cyclic dependencies

11. Wed 03Sep03Simon George RHUL11 Packaging: starting point One kit per package One kit per package Follow existing granularity Follow existing granularity Separate metadata and payload Separate metadata and payload Two parts to each kit Two parts to each kit Performed by librarian as integral part of release procedure Performed by librarian as integral part of release procedure Distribution by web or distributed filesystem (e.g. AFS) Distribution by web or distributed filesystem (e.g. AFS)

12. Wed 03Sep03Simon George RHUL12 Tools used CMT CMT Define and impose conventions on packages Define and impose conventions on packages Query the metadata needed for packaging Query the metadata needed for packaging Pacman Pacman Metadata format Metadata format Tool used to manage kit installation Tool used to manage kit installation Tar and RPM Tar and RPM Payload format – the package itself Payload format – the package itself Deployment tools shell scripts Deployment tools shell scripts Construct the kits using CMT Construct the kits using CMT Control location of Pacman cache and distribution Control location of Pacman cache and distribution Post-installation configuration Post-installation configuration

13. Wed 03Sep03Simon George RHUL13 Overview of process and tools Create kits Web server or AFS Local computers CMT Pacman Developer Local s/w manager Librarian Deployment Tools Fetch packages Install and configure Configure environment Compile and link own library Run software

14. Wed 03Sep03Simon George RHUL14 Pacman A package manager A package manager Packager defines how the software should be fetched, installed, configured, updated, in a Pacman file. The package itself can be in any format as that file is separate. Packager defines how the software should be fetched, installed, configured, updated, in a Pacman file. The package itself can be in any format as that file is separate. A directory of these files is known as a cache, usually available on the web. A directory of these files is known as a cache, usually available on the web. Pacman tool is used to install the software Pacman tool is used to install the software Pacmans feature list is a good match to the requirements for installation. Pacmans feature list is a good match to the requirements for installation. Already used by several Particle Physics and GRID projects. Already used by several Particle Physics and GRID projects. http://physics.bu.edu/~youssef/pacman

15. Wed 03Sep03Simon George RHUL15 Package distribution format Tar vs. RPM Tar vs. RPM Both can be made relocateable Both can be made relocateable Feature set Feature set Tar has a simple feature set but is complementary to CMT and Pacman Tar has a simple feature set but is complementary to CMT and Pacman RPM overlaps with CMT and Pacman RPM overlaps with CMT and Pacman e.g. RPM also handles dependencies and prerequisitese.g. RPM also handles dependencies and prerequisites Platforms Platforms RPM is only widely used on Linux, while tar is standard on pretty much any Unix RPM is only widely used on Linux, while tar is standard on pretty much any Unix Annoyances Annoyances Default RPM database needs root access to write to it Default RPM database needs root access to write to it There are workarounds for this but not prettyThere are workarounds for this but not pretty Conclusion Conclusion Decided to use tar Decided to use tar but retained RPM as an option but retained RPM as an option

16. Wed 03Sep03Simon George RHUL16 Meta data For each package For each package Other packages it uses (dependencies) Other packages it uses (dependencies) Location of constituents Location of constituents Applications and librariesApplications and libraries Header filesHeader files Run time/config filesRun time/config files CMT requirements fileCMT requirements file External packages External packages Pure meta data glue packages Pure meta data glue packages Just define paths to export Just define paths to export All defined in CMT requirements files All defined in CMT requirements files or implied by default conventions of ATLAS or implied by default conventions of ATLAS Can be queried through cmt Can be queried through cmt cmt show uses cmt show uses cmt show macro _export_paths cmt show macro _export_paths

17. Wed 03Sep03Simon George RHUL17 Naming and structure Package naming convention Package naming convention Packages in a sub-project Packages in a sub-project - - External packages External packages - - These names are used when expressing the inter-package dependencies These names are used when expressing the inter-package dependencies Directory structure within each kit Directory structure within each kit / /InstallArea/ / /InstallArea/ contains the sub-directories bin, lib, include, share.contains the sub-directories bin, lib, include, share. / / / /cmt/ / / / /cmt/ Contains the configuration management filesContains the configuration management files / / Assumed to have their own internal structure for versions & buildsAssumed to have their own internal structure for versions & builds This is designed to support coexistence of: This is designed to support coexistence of: Different versions of every piece of software Different versions of every piece of software Different binary versions (platform and build config) Different binary versions (platform and build config)

18. Wed 03Sep03Simon George RHUL18 Examples CMT requirements file: package ExamplePkgA author A. Person use ExamplePkgB use ExampleExtPkg library ExamplePkgA *.cxx apply pattern component_library apply pattern declare_runtime Package name and author Inter-package dependencies Instruction to build a library from source files Type of library to build, implies library file names Default location implied Pacman file: description=Package ExamplePkgA-01-07-02 in release 6.5.0 url=http://atlas.web.cern.ch/Atlas/GROUPS/SOFTWARE/OO source=../dist download = { *:ExamplePkgA-6.5.0.tar.gz } depends = [ ExamplePkgB-6.5.0, ExampleExtPkg-v1 ]

19. Wed 03Sep03Simon George RHUL19 Creating the kits First, build a release First, build a release Discover cycles in the dependencies Discover cycles in the dependencies Use a feature of CMT to discover cycles in the dependencies, as these must not be propagated to the kits. Record the output in a file. Use a feature of CMT to discover cycles in the dependencies, as these must not be propagated to the kits. Record the output in a file. Then, use a feature of CMT to visit every package in a dependency tree and apply a command there Then, use a feature of CMT to visit every package in a dependency tree and apply a command there cmt broadcast cmt broadcast Usage of the script to create a kit: Usage of the script to create a kit: create_kit.sh –release -cycles [-rpm] create_kit.sh –release -cycles [-rpm] Creates a pacman file and tar file, optional RPM file Creates a pacman file and tar file, optional RPM file Finally, there are often a few things to fix by hand specific to each release. Finally, there are often a few things to fix by hand specific to each release. Note that CMT itself is included as a kit Note that CMT itself is included as a kit

20. Wed 03Sep03Simon George RHUL20 Installation Performed by site software manager or end user on desktop or laptop Performed by site software manager or end user on desktop or laptop Straightforward procedure: Straightforward procedure: Install Pacman, if not already done Install Pacman, if not already done Install prerequisite software Install prerequisite software Currently just RedHat 7.3 o/s, gcc-3.2 and Java SDK 1.4.1Currently just RedHat 7.3 o/s, gcc-3.2 and Java SDK 1.4.1 Choose directory for the installation Choose directory for the installation Probably the same as beforeProbably the same as before Choose which release to install Choose which release to install Available releases are listed on a web pageAvailable releases are listed on a web page Use Pacman to download, install and configure it, e.g. pacman –get ATLAS:AtlasRelease-6.5.0 Use Pacman to download, install and configure it, e.g. pacman –get ATLAS:AtlasRelease-6.5.0 Dependencies followed automatically to get everything you needDependencies followed automatically to get everything you need Optionally, run script to set up a user environment and run a test Optionally, run script to set up a user environment and run a test User configures software in the usual way User configures software in the usual way Just choose release and private working area as normal Just choose release and private working area as normal Run a setup script provided by CMT Run a setup script provided by CMT

21. Wed 03Sep03Simon George RHUL21 Conclusions Procedures and tools have been developed for the packaging, distribution and installation of ATLAS software Procedures and tools have been developed for the packaging, distribution and installation of ATLAS software Based on Pacman, CMT, tar/rpm and some shell scripts Based on Pacman, CMT, tar/rpm and some shell scripts The basic principles could be applied more generally The basic principles could be applied more generally Using some or all of the same tools Using some or all of the same tools It satisfies most of the requirements for run-time and developers kits and for installation. It satisfies most of the requirements for run-time and developers kits and for installation. Full source kit still to be done. Full source kit still to be done. Early adopters have given useful feedback and it is now being imported into Grid production systems Early adopters have given useful feedback and it is now being imported into Grid production systems Must now move to its use as part of the standard release procedure in ATLAS Must now move to its use as part of the standard release procedure in ATLAS by December 2003, for our global `Data Challenge 2' by December 2003, for our global `Data Challenge 2'

22. Wed 03Sep03Simon George RHUL22 Future developments Better handling of prerequisite software and platform compatibility checks Better handling of prerequisite software and platform compatibility checks EDG WP4 configuration management task EDG WP4 configuration management task Potential to work with an installation on demand mechanism for GRID farms Potential to work with an installation on demand mechanism for GRID farms LCG/EDG/iVDGL GLUE LCG/EDG/iVDGL GLUE Meta packaging proposal for Grid middleware and applications, O. Barring et al. Meta packaging proposal for Grid middleware and applications, O. Barring et al. Pacman version 3 Pacman version 3