Managing Combinatorial Software Installations with Spack Gregory Becker, Peter Scheibel, Todd Gamblin, Matt LeGendre

There are three levels in the HPC software stack. Application codes Managed by our users, often using Spack for combinatorial versioning Site Deployed Software User support group staff deployed– compilers, MPI implementations, math libraries, etc System Software Single software stack managed by TOSS and RPM Livermore computing has 3,000 users running various scientific applications Users depend on combinatorial versioning in this space to enable their combinatorial versioning The Tri-lab Operating System Stack is our in-house operating system. LC administers over 30 clusters

Users require combinatorial versions of software. HPC software installation is a massively combinatorial space 2 versions of FFTW with… 4 MPI implementations mvapich mvapich2 mpich OpenMPI 2-3 versions of each MPI implementation x 3-ish Platforms Linux BlueGene PowerPC x Up to 6 compilers Intel GCC XLC Clang PGI Pathscale x Oh, and 4-5 versions of each compiler x = ~2,000 combinations Support tens or hundreds of software packages Any one may have 2000 possible configurations Our previous methods did not support the combinatorial versioning users require Manual deployment does not scale.

We re-engineered our software environment to support simple views of combinatorial software. We had the opportunity to do software deployment RightTM Administrators can manage site-deployed software easily Spack Advanced users can select specific versions of software packages Combinatorial versioning Lmod Policy decisions System software works out of the box Environment Independence RPATH

Spack handles combinatorial software complexity. Dependency DAG Each node in the DAG has build metadata Each unique dependency graph is a unique configuration. Each configuration installed in a unique directory. Configurations of the same package can coexist. Hash of entire directed acyclic graph (DAG) is appended to each prefix. Installation Layout Hash spack/opt/ linux-x86_64/ gcc-4.7.2/ mpileaks-1.1-0f54bf34cadk/ intel-14.1/ hdf5-1.8.15-lkf14aq3nqiz/ bgq/ xl-12.1/ hdf5-1-8.16-fqb3a15abrwx/ ...

Spack parameterizes the build. Manage combinatorial build spaces Refer to any point in the space Build any point in the space Leverages a build recipe to complete the build from spack import * class Dyninst(Package): """API for dynamic binary instrumentation."”” homepage = "https://paradyn.org” version('8.2.1', 'abf60b7faabe7a2e’, url="http://www.paradyn.org/release8.2/DyninstAPI-8.2.1.tgz") version('8.1.2', 'bf03b33375afa66f’, url="http://www.paradyn.org/release8.1.2/DyninstAPI-8.1.2.tgz") depends_on("libelf") depends_on("libdwarf") depends_on("boost@1.42:") def install(self, spec, prefix): libelf = spec['libelf'].prefix libdwarf = spec['libdwarf'].prefix with working_dir('spack-build', create=True): cmake('..', '-DBoost_INCLUDE_DIR=%s' % spec['boost'].prefix.include, '-DBoost_LIBRARY_DIR=%s' % spec['boost'].prefix.lib, '-DBoost_NO_SYSTEM_PATHS=TRUE’ *std_cmake_args) make() make("install") @when('@:8.1') configure("--prefix=" + prefix)

Spack provides a spec syntax to describe customized DAG configurations. $ spack install mpileaks unconstrained $ spack install mpileaks@3.3 @ custom version $ spack install mpileaks@3.3 %gcc@4.7.3 % custom compiler $ spack install mpileaks@3.3 %gcc@4.7.3 +threads +/- build option $ spack install mpileaks@3.3 cppflags=\”-O3\” setting compiler flags $ spack install mpileaks@3.3 os=CNL10 target=haswell setting target for X-compile $ spack install mpileaks@3.3 ^mpich@3.2 %gcc@4.9.3 ^ dependency information Each expression is a spec for a particular configuration Each clause adds a constraint to the spec Constraints are optional – specify only what you need. Customize install on the command line! Spec syntax is recursive Full control over the combinatorial build space

Users cannot navigate a fully combinatorial install space easily. The full build space is to large to manage in a multi-user context Spack treats every possible change as a dimension of the build space because it is conservative about ABI compatibility Need upgrade-in-place capabilities We don’t want users to have to rebuild for every possible update For a particular package, many of the possible dimensions may be irrelevant Requires expert knowledge hdf5 @1.10.0 zlib %gcc @4.9.3 @0.8.13 target=x86_64 %gcc @4.9.3 mpich zlib @3.2 target=x86_64 @0.8.13 %gcc %gcc @4.9.3 @4.9.3 target=x86_64 target=x86_64

We represent subspaces with a string. String refers to dimensions we care about Can also include arbitrary strings Includes a syntax for referencing elements of the Spack spec NAME VERSION COMPILER COMPILERNAME COMPILERVER Dependencies handled recursively DEP:name:VERSION Etc. {NAME}-{VERSION}-{COMPILER}-{DEP:mpi:NAME}-{DEP:mpi:VERSION}-hdf5-{DEP:hdf5:VERSION}

We provide a configuration file so that software deployers can define subspaces on a per-package basis. Subspaces must be configured on a per-package basis. An implementation of subspaces consists of a language for representing projections of the build space. Names determine upgrade policy packages: zlib: subspaces: subspace1: name: {NAME}-{VERSION} prefix: /usr/tce/tools hdf5: hdf5subspace: name: {NAME}-{VERSION}-{COMPILER}-{DEP:mpi:NAME}-{DEP:mpi:VERSION}

We use Spack to drive the RPM package manager. Administrators can manage and deploy combinatorially versioned software Spack will generate necessary files to drive build system for package and dependencies RPM package management depends on names – generated by subspaces

We want to build RPMs for each configuration in the subspace. Package configuration manual /usr/tce/packages/ Administrators can manage and deploy combinatorially versioned software Using a tool they know At a system level Spack will generate an RPM spec for a given package and configuration Using build recipe Spec contains all information to build the package Generates dependency information

So, where do we stand so far? We had the opportunity to do things RightTM Administrators can manage package installations Spack Advanced users can select specific versions Combinatorial versioning Lmod Policy decisions Things work out of the box Environment Independence RPATH

A combinatorial build space requires usability features to navigate. Modules Provide defaults Handle environment changes

Lmod provides hierarchical modules to navigate the combinatorial space. $ ssh tce-machine $ module avail ------------------- /usr/tce/modulefiles/MPI/intel/16.0.3/mvapich2/2.2 ---------------------- MUST/1.5 boost/1.62.0 fftw/3.3.4 hdf5-parallel/1.8.17 mpip/3.4.1 ------------------------ /usr/tce/modulefiles/Compiler/intel/16.0.3 ------------------------- hdf5/1.8.17 hpctoolkit/10102016 impi/5.1.3 impi/2017.0 (D) mvapich2/2.2 (L) openmpi/2.0.0 --------------------------------- /usr/tce/modulefiles/Core --------------------------------- StdEnv (L) gcc/4.8-redhat graphlib/3.0.0 (D) intel/17.0.0 python/3.3.1 dyninst/9.1.0 gcc/4.9.3 intel/16.0.3 (L,D) python/2.7.11 (D) totalview/2016.01.06 Where: L: Module is loaded, D: Default Module $ module show mvapich2 ----------------------------------------------------------------------------------------------------------------------------------- /usr/tce/modulefiles/Compiler/intel/16.0.3/mvapich2/2.2.lua: whatis("mpi/mvapich2") whatis("MVAPICH2-2.2 MPI for Intel-16.0.3 compilers") family("mpi") prepend_path("PATH", "/usr/tce/packages/mvapich2/mvapich2-2.2-intel-16.0.3/bin") ...

Lmod’s hierarchy presents a slice of the build space to users. $ module swap intel gcc $ module avail ----------------------------------------- /usr/tce/modulefiles/MPI/gcc/4.9.3/mvapich2/2.2 ------------------------------------------ MUST/1.5 boost/1.62.0 hdf5-parallel/1.8.17 mpip/3.4.1 --------------------------------------------- /usr/tce/modulefiles/Compiler/gcc/4.9.3 ---------------------------------------------- gperftools/2.5 hdf5/1.8.17 hpctoolkit/10102016 impi/5.1.3 impi/2017.0 (D) mvapich2/2.2 openmpi/2.0.0 ---------------------------------------------------- /usr/tce/modulefiles/Core ----------------------------------------------------- StdEnv (L) gcc/4.8-redhat graphlib/3.0.0 (D) intel/17.0.0 python/3.3.1 dyninst/9.1.0 gcc/4.9.3 intel/16.0.3 (D) python/2.7.11 (D) totalview/2016.01.06 Where: L: Module is loaded, D: Default Module $ module show mvapich2 ----------------------------------------------------------------------------------------------------------------------------------- /usr/tce/modulefiles/Compiler/gcc/4.9.3/mvapich2/2.2.lua: whatis("mpi/mvapich2") whatis("MVAPICH2-2.2 MPI for GNU-4.9.3 compilers") family("mpi") prepend_path("PATH", "/usr/tce/packages/mvapich2/mvapich2-2.2-gcc-4.9.3/bin”) ...

A combinatorial build space requires usability features to navigate. Modules Provide defaults Handle environment changes Package visibility Supporting users of various knowledge levels Testing Deprecation Defaults

We maintain levels of visibility for our software packages. Base Packages Public Packages Default Packages $ ls /usr/tce/packages advisor cmake gcc git mvapich2 boost fftw hdf5 make openmpi ... $ ls /usr/tce/packages/gcc default gcc-4.9.3 gcc-6.1.0 gcc-experimental $ module avail gcc -------------- /usr/tce/modulefiles/Core -------------- gcc/4.9.3 (L,D) gcc/6.1.0 ... $ ls –la /usr/tce/bin lrwxrwxrwx 1 root root 39 Sep 28 08:46 git -> /usr/tce/packages/git/git-2.8.3/bin/git lrwxrwxrwx 1 root root 24 Sep 28 08:47 icc -> /usr/tce/bin/icc-16.0.3 One per package Installs the package itself Allows advanced users to find any package installed on the system Installs into /usr/tce/packages One for every base package “Publish” the package for users Adds the package’s module files to /usr/tce/modulefiles May install symlinks to the package from /usr/tce/bin using the full name (E.g. “gcc- 4.9.3”) One per package name Has a version number that aligns with one version of the package Makes its version the default Lmod module for its package May install symlinks to the package from /usr/tce/bin using the base name (E.g. “gcc”)

A combinatorial build space requires usability features to navigate. Modules Provide defaults Handle environment changes Package visibility Testing Deprecation Defaults RPATH Linking errors are among the most common errors Effected by environment changes Change between link time and run time

We use RPATH to mitigate the effect of the user environment. Changes to environment should not affect behavior of executables. RPATHs for compiler libraries Compiler runtimes are found from the environment Can involve arcane dependencies If I install FFTW with the gcc compiler at version 6.1.0, that FFTW should always run with the runtime for gcc 6.1.0 FFTW gcc@6.1.0 gcc@6.1.0 runtime gcc@4.9.3 runtime fftw-3.3-gcc-6.1.0

We use RPATH to mitigate the effect of the user environment. Changes to environment should not affect behavior of executables. RPATHs for compiler libraries Compiler runtimes are found from the environment Can involve arcane dependencies RPATHs for dependency information New installations should not change previous behavior hdf5@1.10.0 hdf5-1.10.0-mpich-3.1-gcc-6.1.0 mpich@3.1 mpich@3.2

Summary Software installations require combinatorial versioning Hard to deploy Hard to manage the user environment Spack provides the engine to do combinatorial versioning Designed as a user-space tool Subspaces allow us to control policy for Spack installations Upgrade-in-place Naming conventions RPM generation for binary installation Allow Spack to drive system level installations Usability features Modules Policies

Future Work Automating updates Allow for multiple packages to be updated automatically Upgrade all dependent packages when a dependency is upgraded Some packages are still manually deployed Create Spack packages (build recipes) for all software we deploy

If you have questions about Spack, come to our tutorial! 1:30 PM in room 255-F