Black All good PowerPoint starts with black.

Compiling VTK-m/Ascent on Stampede2 with Spack Kenneth Moreland, Sandia National Laboratories

Issue 1: Spack sets up compilers wrong Spack finds gcc 6.3.0 and Intel 18.0.2 in your path But, both rely on their respective module to be loaded So, you download Spack, ask it to install a simple package (like libelf), and it fails with an error like configure: error: installation or configuration problem: C compiler cannot create executables.

Issue 1 Solution: Add module to compiler configuration Apparently Spack gets it on some systems (like Cray); not Stampede This should be pushed back to the Spack team From compilers.yaml: - compiler: environment: {} extra_rpaths: [] flags: {} modules: [intel/18.0.2] operating_system: centos7 paths: cc: /opt/intel/compilers_and_libraries_2018.2.199/linux/bin/intel64/icc cxx: /opt/intel/compilers_and_libraries_2018.2.199/linux/bin/intel64/icpc f77: /opt/intel/compilers_and_libraries_2018.2.199/linux/bin/intel64/ifort fc: /opt/intel/compilers_and_libraries_2018.2.199/linux/bin/intel64/ifort spec: intel@18.0.2 target: x86_64

Issue 2: Spack wants to compile CMake The point of CMake is that you have one thing that can manage configuration Spack wants to compile its own version, which requires the installation of about a dozen dependent packages (including autotools), which defeats the purpose Also failed to build because gnutils couldn’t seem to get guile correctly Many HPC systems (including Stampede) have CMake installed, and it is trivial to download and use your own binary (for x86 architecture)

Issue 2 Solution: Configure external packages for CMake You can point to either an external module or executable for CMake Configuration is in packages.yaml It would save a lot of people a lot of time if Spack did this automatically. CMake is ubiquitous enough to have a special CMake set up like compilers packages: cmake: modules: cmake@3.10.2: cmake/3.10.2 paths: cmake@3.9.6: ~/local/stow/cmake-3.9.6-Linux-x86_64 buildable: False

Issue 3: Spack Intel-mpi configuration broken for Stampede Spack makes some assumptions about how Intel-mpi is installed that is violated with Stampede Cyrus can provide more details

Issue 3 Solution: Workaround by setting up mvapich2 to use Intel-mpi Thanks to Cyrus for the solution! Spack has an open issue to address this https://github.com/spack/spack/issues/11522 https://github.com/spack/spack/pull/10753 In packages.yaml: packages: all: providers: mpi: - mvapich2 mvapich2: buildable: False modules: mvapich2: impi

Issue 4: Executables compiled with Intel compiler module need module loaded to run Ascent needs the python module and several other related modules If python is compiled with the intel compiler on Stampede2 (with, for example, the intel/18.0.2 module), you need that compiler module loaded to run Otherwise you get: error while loading shared libraries: libimf.so: cannot open shared object file: No such file or directory This happens in modules like py-setuptools, which run python scripts but do not compile anything Nothing compiled, no compiler module loaded

Issue 4 Solution: Set compiler configuration to add rpath to compiler-provided libraries Using --dirty flag to spack might also work Open Spack issue: https://github.com/spack/spack/issues/11582 - compiler: environment: {} extra_rpaths: - /opt/intel/compilers_and_libraries_2018.2.199/linux/compiler/lib/intel64_lin flags: {} modules: [intel/18.0.2] operating_system: centos7 paths: cc: /opt/intel/compilers_and_libraries_2018.2.199/linux/bin/intel64/icc cxx: /opt/intel/compilers_and_libraries_2018.2.199/linux/bin/intel64/icpc f77: /opt/intel/compilers_and_libraries_2018.2.199/linux/bin/intel64/ifort fc: /opt/intel/compilers_and_libraries_2018.2.199/linux/bin/intel64/ifort spec: intel@18.0.2 target: x86_64

Issue 5: Ascent package in Spack does not work Ironic since Ascent source build scripts use Spack for compiling But the Ascent build wants to pull its own version of Spack and attempt to add patches to get it to work in known places I don’t want to do that because I want my Ascent build to work with other things I build with Spack It is easier to fix Spack issues in my own environment rather than Ascent’s I would rather submit fixes to Spack and have it fixed for everyone than submit to Ascent and have it work only for Ascent builds Main issue is that Ascent requires specific commits for VTK-m, which are not captured in the Spack repository

Issue 5 Solution: WIP I was working on attempting to set up a version 0.4.0 package of Ascent Requires setting up dependencies to correct versions of VTK-m and VTK-h These might be “random” git commits. This should be easier to manage once VTK-m and VTK-h merge.