Using Maali to Efficiently Recompile Software Post-CLE Upgrades on a CRAY XC System Chris Bording, Chris Harris and David Schibeci Pawsey Supercomputing Centre, Perth, Western Australia

Maali Origins The Pawsey Supercomputing Centre, nee iVEC has been one of the fasting growing Petascale HPC centres measured in FLOPS in the last five years. Has gone from < 10 Tflops on a single system to having six systems with 3 Cray XC systems, including Magnus which is ranked #41 on the Top 500.

In the beginning there was two EPIC Not-a-Cray 800 nodes(9600-cores) Intel X5660 Westmere 24 GB memory QDR infiniband Centos 6.3 Lustre PBS Pro 11.3 Fornax Not-a-Cray 96 nodes (1152 -cores) Intel X5650 Westmere 72 GB memory 7TB local disk / node Nvidia Tesla C2075/node Centos 6.3 Lustre PBS Pro 11.3

Maali Origins Multiple systems with different hardware configurations. More projects to support. More researchers with more complex software applications and libraries to support. The Pawsey Supercomputing Centre is going to be a Petascale centre.

Initial Design Criteria Needs to be simple! Establish Conventions and Policies that support those Conventions. Reduce staff effort to maintain the the software stack. Greater use of automation.

A brief primer on compiling and installing software Download source Configure code Compile code Install code Create an environment Module Document process

MaaliCore design Maali is a set of BASH scripts designed initially to allow for the automation of the Autoconf/Automake process of “configure”, “make” and “make install”. Uses a system-level configuration file that defines the Maali environment.

Maali Automatically Downloads applications with wget and maintains a repo of source packages. Runs the configure/cmake make make install Writes a new environment module file. Generates a separate build log file for each application

Maali System Configure File

Maali System Configuration Keypair Values MAALI_OS=cle52 MAALI_ROOT=”/ivec/$MAALI_OS/$MAALI_SY STEM” MAALI_SRC=”$MAALI_ROOT/src” MAALI_BUILD_DIR=”$MAALI_ROOT/build" MAALI_MODULE_DIR="$MAALI_ROOT/modul efiles"

Maali Build Directory The MAALI_BUILD_DIR is the keypair value that defines the absolute path to the directory where here the package sources are copied to from the MAALI_SRC directory then uncompressed and extracted. Ideally this should be somewhere in on the scratch working directory as builds take a lot of space. Not necessary to maintain.

Maali Source Directory MAALI_SRC is the keypair value that defines the absolute path to the directory that is the repository where all the original source packages maintained as tar-balls and compressed formats. Needs to be on a partition of sufficient size. Backed up if possible.

Maali Modulefiles Directory The MAALI_MODULE_DIR is the absolute path where Maali will install all new modules. The MAALI_MODULE_DIR will need to be added to the default MODULEPATH.

Maali build – core functions maali_build Function maali_build { # this is the core function for creating software # allows late evaluation MAALI_TOOL_CONFIGURE_EVAL=`eval echo ”$MAALI_TOOL_CONFIGURE"` cd ”$MAALI_TOOL_BUILD_DIR" maali_run "./configure --prefix=$MAALI_INSTALL_DIR $MAALI_TOOL_CONFIGURE_EVAL" maali_run "make" maali_run "make install" }

Maali cmake build- core function maali cmake build function maali_cmake_build { # for tools that use cmake # allows late evaluation MAALI_TOOL_CONFIGURE_EVAL=`eval echo "$MAALI_TOOL_CONFIGURE"` cd "$MAALI_TOOL_BUILD_DIR" # cmake likes to build in a director of it's own mkdir "$MAALI_TOOL_NAME-build" cd "$MAALI_TOOL_NAME-build" maali_run "cmake -DCMAKE_INSTALL_PREFIX=$MAALI_INSTALL_DIR $MAALI_TOOL_CONFIGURE $MAALI_CMAKE_PATH" if [ $DEBUG ]; then maali_run "make VERBOSE=TRUE" else maali_run "make" fi maali_run "make install" }

Maali Build Scripts Maali uses a build script for each application or library that defines: The environment (PrgEnv-*) Dependent environment modules (cray-hdf, etc) The configuration and optimization flags. Defines a set “Maali” variables to generate a new environment module.

Szip build file ############################################################################## # $Id: szip.maali,v /10/24 02:48:41 stapops Exp $ ############################################################################## # maali config file for szip ############################################################################## # specify which compilers we want to build the tool with MAALI_TOOL_COMPILERS="$MAALI_DEFAULT_COMPILERS" # URL to download the source code from MAALI_URL=" $MAALI_TOOL_VERSION.tar.gz" # location we are downloading the source code to MAALI_DST="$MAALI_SRC/$MAALI_TOOL_NAME-$MAALI_TOOL_VERSION.tar.gz" # where the unpacked source code is located MAALI_TOOL_BUILD_DIR="$MAALI_BUILD_DIR/$MAALI_TOOL_NAME-$MAALI_TOOL_VERSION" # type of tool (eg. apps, devel, python, etc.) MAALI_TOOL_TYPE="devel" # for auto-building module files MAALI_MODULE_SET_LD_LIBRARY_PATH=1 MAALI_MODULE_SET_CPATH=1 MAALI_MODULE_SET_FPATH=1 MAALI_MODULE_SET_FCPATH=1 MAALI_MODULE_WHATIS="SZIP compression software, providing lossless compression of scientific data."

Silo build script key elements # specify which compilers we want to build the tool with MAALI_TOOL_COMPILERS="$MAALI_DEFAULT_COMPILERS" # URL to download the source code from MAALI_URL=" # location we are downloading the source code to MAALI_DST="$MAALI_SRC/$MAALI_TOOL_NAME-$MAALI_TOOL_VERSION.tar.gz" # where the unpacked source code is located MAALI_TOOL_BUILD_DIR="$MAALI_BUILD_DIR/$MAALI_TOOL_NAME-$MAALI_TOOL_VERSION" # type of tool (eg. apps, devel, python, etc.) MAALI_TOOL_TYPE="devel" # tool pre-requisites MAALI_TOOL_PREREQ="hdf5/1.8.12" MAALI_TOOL_CONFIGURE='--enable-silex --enable-fortran --enable-shared --with- hdf5=$HDF5_DIR/include,$HDF5_DIR/lib' MAALI_TOOL_CONFIGURE_NOHDF5='--enable-silex --enable-fortran --enable-shared' # for auto-building module files MAALI_MODULE_SET_PATH=1 MAALI_MODULE_SET_LD_LIBRARY_PATH=1 MAALI_MODULE_WHATIS="Silo is a library which implements and application programing interface(API) designed for reading and writing a wide variety of scientific data to binary, disk files. The files that Silo produces and the data within them can be easily shared and exchanged between wholly independently developed applications running on disparate computing platforms”

Silo build file build function ############################################################################## function maali_build { cd "$MAALI_TOOL_BUILD_DIR" MAALI_TOOL_CONFIGURE_EVAL=`eval echo "$MAALI_TOOL_CONFIGURE_NOHDF5"` maali_run "./configure --prefix=$MAALI_INSTALL_DIR $MAALI_TOOL_CONFIGURE_EVAL" maali_run "make" maali_run "make install" maali_run "make clean" MAALI_TOOL_CONFIGURE_EVAL=`eval echo "$MAALI_TOOL_CONFIGURE"` maali_run "./configure --prefix=$MAALI_INSTALL_DIR $MAALI_TOOL_CONFIGURE_EVAL" maali_run "make" maali_run "make install" } ##############################################################################

GMT build file using cmake # maali config file for GMT # specify which compilers we want to build the tool with MAALI_TOOL_COMPILERS="$MAALI_DEFAULT_GCC_COMPILERS" # URL to download the source code from MAALI_URL=" # location we are downloading the source code to MAALI_DST="$MAALI_SRC/${MAALI_TOOL_NAME}-${MAALI_TOOL_VERSION}-src.tar.bz2" # where the unpacked source code is located MAALI_TOOL_BUILD_DIR="$MAALI_BUILD_DIR/${MAALI_TOOL_NAME}-${MAALI_TOOL_VERSION}" # type of tool (eg. apps, devel, python, etc.) MAALI_TOOL_TYPE="devel” # tool pre-requisites MAALI_TOOL_PREREQ="cray-netcdf/4.3.2 fftw/ gdal/1.11.1" # tool build pre-requisites - not added to the module, only needed for building (loaded after MAALI_TOOL_PREREQ) MAALI_TOOL_BUILD_PREREQ="cmake/ " # add additional configure options MAALI_TOOL_CONFIGURE='-DFFTW3_INCLUDE_DIR=$FFTW_INC -DFFTW3F_LIBRARY=$FFTW_DIR/libfftw3f.a' # for auto-building module files MAALI_MODULE_SET_PATH=1 MAALI_MODULE_SET_MANPATH=1 MAALI_MODULE_SET_LD_LIBRARY_PATH='lib64’ MAALI_MODULE_SET_WHATIS="GMT is an open source collection of about 80 command-line tools for the manipulating geographic and Cartesiandata sets (including filtering, trend setting, gridding,projecting,etc.) and producing PostScript illustrations ranging form simple x0y plots via contour maps to artificially illuminated surfaces adn 3D perspective views; the GMT supplements add another 40 more specialized and discipline-specific tools." export CRAYPE_LINK_TYPE='dynamic'

Maali build files with cmake # tool pre-requisites MAALI_TOOL_PREREQ="cray-netcdf/4.3.2 fftw/ gdal/1.11.1" # tool build pre-requisites - not added to the module, only needed for building (loaded after MAALI_TOOL_PREREQ) MAALI_TOOL_BUILD_PREREQ="cmake/ " # add additional configure options MAALI_TOOL_CONFIGURE='-DFFTW3_INCLUDE_DIR=$FFTW_INC - DFFTW3F_LIBRARY=$FFTW_DIR/libfftw3f.a' # for auto-building module files MAALI_MODULE_SET_PATH=1 MAALI_MODULE_SET_MANPATH=1 MAALI_MODULE_SET_LD_LIBRARY_PATH='lib64’

Haswell Sprint/CLE52 upgrade Magnus all the compute nodes where upgraded from Intel Sandybridge to Intel Haswell. Cray Linux Environment upgrade 50 to 52 Magnus went into production January 2015!

Application Support at the Pawsey A set of 31 applications/libraries were identified from the early adopter’s projects on Magnus. These included: Siesta exabayes mrbayes Beagle-lib Cmake gsl lapack xeres-c ephem gamess qbox ncview Hypre Gromacs Amber Libffi Glib Udunits gts zlib szip scons numpy scipy mercurial Distribute astropy d2to1 python

Maali Command maali –t python –v –c magnus maali –t python –v –c magnus maali –t scons –v –c magnus maali –t numpy –v –c magnus maali –t scipy –v –c magnus maali –t gsl –v 1.16 –c magnus

Haswell Sprint Outcomes Effort was spread out over 2 days with 7 members of the Supercomputing team. Majority of packages installed in a couple of hours. The Sprint was a success!

Haswell Sprint review Be smarter about using Maali, use current source repo, use the compute nodes. Improve the dependency documentation Create a verification and validation tests using the “must” build packages. Validating build outcomes automatically. Did it install stuff in /bin,/lib,/man correctly?

Maali System Configuration Keypair Values MAALI_OS=cle52 MAALI_ROOT=”/ivec/$MAALI_OS/$MAALI_SY STEM” MAALI_BUILD_DIR=”$MAALI_ROOT/build" MAALI_MODULE_DIR="$MAALI_ROOT/modul efiles" MAALI_SRC=”$MAALI_ROOT/src”

Simple Maali bash #!/bin/bash module load maali/1.0b1 maali –t python –v –c magnus maali –t scons –v –c magnus maali –t numpy –v –c magnus maali –t scipy –v –c magnus maali –t gsl –v 1.16 –c magnus

Maali Future OPEN SOURCE in May 2015 Improve the module files prereqs and dependencies and conflicts. Implemented on NeCTAR research cloud to use for bioinformatics pipeline construction. Regression testing of build scripts Improve wiki page creation.

Acknowledgements David Schibeci - the lead developer for Maali. Chris Harris- leading and organizing the Haswell sprint. Ashley Chew – System admin on Fornax. The Supercomputing team Rebecca Hartman-Baker, Paul Ryan, Daniel Grimwood, Charlene Yang, Brian Skjerven,Kevin Stratford, Moshin Shaikh at the Pawsey Supercomputing Centre. George Beckett who leads the Supercomputing Team and Neil Stringfellow the Executive Director at the Pawsey Supercomputing Centre for their support.

2 nd International HPC User Support Tools Workshop HUST15 at SC15 in Austin