CCA Components for Accelerator Physics ComPASS SAP Project and Phase II Doe SBIR Stefan Muszala, Tech-X Corp, Boulder, CO In collaboration with Jim Amundson (FNAL), Lois Curfman McInnes and Boyana Norris (ANL) Christine Roark, Roopa Pundaleeka, Peter Stoltz (Tech-X)

Software reuse and common interfaces Ability to compose simulations Portability Mixed language programming interoperability Performance analysis of composed simulations High-performance accelerator software should allow complex applications while promoting good software engineering practices

Phase 1 recap 1) Design interfaces and implement components for TxPhysics 2) Build Synergia2 components and run an Electron Cloud simulation 3) Begin performance measurement and modeling 4) Additionally, modifications to Bocca Reference: Two-tiered Component Design and Performance Analysis of Synergia2 Accelerator Simulations, S. Muszala, J. Amundson, L. C. McInnes, and B. Norris, accepted for publication in the Proceedings of the 2009 Workshop on Component-Based High Performance Computing, to be held Nov 15-16, 2009, Portland, Oregon.

CQoS (Computational Quality of Service) for accelerator simulations: How, during runtime, can we make make sound choices for reliability, accuracy, and performance, taking into account the problem instance and computational environment? Composition: select initial component implementations and configuration parameters Reconfiguration: change parameters Substitution: change implementations 1) Design and implement CCA Synergia2 components for use on leadership-class supercomputers –Software infrastructure (getting this in place first) –Interfaces and components 2) Explore performance models for parallel architectures networks and algorithms –Performance analysis and modeling –Build CQoS infrastructure 3) Demonstrate and compose various Synergia2 CCA electron cloud component use cases –Compose ECE simulations –Apply CQoS Next major milestones that build toward ECE simulations with CQoS

Building and porting software on Surveyor Babel CCA Tools Synergia VORPAL

Building babel on surveyor Can configure with everything except java support, most (but not all) features are set by runtime/m4/llnl_cross_compiling.m4 Relies on fixing libtool convenience scripts for builds (contrib/libtool_fix.sh) but in particular, for any shared library... find. -name libtool -exec sed -il -e 's|^archive_cmds="\(....\).*|archive_cmds="\1 \\$libobjs \\$deplibs \\$compiler_flags -qmkshrobj -G -e \\$soname -o \\$lib"|g' You need this for many non-babel libraries too! Don't put -G in LDFLAGS – this causes binaries to break

Libtool script fixes to run regression tests on surveyor DON'T build or run from /scratch! Relies on script fixing (both libtool and python runs): contrib/regression_fix.sh – Replaces 'exec' with 'cobalt-mpirun' & env. Variables – Replaces '$PYTHON' with 'cobalt-mpirun' & env. Variables

Running regression tests Make sure PYTHONPATH points to numpy for BG/P in config line Submit tests to the queue: runC, runCxx, runF77, runF90: qsub -A cca-tools -t 15 -n 1 –mode script 'runAll.sh' RunPy2*: qsub -A cca-tools -t 15 -n 1 –mode script 'runPy2*'

Building ccaffeine on surveyor -rpath is not compatible with IBM compilers; always use -Wl,-rpath IBM compilers also do not appreciate –Wall setup.py needs to be handled differently: - Boyana has built a patched distutils Build needs to be staged: Complete build and then rebuild drivers, to link them to the right place Don't use babel-libtool! It causes improper linking

Final Notes, BG/P and CCA tools Just because it builds, that doesn't mean it runs! surveyor:/home/projects/ccatools/cca/ccafe/bin>./ccafe-single./ccafe-single: error while loading shared libraries:./.libs/libccafePreload_0_8_8.so: cannot open shared object file: No such file or directory Libtool and IBM compilers don't always play nice together when linking against uninstalled libraries Be wary of gcc compiler directives beginning with '-r' Know that static libraries installed for BG/P are not position-independent Never use babel-libtool unless it’s for Babel

Synergia Build f --Testing Build. Combining Stephen and Jim A. instructions for a “cut and paste” build -- The usual suspects: Boost and Chef -- Prototype implementation for adding CTEST and CDASH to Chef to implement full testing system

VORPAL Build VORPAL Builds and mostly runs Problem with solve that uses Trilinos Stencil solve slower than original solve…hmmm In the process of making equivalent Synergia2 and VORPAL simulations

Number Processors Time (Sec) VORPAL BG/P and AMD solve performance

Babel and Fortran Struct work continues (My focus for the month of February) -- F90 redo: - F90 Server introduces a Fortran “fSkel” file. - Structs are unrolled in C, passed as individual arguments then repacked in fSkel. F90 Impl stays the same. - If you want zero-copy structs use F2003 binding. -- F77 – Needs clients, servers done -- Combine struct BindC and struct regression tests -- Added nightly runs to 32 and 64 bit AMD and 64 bit Intel with GCC and PGI

BlueGene/P system architecture surveyor is a 1-rack (1024 CPU/4096 core) BG/P PPC 970MP frontend login nodes