Simplifying the Usage of Performance Evaluation Tools: Experiences with TAU and DyninstAPI Paradyn/Condor Week 2010, Rm 221, Fluno Center, U. of Wisconsin, Madison, 10:45am – 11:30 am Tuesday, 14 th April, 2010 Sameer Shende, Allen D. Malony, Alan Morris Performance Research Laboratory University of Oregon, Eugene, OR {sameer, malony,

2 Acknowledgements: University of Oregon  Dr. Allen D. Malony, Professor, CIS Dept, and Director, NeuroInformatics Center  Alan Morris, Senior software engineer  Dr. Chee Wai Lee, Research faculty  Wyatt Spear, Software engineer  Scott Biersdorff, Software engineer  Dr. Robert Yelle, Research faculty  Suzanne Millstein, Ph.D. student And  Matt Legendre and Dan McNulty, University of Wisconsin at Madison

Motivation  We have made great advances in instrumentation, measurement and analysis techniques  Tools are rich in features and have a complex tool dependency  Tools are getting more complex to use and to install  We need to simplify the usage of our performance evaluation tools!

TAU Performance System ®  Integrated toolkit for performance problem solving  Instrumentation, measurement, analysis, visualization  Portable performance profiling and tracing facility  Performance data management and data mining  Based on direct performance measurement approach  Open source  Available on all HPC platforms  Partners  LLNL, ANL, ORNL, LANL, PNNL, LBL  Research Centre Jülich, TU Dresden TAU Architecture

TAU Parallel Performance System Goals  Portable (open source) parallel performance system  Computer system architectures and operating systems  Different programming languages and compilers  Multi-level, multi-language performance instrumentation  Flexible and configurable performance measurement  Support for multiple parallel programming paradigms  Multi-threading, message passing, mixed-mode, hybrid, object oriented (generic), component-based  Support for performance mapping  Integration of leading performance technology  Scalable (very large) parallel performance analysis

TAU Performance System Components TAU Architecture Program Analysis Parallel Profile Analysis PDT PerfDMF ParaProf Performance Data Mining Performance Monitoring TAUoverMRNet (ToM) PerfExplorer

TAU Performance System Architecture

TAU Performance System Architecture

Parallel Profile Visualization: ParaProf

Scalable Visualization: ParaProf (128k cores)

Scatter Plot: ParaProf (128k cores)

ParaProf: Communication Matrix Display

Comparing Effects of Multi-Core Processors AORSA2D  magnetized plasma simulation  Automatic loop level instrumentation  Blue is single node  Red is dual core  Cray XT3 (4K cores)

ParaProf: Mflops Sorted by Exclusive Time low mflops?

Performance Regression Testing

Usage Scenarios: Evaluate Scalability

Scaling NAMD with CUDA (Jumpshot with TAU) Data transfer

Measuring Performance of PGI Accelerated Code

TAU and Eclipse  Provide an interface for configuring TAU’s automatic instrumentation within Eclipse’s build system  Manage runtime configuration settings and environment variables for execution of TAU instrumented programs C/C++/Fortran Project in Eclipse Add or modify an Eclipse build configuration w/ TAU Temporary copy of instrumented code Compilation/linking with TAU libraries TAU instrumented libraries Program execution Performance data Program output

TAU and Eclipse PerfDMF

Choosing PAPI Counters with TAU in Eclipse

TAU Performance System Architecture

TAU Instrumentation Approach  Support for standard program events  Routines, classes and templates  Statement-level blocks  Begin/End events (Interval events)  Support for user-defined events  Begin/End events specified by user  Atomic events (e.g., size of memory allocated/freed)  Selection of event statistics  Support definition of “semantic” entities for mapping  Support for event groups (aggregation, selection)  Instrumentation optimization  Eliminate instrumentation in lightweight routines

TAU Instrumentation Mechanisms  Source code  Manual (TAU API, TAU component API)  Automatic (robust)  C, C++, F77/90/95 (Program Database Toolkit (PDT))  OpenMP (directive rewriting (Opari), POMP2 spec)  Object code  Compiler-based instrumentation (-optCompInst)  Pre-instrumented libraries (e.g., MPI using PMPI)  Statically-linked and dynamically-linked (tau_wrap)  Executable code  Binary re-writing and dynamic instrumentation (DyninstAPI, U. Wisconsin, U. Maryland)  Virtual machine instrumentation (e.g., Java using JVMPI)  Interpreter based instrumentation (Python)  Kernel based instrumentation (KTAU)

Program Database Toolkit (PDT) Application / Library C / C++ parser Fortran parser F77/90/95 C / C++ IL analyzer Fortran IL analyzer Program Database Files IL DUCTAPE PDBhtml SILOON CHASM TAU_instr Program documentation Application component glue C++ / F90/95 interoperability Automatic source instrumentation

Automatic Source-Level Instrumentation in TAU TAU v : If source based instrumentation fails, compiler-based instrumentation is used automatically

Using TAU with Source Code Instrumentation  TAU supports several measurement options (profiling, tracing, profiling with hardware counters, etc.)  Each measurement configuration of TAU corresponds to a unique stub makefile that is generated when you configure it  To instrument source code using PDT  Choose an appropriate TAU stub makefile in /lib: % export TAU_MAKEFILE=/usr/local/packages/tau/x86_64/lib/Makefile.tau-mpi-pdt % export TAU_OPTIONS=‘-optVerbose …’ (see tau_compiler.sh -help) And use tau_f90.sh, tau_cxx.sh or tau_cc.sh as Fortran, C++ or C compilers: % mpif90 foo.f90 changes to % tau_f90.sh foo.f90  Execute application and analyze performance data: % pprof (for text based profile display) % paraprof (for GUI)

TAU Measurement Configuration – Examples % cd /usr/local/packages/tau/x86_64/lib; ls Makefile.* Makefile.tau-pdt Makefile.tau-mpi-pdt Makefile.tau-papi-mpi-pdt Makefile.tau-pthread-pdt Makefile.tau-pthread-mpi-pdt Makefile.tau-openmp-opari-pdt Makefile.tau-openmp-opari-mpi-pdt Makefile.tau-papi-openmp-opari-mpi-pdt …  For an MPI+F90 application, you may want to start with: Makefile.tau-mpi-pdt  Supports MPI instrumentation & PDT for automatic source instrumentation  % setenv TAU_MAKEFILE /usr/local/packages/tau/x86_64/lib/Makefile.tau-mpi-pdt  % tau_f90.sh application.f90; mpirun –np 256./a.out

Compile-Time Environment Variables  Optional parameters for TAU_OPTIONS: [tau_compiler.sh –help] -optVerboseTurn on verbose debugging messages -optCompInstUse compiler based instrumentation -optNoCompInstDo not revert to compiler instrumentation if source instrumentation fails. -optDetectMemoryLeaks Turn on debugging memory allocations/ de-allocations to track leaks -optKeepFiles Does not remove intermediate.pdb and.inst.* files -optPreProcess Preprocess Fortran sources before instrumentation -optTauSelectFile="" Specify selective instrumentation file for tau_instrumentor -optLinking="" Options passed to the linker. Typically $(TAU_MPI_FLIBS) $(TAU_LIBS) $(TAU_CXXLIBS) -optCompile="" Options passed to the compiler. Typically $(TAU_MPI_INCLUDE) $(TAU_INCLUDE) $(TAU_DEFS) -optPdtF95Opts="" Add options for Fortran parser in PDT (f95parse/gfparse) -optPdtF95Reset="" Reset options for Fortran parser in PDT (f95parse/gfparse) -optPdtCOpts="" Options for C parser in PDT (cparse). Typically $(TAU_MPI_INCLUDE) $(TAU_INCLUDE) $(TAU_DEFS) -optPdtCxxOpts="" Options for C++ parser in PDT (cxxparse). Typically $(TAU_MPI_INCLUDE) $(TAU_INCLUDE) $(TAU_DEFS)...

Runtime Environment Variables in TAU Environment VariableDefaultDescription TAU_TRACE0Setting to 1 turns on tracing TAU_CALLPATH0Setting to 1 turns on callpath profiling TAU_TRACK_HEAP or TAU_TRACK_HEADROOM 0Setting to 1 turns on tracking heap memory/headroom at routine entry & exit using context events (e.g., Heap at Entry: main=>foo=>bar) TAU_CALLPATH_DEPTH2Specifies depth of callpath. Setting to 0 generates no callpath or routine information, setting to 1 generates flat profile and context events have just parent information (e.g., Heap Entry: foo) TAU_SYNCHRONIZE_CLOCKS1Synchronize clocks across nodes to correct timestamps in traces TAU_COMM_MATRIX0Setting to 1 generates communication matrix display using context events TAU_THROTTLE1Setting to 0 turns off throttling. Enabled by default to remove instrumentation in lightweight routines that are called frequently TAU_THROTTLE_NUMCALLS100000Specifies the number of calls before testing for throttling TAU_THROTTLE_PERCALL10Specifies value in microseconds. Throttle a routine if it is called over times and takes less than 10 usec of inclusive time per call TAU_COMPENSATE0Setting to 1 enables runtime compensation of instrumentation overhead TAU_PROFILE_FORMATProfileSetting to “merged” generates a single file. “snapshot” generates xml format TAU_METRICSTIMESetting to a comma separted list generates other metrics. (e.g., TIME:linuxtimers:PAPI_FP_OPS:PAPI_NATIVE_ )

Simplifying Instrumentation using DyninstAPI  TAU uses DyninstAPI to create a binary re-writer (tau_run)  TAU’s measurement library (DSO) is loaded by tau_run  Both runtime instrumentation and binary re-writing are supported  Selection of files and routines based on exclude/include lists  Simplifies tool usage greatly!  Available on POINT LiveDVD [  Usage:  % tau_run a.out –o a.inst.out  % mpirun –np 4 a.inst.out  % paraprof

Issues  Re-writing static executables limited to gcc, limited platforms in beta  Currently, we support dynamic executables (v6.1)  We are working on supporting both static and dynamic executables  We hope to support more platforms, compilers and runtime systems in the future  Rewriting shared libraries used by the application  LD_PRELOAD’able wrapper libraries can be created using tau_wrap  requires interface information in header file

Binary Rewriting in TAU using DyninstAPI

Wish List for tau_run  Support for more platforms  Apple Mac OS X, Windows, IBM BG/P, AIX, …  Support for more compilers  Support for rewriting shared objects  Support for static binary rewriting with validation for compilers other than gcc  XLC, PathScale, Cray CCE, Intel, PGI,…

Other Tools…  Other TAU tools that use technologies from the ParaDyn/DyninstAPI group  TAU over MRNet (ToM) for runtime  Stackwalker API for accessing callstack

StackWalkerAPI in TAU  Requirements overview:  Minimal information required (PC is enough)  Threaded support necessary  Low overhead (for high sample rates)  Stack unwinding from a signal handler  Malloc could be interrupted  Need to walk through signal handler frame

Issues encountered with StackWalkerAPI  StackWalkerAPI:  Isn’t thread safe (and locking to use it can cause significant overhead)  Uses malloc/new (and so do dependent libraries such as libdwarf)  C++ (we would prefer C)  Issues walking certain kinds of stack frames  Matt Legendre was able to help us out a lot though!  Alternatives:  TAU is currently using stack walking constructs from HPCToolkit

Online Monitoring using TAU over MRNet (ToM)  Back-End (BE) TAU adapter offloads performance data  Filters  reduction  distributed analysis  upstream / downstream  Front-End (FE) unpacks, interprets, stores  Paths  reverse data reduction path  multicast control path  Push-Pull model  source pushes, sink pulls

Conclusions  TAU and DyninstAPI represents mature technology for performance instrumentation, measurement and analysis  Using DyninstAPI’s binary re-writing capabilities, we have produced a tool that simplifies code instrumentation  We hope to collaborate on other projects and include support for an enhanced stack walker API Questions?

Support Acknowledgements  Department of Energy (DOE)  Office of Science  MICS, Argonne National Lab  ASC/NNSA  University of Utah ASC/NNSA Level 1  ASC/NNSA, LLNL  Department of Defense (DoD)  NSF SDCI  Partners:  Research Centre Juelich  LBL, ORNL, ANL, LANL, PNNL, LLNL  TU Dresden  ParaTools, Inc.