Martin Schulz Center for Applied Scientific Computing Lawrence Livermore National Laboratory Lawrence Livermore National Laboratory, P. O. Box 808, Livermore, CA This work performed under the auspices of the U.S. Department of Energy by Lawrence Livermore National Laboratory under Contract DE-AC52-07NA27344 Introduction Scope, Tool Categories, Definitions LLNL-PRES

Lawrence Livermore National Laboratory Development Tools for HPC  Wide variety of tools exist Open source & commercial Cross platform & vendor specific Area of active research  The tool teams at all three labs support many tools Production support across all labs Experimental installs at individual sites/machines Research at the laboratories  Goals of this workshop Make users aware of what is available Provide basic usage instructions More in depth for selected few in the afternoon sessions Identify gaps / missing functionality

Lawrence Livermore National Laboratory Questions we will try to answer  Which tool can I use to... … debug my code? … find memory corruptions? … profile the performance of my code? … understand the communication behavior?  Where can I find the tools and how can I use them? Supported platforms Installation locations Basic user guides  Who can I contact for more help & information?

Lawrence Livermore National Laboratory Questions we would like answered  Which tools are you using? Active and regularly vs. occasionally vs. not at all? Do you have any problems with them?  Which concrete tools are you missing? Do you know of concrete tools that you would like? Are there tools on some platforms that you like, but are not available on platforms that you need?  What tool capabilities are you missing? What about your codes do you need to know?  Would you like extended training sessions for any tools? On-site tutorials? One-on-one sessions with tool developers?

Lawrence Livermore National Laboratory Categories of Program Development Tools  Debugging Tools What should I do if my program fails? What should I do if my program hangs?  Performance Analysis Tools Where is my code spending its time? Which communication operations take the most time? What is the message pattern of my code? Which call paths are most often taken?  Memory Analysis Tools Two separate categories  How can I detect silent memory corruptions?  How can I find out how much memory I am using?

Lawrence Livermore National Laboratory Debugging Options  Traditional debugging Interactive sessions  Ability to attach and control jobs  Set breakpoints, single stepping, inspect state BUT: inherent scalability problems  Should work up to 4096 processes  Likely to be infeasible after that  Debugging at scale Need a pre-selection of processes  Subset detection  Representatives of equivalence classes Application information or light-weight tools Subset attach features of traditional debuggers

Lawrence Livermore National Laboratory Performance Analysis Options  Tools to analyze, understand, and optimize performance Wide range of tools with varying level of detail Varying functionality: from display to automatic tuning  Typical workflow 1.Instrument code 2.Run and gather data 3.Analyze data (command line or GUI)  Instrumentation Options Transparent instrumentation (online or pre-loading) Binary or offline instrumentation Automatic source code instrumentation Manual source code instrumentation

Lawrence Livermore National Laboratory Profiling Techniques  Aggregate events over time Aggregate metrics (e.g., time spent in all MPI calls) Statistical sampling  When should I use it? Getting a first overview of performance Finding hot spots  Tradeoffs (+) Easy to use, low overhead, small result files (-) Little details, sometimes hard to correlate

Lawrence Livermore National Laboratory Tracing Techniques  Gather information about individual events Optional: combine with profile data Typically visualized as timeline graph  Examples Collect information on all MPI calls Find all calls to I/O routines  When should I use it? After profiling points to a particular segment Understand individual event interactions  Tradeoffs (+) Very detailed information, catch outliers (-) Higher overhead, potentially huge output files

Lawrence Livermore National Laboratory Memory Analysis Tools  Use case 1: Debugging Detect memory access problems  Stray writes / memory corruptions  Repeated free’s / stale pointers Options  Guard blocks, “Electric Fence”  Emulation / simulation of each memory access  Use case 2: Memory profiling How much memory am I using? Where does memory get allocated?

Lawrence Livermore National Laboratory List of Tools (in order)  Debuggers Totalview STAT  Performance Analysis TAU perflib Open|SpeedShop Vampir mpiP gprof CrayPAT CrayApprentice  Memory Analysis Totalview Valgrind memP  Other tools Scalasca Libra load balancer LOBA HPC Toolkit Paraver DDT Javelina ThreadSpotter

Lawrence Livermore National Laboratory List of Tools (by category)  Debuggers Full featured  Totalview, DDT Scalable pre-selection  STAT  Memory Analysis Corruption Detection  Valgrind (Emulation)  Totalview (Guard blocks) Memory usage  memP  Other categories Thread Spotter (threading) Javelina (code coverage)  Performance Analysis Profiling (transparent)  mpiP, gprof, HPCToolkit  Libra (load balance)  LOBA (MPI details) Tracing (code instr.)  Vampir, Paraver Profiling & Tracing  Open|SpeedShop, (transparent)  TAU (code or manual instrument.)  CrayPAT/Apprentice (code instr.)  HPC Toolkit (binary instrument.) Hybrid  Scalasca (Trace analysis) Manual Instrumentation  perflib