1. Tuning and Analysis Utilities Sameer Shende University of Oregon

2. General Problems How do we create robust and ubiquitous performance technology for the analysis and tuning of parallel and distributed software and systems in the presence of (evolving) complexity challenges? How do we apply performance technology effectively for the variety and diversity of performance problems that arise in the context of complex parallel and distributed computer systems.

3. Computation Model for Performance Technology  How to address dual performance technology goals?  Robust capabilities + widely available methodologies  Contend with problems of system diversity  Flexible tool composition/configuration/integration  Approaches  Restrict computation types / performance problems  limited performance technology coverage  Base technology on abstract computation model  general architecture and software execution features  map features/methods to existing complex system types  develop capabilities that can adapt and be optimized

4. General Complex System Computation Model  Node: physically distinct shared memory machine  Message passing node interconnection network  Context: distinct virtual memory space within node  Thread: execution threads (user/system) in context memory Node VM space Context SMP Threads node memory … … Interconnection Network Inter-node message communication * * physical view model view

5. Definitions – Profiling  Profiling  Recording of summary information during execution  execution time, # calls, hardware statistics, …  Reflects performance behavior of program entities  functions, loops, basic blocks  user-defined “semantic” entities  Very good for low-cost performance assessment  Helps to expose performance bottlenecks and hotspots  Implemented through  sampling: periodic OS interrupts or hardware counter traps  instrumentation: direct insertion of measurement code

6. Definitions – Tracing  Tracing  Recording of information about significant points (events) during program execution  entering/exiting code region (function, loop, block, …)  thread/process interactions (e.g., send/receive message)  Save information in event record  timestamp  CPU identifier, thread identifier  Event type and event-specific information  Event trace is a time-sequenced stream of event records  Can be used to reconstruct dynamic program behavior  Typically requires code instrumentation

7. Definitions – Instrumentation  Instrumentation  Insertion of extra code (hooks) into program  Source instrumentation  Done by compiler, source-to-source translator, or manually  Object code instrumentation  “re-writing” the executable to insert hooks  Dynamic code instrumentation  a debugger-like instrumentation approach  executable code instrumentation on running program  DynInst and DPCL are examples  Pre-instrumented library  supported by link-time library interposition

8. Event Tracing: Instrumentation, Monitor, Trace 1master 2slave 3... void slave { trace(ENTER, 2);... recv(A, tag, buf); trace(RECV, A);... trace(EXIT, 2); } void master { trace(ENTER, 1);... trace(SEND, B); send(B, tag, buf);... trace(EXIT, 1); } MONITOR 58AENTER1 60BENTER2 62ASENDB 64AEXIT1 68BRECVA... 69BEXIT2... CPU A: CPU B: Event definition timestamp

9. Event Tracing: “Timeline” Visualization 1master 2slave 3... 58AENTER1 60BENTER2 62ASENDB 64AEXIT1 68BRECVA... 69BEXIT2... main master slave 58606264666870 B A

10. TAU Performance System Framework  Tuning and Analysis Utilities  Performance system framework for scalable parallel and distributed high-performance computing  Targets a general complex system computation model  nodes / contexts / threads  Multi-level: system / software / parallelism  Measurement and analysis abstraction  Integrated toolkit for performance instrumentation, measurement, analysis, and visualization  Portable performance profiling/tracing facility  Open software approach

11. TAU Performance System Architecture

12. Levels of Code Transformation  As program information flows through stages of compilation/linking/execution, different information is accessible at different stages  Each level poses different constraints and opportunities for extracting information  At what level should performance instrumentation be done?

13. Source-Level Instrumentation  Manually annotate source code + portable + links back to program code + fine grained instrumentation – re-compile is necessary for (change in) instrumentation – requires source to be available – hard to use in standard way for mix-language programs – source-to-source translators hard to develop for C++, F90

14. Preprocessor-Level Instrumentation  Parse the source code, insert instrumentation, rewrite the source code (source-to-source translator) + automates the process of instrumentation - need parsers for new languages - difficult to parse C++, F90 - limits of static analysis  Examples: Sage++, TAU/PDT, TAU/Opari Source codePreprocessorSource code

15. Issues with Optimizations  The presence of fine-grained instrumentation may interfere with optimizations:  instrumentation may inhibit optimizations  compiler may not preserve the semantics of instrumentation and measure something other than what may be expected  Instrumentation-Aware Compilation [Shende PhD’01]

16. Compiler-Based Instrumentation  Instrumentation inserted in the object code during compilation + automates the process of inserting instrumentation + fine-grained instrumentation + knowledge of optimizations - may not see all routines in the executable - compiler specific

17. Library-Level Instrumentation  Pre-instrumented libraries  Wrapper interposition libraries int MPI_Recv(…)... { int returnVal, size; TAU_PROFILE_TIMER(tautimer, "MPI_Recv()", " ", TAU_MESSAGE); TAU_PROFILE_START(tautimer); returnVal = PMPI_Recv(buf, count, datatype, src, tag, comm, status); if (src != MPI_PROC_NULL && returnVal == MPI_SUCCESS) { PMPI_Get_count( status, MPI_BYTE, &size ); TAU_TRACE_RECVMSG(status->MPI_TAG, status->MPI_SOURCE, size); } TAU_PROFILE_STOP(tautimer); return returnVal; }

18. Executable-Level Instrumentation  Binary rewrite (e.g., pixie)  Dynamic instrumentation (e.g., DyninstAPI, debuggers)  Mutator inserts instrumentation snippets in the address space of mutatee  Replace an instruction with a branch instruction  Code snippets can load dynamic shared objects, call routines  Instrumentation can be inserted and removed at runtime.

19. Virtual-Machine Level Instrumentation  Integrate performance system with VM  Captures robust performance data (e.g., thread events)  Maintain features of environment  portability, concurrency, extensibility, interoperation  Allow use in optimization methods  JVM Profiling Interface (JVMPI)  Generation of JVM events and hooks into JVM  Profiler agent loaded as shared object  registers events of interest and address of callback routine  Access to information on dynamically loaded classes  No need to modify Java source, bytecode, or JVM

20. TAU Instrumentation  Flexible instrumentation mechanisms at multiple levels  Source code  manual  automatic using Program Database Toolkit (PDT), OPARI  Object code  pre-instrumented libraries (e.g., MPI using PMPI)  statically linked  dynamically linked (e.g., Virtual machine instrumentation)  fast breakpoints (compiler generated)  Executable code  dynamic instrumentation (pre-execution) using DynInstAPI

21. TAU Instrumentation (continued)  Targets common measurement interface (TAU API)  Object-based design and implementation  Macro-based, using constructor/destructor techniques  Program units: function, classes, templates, blocks  Uniquely identify functions and templates  name and type signature (name registration)  static object creates performance entry  dynamic object receives static object pointer  runtime type identification for template instantiations  C and Fortran instrumentation variants  Instrumentation and measurement optimization

22. Multi-Level Instrumentation  Uses multiple instrumentation interfaces  Shares information: cooperation between interfaces  Taps information at multiple levels  Provides selective instrumentation at each level  Targets a common performance model  Presents a unified view of execution

23. Program Database Toolkit (PDT)  Program code analysis framework for developing source- based tools  High-level interface to source code information  Integrated toolkit for source code parsing, database creation, and database query  commercial grade front end parsers  portable IL analyzer, database format, and access API  open software approach for tool development  Target and integrate multiple source languages  Use in TAU to build automated performance instrumentation tools

24. PDT Architecture and Tools C/C++ Fortran 77/90

25. PDT Components  Language front end  Edison Design Group (EDG): C, C++, Java  Mutek Solutions Ltd.: F77, F90  creates an intermediate-language (IL) tree  IL Analyzer  processes the intermediate language (IL) tree  creates “program database” (PDB) formatted file  DUCTAPE (Bernd Mohr, ZAM, Germany)  C++ program Database Utilities and Conversion Tools APplication Environment  processes and merges PDB files  C++ library to access the PDB for PDT applications

26. TAU Measurement  Performance information  High-resolution timer library (real-time / virtual clocks)  General software counter library (user-defined events)  Hardware performance counters  PCL (Performance Counter Library) (ZAM, Germany)  PAPI (Performance API) (UTK, Ptools Consortium)  consistent, portable API  Organization  Node, context, thread levels  Profile groups for collective events (runtime selective)  Performance data mapping between software levels

27. TAU Measurement (continued)  Parallel profiling  Function-level, block-level, statement-level  Supports user-defined events  TAU parallel profile database  Function callstack  Hardware counts values (in replace of time)  Tracing  All profile-level events  Interprocess communication events  Timestamp synchronization  User-configurable measurement library (user controlled)

28. TAU Measurement System Configuration  configure [OPTIONS]  {-c++=, -cc= } Specify C++ and C compilers  {-pthread, -sproc}Use pthread or SGI sproc threads  -openmpUse OpenMP threads  -jdk= Specify location of Java Dev. Kit  -opari= Specify location of Opari OpenMP tool  {-pcl, -papi}= Specify location of PCL or PAPI  -pdt= Specify location of PDT  -dyninst= Specify location of DynInst Package  {-mpiinc=, mpilib= }Specify MPI library instrumentation  - TRACE Generate TAU event traces  -PROFILE Generate TAU profiles  -CPUTIMEUse usertime+system time  -PAPIWALLCLOCKUse PAPI to access wallclock time  -PAPIVIRTUALUse PAPI for virtual (user) time

29. TAU Measurement Configuration – Examples ./configure -c++=KCC –SGITIMERS  Use TAU with KCC and fast nanosecond timers on SGI  Enable TAU profiling (default) ./configure -TRACE –PROFILE  Enable both TAU profiling and tracing ./configure -c++=guidec++ -cc=guidec -papi=/usr/local/packages/papi –openmp -mpiinc=/usr/packages/mpich/include -mpilib=/usr/packages/mpich/lib  Use OpenMP+MPI using KAI's Guide compiler suite and use PAPI for accessing hardware performance counters for measurements  Typically configure multiple measurement libraries

30. TAU Measurement API  Initialization and runtime configuration  TAU_PROFILE_INIT(argc, argv); TAU_PROFILE_SET_NODE(myNode); TAU_PROFILE_SET_CONTEXT(myContext); TAU_PROFILE_EXIT(message); TAU_REGISTER_THREAD();  Function and class methods  TAU_PROFILE(name, type, group);  Template  TAU_TYPE_STRING(variable, type); TAU_PROFILE(name, type, group); CT(variable);  User-defined timing  TAU_PROFILE_TIMER(timer, name, type, group); TAU_PROFILE_START(timer); TAU_PROFILE_STOP(timer);

31. TAU Measurement API (continued)  User-defined events  TAU_REGISTER_EVENT(variable, event_name); TAU_EVENT(variable, value); TAU_PROFILE_STMT(statement);  Mapping  TAU_MAPPING(statement, key); TAU_MAPPING_OBJECT(funcIdVar); TAU_MAPPING_LINK(funcIdVar, key);  TAU_MAPPING_PROFILE (funcIdVar); TAU_MAPPING_PROFILE_TIMER(timer, funcIdVar); TAU_MAPPING_PROFILE_START(timer); TAU_MAPPING_PROFILE_STOP(timer);  Reporting  TAU_REPORT_STATISTICS(); TAU_REPORT_THREAD_STATISTICS();

32. Compiling: TAU Makefiles  Include TAU Makefile in the user’s Makefile.  Variables:  TAU_CXXSpecify the C++ compiler  TAU_CCSpecify the C compiler used by TAU  TAU_DEFSDefines used by TAU. Add to CFLAGS  TAU_LDFLAGSLinker options. Add to LDFLAGS  TAU_INCLUDEHeader files include path. Add to CFLAGS  TAU_LIBSStatically linked TAU library. Add to LIBS  TAU_SHLIBSDynamically linked TAU library  TAU_MPI_LIBSTAU’s MPI wrapper library for C/C++  TAU_MPI_FLIBSTAU’s MPI wrapper library for F90  TAU_FORTRANLIBSMust be linked in with C++ linker for F90.  Note: Not including TAU_DEFS in CFLAGS disables instrumentation in C/C++ programs.

33. Including TAU Makefile - Example include /usr/tau/sgi64/lib/Makefile.tau-pthread-kcc CXX = $(TAU_CXX) CC = $(TAU_CC) CFLAGS = $(TAU_DEFS) LIBS = $(TAU_LIBS) OBJS =... TARGET= a.out TARGET: $(OBJS) $(CXX) $(LDFLAGS) $(OBJS) -o $@ $(LIBS).cpp.o: $(CC) $(CFLAGS) -c $< -o $@

34. TAU Makefile for PDT include /usr/tau/include/Makefile CXX = $(TAU_CXX) CC = $(TAU_CC) PDTPARSE = $(PDTDIR)/$(CONFIG_ARCH)/bin/cxxparse TAUINSTR = $(TAUROOT)/$(CONFIG_ARCH)/bin/tau_instrumentor CFLAGS = $(TAU_DEFS) LIBS = $(TAU_LIBS) OBJS =... TARGET= a.out TARGET: $(OBJS) $(CXX) $(LDFLAGS) $(OBJS) -o $@ $(LIBS).cpp.o: $(PDTPARSE) $< $(TAUINSTR) $*.pdb $< -o $*.inst.cpp $(CC) $(CFLAGS) -c $*.inst.cpp -o $@

35. Setup: Running Applications % setenv PROFILEDIR /home/data/experiments/profile/01 % setenv TRACEDIR/home/data/experiments/trace/01 % set path=($path / /bin) % setenv LD_LIBRARY_PATH $LD_LIBRARY_PATH\: / /lib For PAPI/PCL: % setenv PAPI_EVENT PAPI_FP_INS % setenv PCL_EVENT PCL_FP_INSTR For Java (without instrumentation): % java application With instrumentation: % java -XrunTAU application % java -XrunTAU:exclude=sun/io,java application For DyninstAPI: % a.out % tau_run a.out % tau_run -XrunTAUsh-papi a.out

36. TAU Analysis  Profile analysis  Pprof  parallel profiler with text-based display  Racy  graphical interface to pprof (Tcl/Tk)  jRacy  Java implementation of Racy  Trace analysis and visualization  Trace merging and clock adjustment (if necessary)  Trace format conversion (ALOG, SDDF, Vampir)  Vampir (Pallas) trace visualization

37. Pprof Command  pprof [-c|-b|-m|-t|-e|-i] [-r] [-s] [-n num] [-f file] [-l] [nodes]  -cSort according to number of calls  -bSort according to number of subroutines called  -mSort according to msecs (exclusive time total)  -tSort according to total msecs (inclusive time total)  -eSort according to exclusive time per call  -iSort according to inclusive time per call  -vSort according to standard deviation (exclusive usec)  -rReverse sorting order  -sPrint only summary profile information  -n numPrint only first number of functions  -f fileSpecify full path and filename without node ids  -l nodesList all functions and exit (prints only info about all contexts/threads of given node numbers)

38. Pprof Output (NAS Parallel Benchmark – LU)  Intel Quad PIII Xeon, RedHat, PGI F90  F90 + MPICH  Profile for: Node Context Thread  Application events and MPI events

39. jRacy (NAS Parallel Benchmark – LU) n: node c: context t: thread Global profiles Individual profile Routine profile across all nodes

40. TAU and PAPI (NAS Parallel Benchmark – LU )  Floating point operations  Replaces execution time  Only requires relinking to different measurement library

41. Vampir Trace Visualization Tool  Visualization and Analysis of MPI Programs  Originally developed by Forschungszentrum Jülich  Current development by Technical University Dresden  Distributed by PALLAS, Germany  http://www.pallas.de/pages/vampir.htm

42. Vampir (NAS Parallel Benchmark – LU) Timeline display Callgraph display Communications display Parallelism display

43. Complexity Scenarios  Multi-threaded performance measurements  Mixed-mode parallel programs:  Java+MPI  OpenMP+MPI

44. Multi-Threading Performance Measurement  General issues  Thread identity and per-thread data storage  Performance measurement support and synchronization  Fine-grained parallelism  different forms and levels of threading  greater need for efficient instrumentation  TAU general threading and measurement model  Common thread layer and measurement support  Interface to system specific libraries (reg, id, sync)  Target different thread systems with core functionality  Pthreads, Windows, Java, SMARTS, Tulip, OpenMP

45. Java Multi-Threading Performance (Test Case)  Profile and trace Java (JDK 1.2+) applications  Observe user-level and system-level threads  Observe events for different Java packages  /lang, /io, /awt, …  Test application  SciVis, NPAC, Syracuse University %./configure -jdk= % setenv LD_LIBRARY_PATH $LD_LIBRARY_PATH\: / /lib % java -XrunTAU svserver

46. TAU Profiling of Java Application (SciVis) 24 threads of execution! Profile for each Java thread Captures events for different Java packages Global routine profile

47. TAU Tracing of Java Application (SciVis) Timeline display Parallelism view Performance groups

48. Vampir Dynamic Call Tree View (SciVis) Per thread call tree Expanded call tree Annotated performarnce

49. Virtual Machine Performance Instrumentation  Integrate performance system with VM  Captures robust performance data (e.g., thread events)  Maintain features of environment  portability, concurrency, extensibility, interoperation  Allow use in optimization methods  JVM Profiling Interface (JVMPI)  Generation of JVM events and hooks into JVM  Profiler agent (TAU) loaded as shared object  registers events of interest and address of callback routine  Access to information on dynamically loaded classes  No need to modify Java source, bytecode, or JVM

50. JVMPI Events  Method transition events  Memory events  Heap arena events  Garbage collection events  Class events  Global reference events  Monitor events  Monitor wait events  Thread events  Dump events  Virtual machine events

51. TAU Java JVM Instrumentation Architecture Java program Profile DB JNI TAU JVMPI Thread API Event notification  Robust set of events  Portability  Access to thread info  Measurement options  Limitations  Overhead  Many events  Event control  No user-defined events

52. TAU Java Source Instrumentation Architecture  Any code section can be measured  Portability  Measurement options  Profiling, tracing  Limitations  Source access only  Lack of thread information  Lack of node information Java program TAU.Profile class (init, data, output) TAU package Profile database stored in JVM heap TAU as dynamic shared object JNI C bindings Profile DB JNI TAU

53. Java Source-Level Instrumentation  TAU Java package  User-defined events  TAU.Profile class for new “timers”  Start/Stop  Performance data output at end

54. Mixed-mode Parallel Programs (Java + MPI)  Explicit message communication libraries for Java  MPI performance measurement  MPI profiling interface - link-time interposition library  TAU wrappers in native profiling interface library  Send/Receive events and communication statistics  mpiJava (Syracuse, JavaGrande, 1999)  Java wrapper package  JNI C bindings to MPI communication library  Dynamic shared object (libmpijava.so) loaded in JVM  prunjava calls mpirun to distribute program to nodes  Contrast to Java RMI-based schemes (MPJ, CCJ)

55. TAU mpiJava Instrumentation Architecture  No source instrumentation required  Portability  Measurement options  Limitations  MPI events only  No mpiJava events  Node info only  No thread info mpiJava package Native MPI library Java program JNI Profile DB TAU TAU package TAU wrapper Native MPI library MPI profiling interface

56. Java Multi-threading and Message Passing  Java threads and MPI communications  Shared-memory multi-threading events  Message communications events  Unified performance measurement and views  Integration of performance mechanisms  Integrated association of performance events  thread event and communication events  user-defined (source-level) performance events  JVM events  Requires instrumentation and measurement cooperation

57. Instrumentation and Measurement Cooperation  Problem  JVMPI doesn’t see MPI events (e.g., rank (node))  MPI profiling interfaces doesn’t see threads  Source instrumentation doesn’t see either!  Need cooperation between interfaces  MPI exposes rank and gets thread information  JVMPI exposes thread information and gets rank  Source instrumentation gets both  Post-mortem matching of sends and receives  Selective instrumentation  java -XrunTAU:exclude=java/io,sun

58. JVMPI Thread API Event notification TAU Java Instrumentation Architecture Java program TAU package mpiJava package MPI profiling interface TAU wrapper Native MPI library Profile DB JNI TAU

59. Parallel Java Game of Life (Profile)  mpiJava testcase  4 nodes, 28 threads Node 0Node 1Node 2 Only thread 4 executes MPI_Init Merged Java and MPI event profiles

60. Parallel Java Game of Life (Trace)  Integrated event tracing  Merged trace viz  Node process grouping  Thread message pairing  Vampir display  Multi-level event grouping

61. Integrated Performance View (Callgraph)  Source level  MPI level  Java packages level

62. Mixed-mode Parallel Programs (OpenMPI + MPI)  Portable mixed-mode parallel programming  Multi-threaded shared memory programming  Inter-node message passing  Performance measurement  Access to runtime system and communication events  Associate communication and application events  2-Dimensional Stommel model of ocean circulation  OpenMP for shared memory parallel programming  MPI for cross-box message-based parallelism  Jacobi iteration, 5-point stencil  Timothy Kaiser (San Diego Supercomputing Center)

63. OpenMP Instrumentation  POMP [Mohr EWOMP’01]  OpenMP Directive Instrumentation  OpenMP Runtime Library Routine Instrumentation  Performance Monitoring Library Control  User Code Instrumentation  Context Descriptors  Conditional Compilation  Conditional / Selective Transformations  Implemented in OPARI tool (FZJ, Germany)

64. OPARI: !$OMP PARALLEL DO !$OMP PARALLEL DO clauses... do loop !$OMP END PARALLEL DO !$OMP PARALLEL other-clauses... !$OMP DO schedule-clauses, ordered-clauses, lastprivate-clauses do loop !$OMP END DO !$OMP END PARALLEL DO NOWAIT !$OMP BARRIER call pomp_parallel_fork(d) call pomp_parallel_begin(d) call pomp_parallel_end(d) call pomp_parallel_join(d) call pomp_do_enter(d) call pomp_do_exit(d) call pomp_barrier_enter(d) call pomp_barrier_exit(d)

65. Stommel Instrumentation  OpenMP directive instrumentation pomp_for_enter(&omp_rd_2); #line 252 "stommel.c" #pragma omp for schedule(static) reduction(+: diff) private(j) firstprivate (a1,a2,a3,a4,a5) nowait for( i=i1;i<=i2;i++) { for(j=j1;j<=j2;j++){ new_psi[i][j]=a1*psi[i+1][j] + a2*psi[i-1][j] + a3*psi[i][j+1] + a4*psi[i][j-1] - a5*the_for[i][j]; diff=diff+fabs(new_psi[i][j]-psi[i][j]); } pomp_barrier_enter(&omp_rd_2); #pragma omp barrier pomp_barrier_exit(&omp_rd_2); pomp_for_exit(&omp_rd_2); #line 261 "stommel.c"

66. OpenMP + MPI Ocean Modeling (Trace) Thread message passing Integrated OpenMP + MPI events

67. OpenMP + MPI Ocean Modeling (HW Profile) % configure -papi=../packages/papi -openmp -c++=pgCC -cc=pgcc -mpiinc=../packages/mpich/include -mpilib=../packages/mpich/lib FP instructions Integrated OpenMP + MPI events

68. TAU Performance System Status  Computing platforms  IBM SP, SGI Origin 2K/3K, Intel Teraflop, Cray T3E, Compaq SC, HP, Sun, Windows, IA-32, IA-64, Linux, …  Programming languages  C, C++, Fortran 77/90, HPF, Java, OpenMP  Communication libraries  MPI, PVM, Nexus, Tulip, ACLMPL, MPIJava  Thread libraries  pthreads, Java,Windows, Tulip, SMARTS, OpenMP  Compilers  KAI, PGI, GNU, Fujitsu, Sun, Microsoft, SGI, Cray, IBM, Compaq

69. TAU Performance System Status (continued)  Application libraries  Blitz++, A++/P++, ACLVIS, PAWS, SAMRAI, Overture  Application frameworks  POOMA, POOMA-2, MC++, Conejo, Uintah, UPS  Projects  Aurora / SCALEA: ACPC, University of Vienna  TAU full distribution (Version 2.10, web download)  Measurement library and profile analysis tools  Automatic software installation  Performance analysis examples  Extensive TAU User’s Guide

70. PDT Status  Program Database Toolkit (Version 2.0, web download)  EDG C++ front end (Version 2.45.2)  Mutek Fortran 90 front end (Version 2.4.1)  C++ and Fortran 90 IL Analyzer  DUCTAPE library  Standard C++ system header files (KCC Version 4.0f)  PDT-constructed tools  Automatic TAU performance instrumentation  C, C++, Fortran 77, and Fortran 90  Program analysis support for SILOON and CHASM

71. Evolution of the TAU Performance System  Future parallel computing environments need to be more adaptive to achieve and sustain high performance levels  TAU’s existing strength lies in its robust support for performance instrumentation and measurement  TAU will evolve to support new performance capabilities  Online performance data access via application-level API  Dynamic performance measurement control  Generalize performance mapping  Runtime performance analysis and visualization

72. Information  TAU (http://www.acl.lanl.gov/tau)  PDT (http://www.acl.lanl.gov/pdtoolkit)  Tutorial at SC’01: M11 B. Mohr, A. Malony, S. Shende, “Performance Technology for Complex Parallel Systems” Nov. 7, 2001, Denver, CO.  LANL, NIC Booth, SC’01.

73. Support Acknowledgement  TAU and PDT support:  Department of Engergy (DOE)  DOE 2000 ACTS contract  DOE MICS contract  DOE ASCI Level 3 (LANL, LLNL)  DARPA  NSF National Young Investigator (NYI) award

74. Hands-on session  On mcurie.nersc.gov, copy files from /usr/local/pkg/acts/tau/tau2/tau-2.9/training  See README file  Set correct path e.g., % set path=($path /usr/local/pkg/acts/tau/tau2/tau2.9/t3e/bin)  Examine the Makefile.  Type “make” in each directory; then execute the program  Type “racy” or “vampir”  Type a project name e.g., “matrix.pmf” and click OK to see the performance data.

75. Examples The training directory contains example programs that illustrate the use of TAU instrumentation and measuremen options. instrument -This contains a simple C++ example that shows how TAU's API can be used for manually instrumenting a C++ program. It highlights instrumentation for templates and user defined events. threads - A simple multi-threaded program that shows how the main function of a thread is instrumented. Performance data is generated for each thread of execution. Configure with -pthread. cthreads - Same as threads above, but for a C program. An instrumented C program may be compiled with a C compiler, but needs to be linked with a C++ linker. Configure with -pthread. pi - An MPI program that calculates the value of pi and e. It highlights the use of TAU's MPI wrapper library. TAU needs to be configured with -mpiinc= and -mpilib=. Run using mpirun -np cpi. papi - A matrix multiply example that shows how to use TAU statement level timers for comparing the performance of two algorithms for matrix multiplication. When used with PAPI or PCL, this can highlight the cache behaviors of these algorithms. TAU should be configured with -papi= or -pcl= and the user should set PAPI_EVENT or PCL_EVENT respective environment variables, to use this.

76. Examples - (cont.) papithreads - Same as papi, but uses threads to highlight how hardware performance counters may be used in a multi-threaded application. When it is used with PAPI, TAU should be configured with -papi= -pthread autoinstrument - Shows the use of Program Database Toolkit (PDT) for automating the insertion of TAU macros in the source code. It requires configuring TAU with the -pdt= option. The Makefile is modified to illustrate the use of a source to source translator (tau_instrumentor). NPB2.3 - The NAS Parallel Benchmark 2.3 [from NASA Ames]. It shows how to use TAU's MPI wrapper with a manually instrumented Fortran program. LU and SP are the two benchmarks. LU is instrumented completely, while only parts of the SP program are instrumented to contrast the coverage of routines. In both cases MPI level instrumentation is complete. TAU needs to be configured with -mpiinc= and -mpilib= to use this.