Portable Parallel Performance Tools Shirley Browne, UTK Clay Breshears, CEWES MSRC Jan 27-28, 1998

Overview Tool-assisted performance analysis and tuning Post-mortem analysis of trace files –AIMS –MPE logging and nupshot –Pablo –VAMPIR Run-time analysis using dynamic instrumentation –Paradyn

Schedule - Tuesday, Jan. 27 9:00-10:00 –Trace file generation, MPI profiling interface, trace file formats, analysis techniques 10:00-12:00 –Overview of AIMS, nupshot, VAMPIR, SvPablo 12:00-1:30 –Lunch 1:30-5:00 –Introductory hands-on exercises using trace file generation and analysis tools

Schedule - Wed., Jan. 28 9:00-10:15 –Pablo TraceLibrary and Analysis GUI 10:30-11:30 –Dynamic instrumentation and Paradyn 11:30-1:00 –Lunch 1:00-2:00 –Hands-on exercise using Paradyn 2:00-5:00 –Bring Your Own Code hands-on exercise

Tool-assisted Performance Analysis Develop performance model Instrument source code (optional or automatic) Link with instrumentation library Generate tracefile during program execution Analyze trace file Compare model predictions with measurements Tune code and/or revise model

Source Code Instrumentation MPI profiling interface (MPE logging, Pablo MPI trace library, VAMPIRtrace) Automatic parsing and insertion of calls to tracing routines (AIMS, SvPablo) By hand insertion of calls to tracing routines (MPE logging, Pablo basic and I/O trace library, VAMPIRtrace)

MPI Profiling Interface Conforming implementation of MPI must –Provide a mechanism through which all the MPI functions may be accessed with a name shift (All MPI functions which normally start with MPI_ should also be accessible with the prefix PMPI_) –Provide a no-op routine MPI_PCONTROL

MPI Profiling Interface (cont) Goal - enable authors of profiling (and other) tools to interface to MPI implementations without access to the underlying implementation Enables interception of all MPI calls made by the user program

Example Profiling Routine static int totalBytes; static double totalTime; int MPI_Send(void *buffer, const int count, MPI_Datatype datatype, int dest, int tag, MPI_comm comm) { double tstart = MPI_Wtime(); int extent; int result = PMPI_Send(buffer,count,datatype,dest, tag,comm); MPI_Type_size(datatype,&extent); /* Compute size */ totalBytes += count * extent; totalTime += MPI_Wtime() - tstart; return result; }

Control of Profiling MPI_PCONTROL() allows user to –enable and disable profiling during execution –flush trace buffers at non-critical points –add user events to a trace file Default state after MPI_INIT has been called is for profiling to be enabled at normal default level

Trace File Formats Self-Defining Data Format (SDDF) - Pablo Alog - MPE logging VAMPIR AIMS IBM VT format PICL

Parallel Tools Library (PTLIB) Access to and information about parallel tools Communication libraries, compilers, debuggers, performance tools Initial focus on parallel performance tools Parallel Tools Digest Discussion forums

PTLIB Evaluation of Performance Tools for MPI Tracefile generation and analysis tools –AIMS –MPE logging and nupshot –Pablo –VAMPIR Automated runtime analysis tool –Paradyn

PTLIB Evaluation Criteria Robustness Usability Scalability Portability Versatility

PTLIB Evaluation Platforms UNIX workstations IBM SP Cray T3E SGI Power Challenge Array SGI Origin 2000

Parallel Performance Tool Capabilities

SDDF Example (cont.) “Seek” { [2] { , 0 }, , 70013, 0, , 3, 0, 0, 0 };;