Profiling Tools on the NERSC Crays and IBM/SP NERSC User Services N ATIONAL E NERGY R ESEARCH S CIENTIFIC C OMPUTING C ENTER

2 Outline Profiling Tools on NERSC platforms –Cray PVP (killeen, seymour) –Cray T3E (mcurie) –IBM/SP (gseaborg) UNIX profiling/performance analysis tools References

N ATIONAL E NERGY R ESEARCH S CIENTIFIC C OMPUTING C ENTER 3 Why Profile? Characterise application : –Is code cpu bound? –Is code I/O bound? –Is code memory bound? –Analyse communication patterns - D.M. codes Focus optimisation effort... and ultimately.. Improve performance and resource utilisation

N ATIONAL E NERGY R ESEARCH S CIENTIFIC C OMPUTING C ENTER 4 Cray PVP/T3E - Application Characterization Job accounting (ja) ja./a.out ja -st -n a.out - see next slide for sample output Look out for : Maximum Memory Used > available memory Total I/O wait time (locked+unlocked) > 50% User CPU time Multitasking breakdown for parallel codes

N ATIONAL E NERGY R ESEARCH S CIENTIFIC C OMPUTING C ENTER 5 Job accounting : summary report Elapsed Time : 8 Seconds User CPU Time : Seconds Multitasking/ Multistreaming Breakdown (Concurrent CPUs * Connect seconds = CPU seconds) 1 * = * = * = * = (Avg.) (total) (total) 3.99 * = System CPU Time : Seconds I/O Wait Time (Locked) : I/O Wait Time (Unlocked) : CPU Time Memory Integral : Mword-seconds Data Transferred : MWords Maximum memory used : MWords

N ATIONAL E NERGY R ESEARCH S CIENTIFIC C OMPUTING C ENTER 6 HPM - Hardware Performance Monitor Helps locate CPU related code bottlenecks reports use of vector registers, instruction buffers, memory ports hpm {options}./a.out {prog_arguments} options = -g2 -> memory access information options = -g3 -> vector register information Look for : Ratio of Floating Ops/CPU second to CPU mem. references per sec should reflect the FpOps in the code

N ATIONAL E NERGY R ESEARCH S CIENTIFIC C OMPUTING C ENTER 7 Sample hpm output : (hpm -g0./a.out) Million inst/sec (MIPS) : 7.67 Instructions : Avg. clock periods/inst : % CP holding issue : CP holding issue : Inst.buffer fetches/sec : 0.04M Inst.buf. fetches: Floating adds/sec : 15.40M F.P. adds : Floating multiplies/sec : 24.36M F.P. multiplies : Floating reciprocal/sec : 0.28M F.P. reciprocals : Cache hits/sec : 0.00M Cache hits : CPU mem. references/sec : 34.64M CPU references : Floating ops/CPU second: 40.5M

N ATIONAL E NERGY R ESEARCH S CIENTIFIC C OMPUTING C ENTER 8 Cray PVP : CPU Bound Codes: prof/profview Instruments code to provide % cpu time in function calls f90 -lprof prog.f90./a.out -> generates prof.data prof -st./a.out > prof.report Chart (over) indicates relative distribution of CPU execution time by function call –prof -x a.out > pgm.prof –profview pgm.prof

N ATIONAL E NERGY R ESEARCH S CIENTIFIC C OMPUTING C ENTER 9 Profview - Sample Output

N ATIONAL E NERGY R ESEARCH S CIENTIFIC C OMPUTING C ENTER 10 I/O and Memory Bound Codes : procstat/procview procstat -m -i -R a.raw a.out procview a.raw –I/O Analysis : Reports, Files -> All User Files (Long Report) Bytes Processed or I/O Wait Time –Memory Analysis : Reports -> Processes -> Maximum Memory Used (Long Format)

N ATIONAL E NERGY R ESEARCH S CIENTIFIC C OMPUTING C ENTER 11 I/O Bound Codes : procview procview indicates which files consume most real time for I/O processing

Memory Bound Codes : procview –“High” (> 10% Elapsed Time) Time to complete Memory requests may indicate memory bound code –Use Graphs option to produce plot of Memory use over elapsed time of application N ATIONAL E NERGY R ESEARCH S CIENTIFIC C OMPUTING C ENTER 12

N ATIONAL E NERGY R ESEARCH S CIENTIFIC C OMPUTING C ENTER 13 ATExpert - Autotasking Prediction Analysis of source code to predict autotasking performance on dedicated Cray PVP f90 -eX -O3 -r4 -o {prog_name} prog.f90 –./a.out –atexpert -> shows predicted speed-up

N ATIONAL E NERGY R ESEARCH S CIENTIFIC C OMPUTING C ENTER 14 ATExpert Sample output Indicates predicted speed-up of 4.3 on dedicated 8 processor PVP when source code is autotasked

N ATIONAL E NERGY R ESEARCH S CIENTIFIC C OMPUTING C ENTER 15 Also available on Cray PVP Flowtrace/flowview times (using Operating System timers) subroutines and functions during program execution jumptrace/jumpview provides exact timing in function/subroutine by analysis of machine instructions in program perftrace/perfview times subroutines/functions based on statistics gathered from HPM tool

N ATIONAL E NERGY R ESEARCH S CIENTIFIC C OMPUTING C ENTER 16 Cray T3E - Apprentice Locate performance problems /inefficiencies MPI and shared memory performance, load balance and communication, memory use Provides hardware performance information and tuning recommendations (Displays -> Observations) Compile/link f90 -o {prog} -eA {prog_name.f90} -lapp cc -o {prog} -happrentice {prog_name.c} -lapp Run code to generate app.rif

N ATIONAL E NERGY R ESEARCH S CIENTIFIC C OMPUTING C ENTER 17 Output from : apprentice app.rif

N ATIONAL E NERGY R ESEARCH S CIENTIFIC C OMPUTING C ENTER 18 Cray T3E - PAT Generates profile of CPU time in functions; load balance across PEs; h/w counter info. Compile and Link with PAT library f90 -o exe -lpat {source.f} pat.cld Run program as normal mpprun -n {procs} {exe} -> generate exe.pif pat executable exe.pif

N ATIONAL E NERGY R ESEARCH S CIENTIFIC C OMPUTING C ENTER 19 Profile based on relative CPU time in function calls Load Balance Histogram for routine “COLL”

N ATIONAL E NERGY R ESEARCH S CIENTIFIC C OMPUTING C ENTER 20 Cray T3E - ACTS/TAU Performance analysis of distributed/shared memory applications (C++ in particular) module load tau instrument programs with TAU macros add $(TAU_DEFS), $(TAULIBS) to compile/link run application; view tracefile with pprof, VAMPIR Reference

N ATIONAL E NERGY R ESEARCH S CIENTIFIC C OMPUTING C ENTER 21 Cray T3E - Vampir Analysis of message passing characteristics - generates display of MPI activity over instrumented time period (e.g. sender, receiver, message size, elapsed time) module load VAMPIR; module load vampirtrace Facility to instrument with VAMPIRtrace calls Generate trace file using TAU or VAMPIRtrace Reference :

N ATIONAL E NERGY R ESEARCH S CIENTIFIC C OMPUTING C ENTER 22 IBM/SP - Xprofiler Graphical interface for gprof profiles of parallel applications –Compile and link code with “-g -pg” –poe./a.out -procs {n} generates gmon.out.{n} file for each process may introduce significant (upto factor of 2) overhead –(In $TMPDIR) xprofiler./a.out gmon.out.* Report menu provides (gprof) text profile Source statement profiling shown

N ATIONAL E NERGY R ESEARCH S CIENTIFIC C OMPUTING C ENTER 23

N ATIONAL E NERGY R ESEARCH S CIENTIFIC C OMPUTING C ENTER 24 Statement level profile available by clicking on relevant function graphical output - use Show Source Code option

N ATIONAL E NERGY R ESEARCH S CIENTIFIC C OMPUTING C ENTER 25 IBM/SP - Visualization Tool (VT) Message passing trace visualization Realtime system activity monitor (limited) MPI load balance overview : poe./a.out -procs {n} -tlevel=3 copy a.out.trc to $TMPDIR (In $TMPDIR) Invoke vt In trace visualization mode, “Play” a.out.trc see next slide for sample of Interprocessor Communication during program execution

N ATIONAL E NERGY R ESEARCH S CIENTIFIC C OMPUTING C ENTER 26

N ATIONAL E NERGY R ESEARCH S CIENTIFIC C OMPUTING C ENTER 27 IBM/SP : system_stats IBM Internal Tool module load sptools instrument code with system_stats() call Link with $(SPTOOLS), run code as normal Sample output Summary of the utilization of system resources: node hostname wall(s) user(s) sys(s) size(KB) pswitches 0 gs gs gs gs

N ATIONAL E NERGY R ESEARCH S CIENTIFIC C OMPUTING C ENTER 28 IBM/SP - trace-mpi IBM Internal tool - Quantitative information on MPI calls –module load USG ; module load trace-mpi –Fortran - add $(TRACE_MPIF) to build –C - add $(TRACE_MPI) to build –poe./a.out -procs {n} - generates mpi.trace_file for each process (executable must call MPI_Finalize) –summary mpi.trace_file.{n} (see over) Useful check for load balance : –grep “Total Communication” mpi.trace.file.*

N ATIONAL E NERGY R ESEARCH S CIENTIFIC C OMPUTING C ENTER 29 MPI message-passing summary for mpi.trace_file.3 MPI Function #calls Avg Bytes Time (sec) MPI_Allreduce: MPI_Barrier: MPI_Bcast: MPI_Scatter: MPI_Comm_rank: MPI_Comm_size: MPI_Isend: MPI_Recv: MPI_Wait: Total Communication Information: WALL = , CPU = 15.53, MBYTES = The total amount of wall time =

N ATIONAL E NERGY R ESEARCH S CIENTIFIC C OMPUTING C ENTER 30 Upcoming on the SP ACTS/TAU (C/C++) currently being ported to the IBM/SP VAMPIR has been ordered, awaiting delivery Performance Monitor Toolkit (HPM) should be available with Phase II system (requires AIX 4.3.4) Also, see Performance API project: –

N ATIONAL E NERGY R ESEARCH S CIENTIFIC C OMPUTING C ENTER 31 General/UNIX Profiling Tools Command line profilers and system analysis prof/gprof (enabled for MPI on IBM/SP) csh time command : time./a.out vmstat -> look for high paging over extended time period (application may require more memory) Fortran/C function timers getrusage rtc, irtc etime, dtime, mclock MPI_Wtime

N ATIONAL E NERGY R ESEARCH S CIENTIFIC C OMPUTING C ENTER 32 Reference Material NERSC web pages Cray PVP/Cray T3E –Optimizing Code on Cray PVP Systems –Cray T3E C, Fortran Optimization Guides IBM/SP LLNL Workshop on Performance Tools