1. Profiling, Tracing, Debugging and Monitoring Frameworks Sathish Vadhiyar Courtesy: Dr. Shirley Moore (University of Tennessee)

2. Outline  Profiling,Tracing, Debugging Tools – Main thrust on Vampir and TotalView  Debugging – TotalView

3. Definitions – Profiling and Tracing  Profiling – Summary information about various statistics - Number of function calls made - Execution times for functions - Number of floating point instructions - FLOP count etc.  Tracing –Information about events - Time stamps about entering/exiting calls - Interaction between processes (send and recv traces)

4. Vampir  For performance analysis of parallel applications  Event-based and trace-based performance analysis tool  Different statistical displays – - timeline displays - profiling statistics (execution times of routines) - communication statistics  Can be analyzed at different levels – zoom in facility  Needs VampirTrace for instrumentation

5. Problems Identified  Excessive communication cost  load (im)balance  Communication bottle-necks  Shows percentage spent in all MPI procedures and actual applications  Shows percentage time spent for each MPI procedure

6. Displays

7. Displays (Summary Charts)

8. Activity Charts (Identifying Load Balance)

9. Communication Statistics

10. Timeline Diagram

11. VampirTrace  Vampir graphical tool uses VampirTrace tracing library  VampirTrace uses MPI profiling interface to intercept MPI calls  E.g.: mpicc –o my_program my_program.c -L /lib –lVT Gives rise to my_program.stf

12. My VAMPIR results  [secvss@altix openmp]$ vampir /opt/vampir/examples/redblack_sndrcv.stf  VAMPIR: Could not find matching license.  VAMPIR: Need license for product VA40.  VAMPIR: platform IA64-LIN  VAMPIR: hostid 0x10908e31, network address 144.16.49.142, userid 2000  VAMPIR: Environment variable PAL_ROOT set to  /opt/vampir.  VAMPIR: Environment variable PAL_LICENSEFILE set to  /opt/vampir/etc/license.dat.  VAMPIR: Environment variable VT_ROOT set to  /opt/vampir_trace.

13. Some more Parallel Profiling Libraries  GuideView (for OpenMP from Pallas)  Upshot / Jumpshot (MPI from Argonne)  SvPablo (UIUC for MPI and OpenMP)  DynaProf (UT)

14. Parallel Debugging – TotalView (Etnus)  Source level parallel debugger  Multithreaded, multiprogram debugger for MPI, PVM, OpenMP  Examine and change data  Can attach to all or some of running processes  Separate window for each process  Can change source code while debugging  Start/stop execution

15. TotalView Introduction  Detailed information by diving  Computational steering by changing variable values or by using expressions  Displaying variables across processes/threads

16. Action Points  Breakpoints  Barrier breakpoints  Conditional breakpoints  Evaluation point  Watchpoints

17. MPI and TotalView  mpirun [ MPICH-arguments ] -tv program [ program-arguments ]  totalview poe -a program [ arguments ] [ PE_arguments ]

18. Message Queue Displays

19. Break point Example 1 pid = fork(); 2 if (pid == -1) 3 error ("fork failed"); 4 else if (pid == 0) 5 children_play(); 6 else 7 parents_work(); Conditional Breakpoint: if (i > 20 && i < 25) $stop;

21. Visualization of Data

22. Research Areas  Compact representation of trace files  Minimize overhead due to tracing

23. Further Reading  Review of performance analysis tools for MPI programs (www.netlib.org/utk/people/JackDongarra/PAPERS/perftools-review2.pdf)www.netlib.org/utk/people/JackDongarra/PAPERS/perftools-review2.pdf  TotalView Hands-on Exercises (http://www/llnl/gov/computing/tutorials/workshops/workshop/total_view/MAIN.html