Debugging Lab Antonio Gómez-Iglesias Texas Advanced Computing Center

Setup – X11 Login to Stampede: -ssh -X Untar the lab files: – cd – tar zxvf ~train00/dbg_lab.tgz Change directories and ls to see the files: – cd dbg_lab – ls

GNU DEBUGGING LAB

Find a bug with gdb In this example we will use gdb to debug a code that crashes. First, you should be on a compute node when debugging with gdb % idev -p normal -A TRAINING-HPC % idev -p normal -A TG-TRA After your session begins, compile the deadlock example: % icc -g -o divcrash./divcrash.c # C example Run the code %./divcrash What happened?

Find a bug with gdb Now, we’ll run with gdb Use gdb to load the executable: % gdb./divcrash Kick off the run (gdb) run Where does the code crash? What line was it on? Set a breakpoint at the line that the code crashed on Step through each iteration using the continue command Print the value of x and y at each breakpoint (this will not work for codes compiled with ifort with the default gdb)

DDT DEBUGGING LAB

Finding a deadlock with DDT In this example we will use DDT to debug a code that deadlocks (you need X11, so ssh -X ). Exit from your idev session Compile the deadlock example: % mpicc -g -O0 -o deadlock./deadlock.c #C example % mpif90 -g -O0 -o deadlock./deadlock.f90#F90 Load the DDT module: % module load ddt Start up DDT: % ddt./deadlock

DDT Run Window If you are using X11, you should see the default Run window with the executable name in the Application text box. Don’t click submit yet! We need to configure: MPI type Number of processes Memory Debugging Queue Submission Parameters Advanced Options

Configure DDT: Options Choose the correct version of MPI – mvapich 2 – Intel MPI Leave the default MPI (MVAPICH 2) Leave Debugger on the Automatic setting

Configure DDT: Tasks + Memory Debugging Set the number of tasks to 8 Enable memory debugging Enter Memory Debugging Pane

Configure DDT: Memory Debugging Ensure the library is set to C/Fortran If your code is C++, reset this to C++ If ddt crashes with memory debugging enabled, try setting guard pages

Configure DDT: Batch Submission This step is specific to this lab. Normally, you will not have to change this. Open the Configure pane for batch submission

Configure DDT: Batch Submission Update the Submission template file entry to point to the custom template file in the local directory This template file has the reservation command for the batch script created by ddt. Edit the queue parameters

Configure DDT: Queue Parameters Choose the “normal” queue Set your project code - for this training class use: TRAINING-HPC or TG-TRA The reservation text box is specific to this custom template Select the OK button

DDT: Job Queuing Add any necessary arguments to the program (none for the example) Click the Submit button. A new window will open: The job is submitted to the specified queue. An automatically refreshing job status window appears. The debug session will begin when the job starts.

DDT: The debug session Code pane Stack view, input/output, breakpoints, etc. Variable pane Evaluation pane Source navigator Process controls Automatically stops at the first command past MPI_Init

DDT: Program Hangs The output we expect does not appear in the Stdout window. No active communication between procs. Pause execution to analyze the program status (top left).

DDT: Stacks On the bottom left window select the Stacks view. Note that since this version of MPI was compiled with debug information, the halting point is within the MPI library

DDT: Stacks Move to the top of the stack to determine which lines of your code are at the halting point. All processors seem to be stuck on a MPI_Send().

DDT: What’s Wrong? This is a classic deadlock situation. All tasks are engaged in a blocking send, but there are no tasks receiving. Rearrange the order of communications to remove the deadlock.