MPI Message Queue Debugging Interface Chris Gottbrath Director, Product Management

2 TotalView Technologies –Proprietary– Plans Subject to Change without Notice MPI Message Queue Helper Library Provides access to MPI message queue state at a point in time –Within one paused, deadlocked, hung, or crashed MPI process Certain TotalView functions apply this API across many processes to gather a global view of MPI process queue state within an MPI job –For troubleshooting, debugging and diagnostic purposes Message Queue Model –Three conceptual queues MPI Send Queue MPI Receive Queue MPI Unexpected Queue –MPI can provide info for any of all of these queues –The MPI may have more or less (or different) queues –Message traffic that is part of collective operations may or may not be included Data Presented includes –local communicator handle, local and remote rank id, message tag, message data or destination buffer into which the data will be written

3 TotalView Technologies –Proprietary– Plans Subject to Change without Notice MPI Message Queue Helper Library Helper library –Provided by MPI vendor –Loaded by debugging tool Tool provides basic services via callback functions –Locating symbols in target –Reading data from target Library encapsulates MPI library specific details –Isolates debugging tools from MPI library internals –Multiple library version compatibility with the same debugging tool –New MPI libraries can be supported with no tool changes Advantages of library implementation –Live target or post-mortem corefile target –Doesnt require target to make progress –Target types are explicitly declared so the target architecture type can differ from that of the library and of the debugger (which do need to match)

4 TotalView Technologies –Proprietary– Plans Subject to Change without Notice Message Queue Helper Library Current Status –Wide industry acceptance Numerous MPI implementations have adopted this strategy –MPICH 1 and MPICH 2 –Open MPI –Intel MPI –IBM POE –SGI MPT –MVAPICH –Sun Cluster Tools Library is used by multiple debugging tools Open source & proprietary closed source –Publicly documented Cownie and Gropp, EuroPVM/MPI 1999 Multiple open source implementations serve as further documentation Standardization –Should this be part of the MPI standard?