1. Introduction to MPI Programming Ganesh C.N.

2. Introduction MPI – Message Passing Interface.
A programming paradigm used widely in parallel computers (specially scalable ones)
Standard and portable message passing system which defines the syntax and semantics of a set of core libraries.
Allows bindings for Fortran 77 and C.
Goal – Allow efficient communication avoiding memory to memory copying and allow overlap of computations and communication.

3. Introduction What is included in MPI - Point to point communication
- Communication domains
- Processor topologies
What is not included in MPI
- Shared memory operations
- Interrupt driven message passing
- Debugging tools
- I/O support

4. General MPI Programs ….
Every MPI Program must contain the preprocessor directive
#include “mpi.h”
The file “mpi.h” contains all definitions, macros and function prototypes needed for compiling an MPI program.
Typical layout of an MPI program:
………
main(int argc, char *argv[]) { ……
/* No MPI functions called before this */
MPI_Init(&argc, &argv);
MPI_Finalize();
/* No MPI functions after this */ }

5. MPI_Init & MPI_Finalize ….
MPI_Init should be called before any other MPI function. Should be called only once.
Arguments – pointers to argc and argv.
Allows system to do startup so that MPI libraries can be used.
It is a directive to the OS which places a copy of the executable on each processor.

6. MPI_Init & MPI_Finalize ….
Each processor begins execution of its copy of the executable.
Different statements can be executed by branching within the program. Typically based on rank.
After all MPI functions, MPI_Finalize must be called. This cleans up any unfinished activities left by MPI.

7. Programming Paradigm ….
The paradigm used by MPI is the SPMD model of parallel programming.
The same program is loaded in all the processors and each processor runs its copy.
But we can obtain the effect of different programs running on different processors by executing branches based on process rank.
Most common method of writing MIMD programs.

8. MPI_Comm_rank and MPI_Comm_size
MPI_Comm_rank returns the rank of the process.
Syntax :
MPI_Comm_rank(MPI_Comm comm, int *rank)
The first argument is a communicator. It is a collection of processes that can send messages to each other.
For basic programs, the communicator is MPI_COMM_WORLD. Predefined in MPI and consists of all the processes running.
MPI_Comm_size returns the number of processes.
Syntax:
MPI_Comm_size(MPI_Comm comm, int *size);

9. Point to point communication
The actual message passing is carried out by MPI_Send and MPI_Recv.
System must append some information called envelope.
Consists of the ranks of sender and receiver, a tag and the communicator.
Syntax:
int MPI_Send(void *msg, int count, MPI_Datatype type, int dest, int tag, MPI_Comm comm);

10. MPI_Send (contd…) The message to be passed is stored in msg.
count and type allow the system to identify the end of message.
type is NOT a C type. But they correspond to standard C types. There are predefined MPI data types such as MPI_INT, MPI_CHAR, MPI_LONG, MPI_FLOAT, MPI_DOUBLE etc.
Receive buffer size and send buffer size need not be equal.

11. MPI_Recv
int MPI_Recv(void *msg, int count, MPI_Datatype type, int src, int tag, MPI_Comm comm, MPI_Status *status);
src can be a ‘wildcard’ called MPI_ANY_SOURCE. Can receive message from any sending process.
No such wildcard for destination.
The communicator used in MPI_Send and MPI_Recv must be the same.
status returns information on the data that was received. It refers to a record with two fields – source and the tag.

12. Process synchronization - Barrier
Barrier – for synchronizing all processes. Each process blocks until every process reaches the same point.
Syntax:
int MPI_Barrier(MPI_Comm comm);

13. Collective Communication - Broadcast
Involves all the processes in a communicator.
Broadcast – a single process sends the same data to every process.
Syntax:
int MPI_Bcast(void *msg, int count, MPI_Datatype type, int root, MPI_Comm comm);
Should be called by all processes with same arguments for root and comm.
A broadcast message CANNOT be received by calling MPI_Recv.

14. Collective Communication – Scatter
Scatter – Distributes a block of memory among processes.
Syntax:
MPI_Scatter(void *send_buf, int send_count, MPI_Datatype send_type, void *recv_buf, int recv_count, MPI_Datatype recv_type, int root, MPI_Comm comm);
Contents of send_buf are split into n segments each consisting of send_count items and sent to all processes in rank order. The send arguments are significant only in the root process.

15. Collective Communication - Gather
Gather – each process sends a block of data. The root process concatenates the data in rank order.
Syntax:
MPI_Gather(void *send_buf, int send_count, MPI_Datatype send_type, void *recv_buf, int recv_count, MPI_Datatype recv_type, int root, MPI_Comm comm);
data
processes
data
processes a0 a1 a2
scatter a0 a1
gather a2

16. Collective Comm. – vector variants
Varying count of data from each process
Can also specify displacements in root
MPI_Gatherv ( void *sendbuf, int sendcnt, MPI_Datatype sendtype, void *recvbuf, int *recvcnts, int *displs, MPI_Datatype recvtype, int root, MPI_Comm comm )
MPI_Scatterv (void *sendbuf, int *sendcnts, int *displs, MPI_Datatype sendtype, void *recvbuf, int recvcnt, MPI_Datatype recvtype, int root, MPI_Comm comm )

17. Collective comm. – Allgather / Alltoall
MPI_Allgather(void *send_buf, int send_count, MPI_Datatype send_type, void *recv_buf, int recv_count, MPI_Datatype recv_type, MPI_Comm comm);
Can be thought of as Gather except that all processes receive the result (instead of just the root)
MPI_Alltoall (void *sendbuf, int sendcount, MPI_Datatype sendtype, void *recvbuf, int recvcount, MPI_Datatype recvtype, MPI_Comm comm)
Like Allgather except that each process sends distinct data to each of the receivers
The jth block sent from process i received by process j and placed in the ith block of recvbuf

18. Collective comm. – Allgather / Alltoall
data
processes
data
processes a0 a0 b0 c0
allgather b0 a0 b0 c0 c0 a0 b0 c0
data
processes
data
processes a0 a1 a2 a0 b0 c0
alltoall b0 b1 b2 a1 b1 c1 c0 c1 c2 a2 b2 c2

19. Collective Communication – Reduce
Reduce – all the processes contribute data which is combined using a binary operation.
Typical binary ops – addition, max, min, and etc.
Syntax:
MPI_Reduce(void *operand, void *result, int count, MPI_Datatype type, MPI_Op op, int root, MPI_Comm comm);
op – binary operator. Eg: MPI_MAX, MPI_MIN, MPI_SUM, MPI_PROD, MPI_LAND, MPI_BAND etc.
All processes must call MPI_Reduce with the same arguments.

20. Other reduce operations
MPI_Allreduce (void *sendbuf, void *recvbuf, int count, MPI_Datatype datatype, MPI_Op op, MPI_Comm comm)
Same as MPI_Reduce except the result appears in the recvbuf of all processes in the group.
Other reduction operations
MPI_Reduce_scatter
MPI_Scan

21. Derived datatypes MPI can handle user defined datatypes also
Type constructor functions – describe member types and their relative locations in memory
MPI_Type_struct (int count, int *blocklengths, MPI_Aint *displacements, MPI_Datatype *types, MPI_Datatype *newtype)
Must always call MPI_Type_commit before using a constructed datatype.

22. Derived datatypes – pack/unpack
Explicitly store non-contiguous data into a contiguous buffer for transmission, then unpack it at the other end.
When sending/receiving packed messages, must use MPI_PACKED datatype in send/receive calls.
MPI_Pack (void *inbuf, int incount, MPI_Datatype datatype, void *outbuf, int outsize, int *position, MPI_Comm comm)
MPI_Unpack (void *inbuf, int insize, int *position, void *outbuf, int outcount, MPI_Datatype datatype, MPI_Comm comm)

23. Subcommunicators
Mechanism for treating a subset of processes as a collection
Consists of :
Group – ordered collection of processes
Context – a tag attached to a group
MPI_Group_incl ( MPI_Group group, int n, int *ranks, MPI_Group *group_out)
MPI_Comm_create ( MPI_Comm comm, MPI_Group group, MPI_Comm *comm_out )

24. Topologies
Linear process ranks don’t reflect logical communication pattern of processes
Optional mapping mechanism between processes and hardware
Grid and graph topologies supported
MPI_Cart_create (MPI_Comm comm_old, int dims, int *num_proc_dims, int *circular, int reorder, MPI_Comm *comm_cart )