1. CS 584

2. Message Passing Based on multi-processor
Set of independent processors
Connected via some communication net
All communication between processes is done via a message sent from one to the other

3. MPI Message Passing Interface Computation is made of:
One or more processes
Communicate by calling library routines
MIMD programming model
SPMD most common.

4. MPI Processes use point-to-point communication operations
Collective communication operations are also available.
Communication can be modularized by the use of communicators.
MPI_COMM_WORLD is the base.
Used to identify subsets of processors

5. MPI
Complex, but most problems can be solved using the 6 basic functions.
MPI_Init
MPI_Finalize
MPI_Comm_size
MPI_Comm_rank
MPI_Send
MPI_Recv

6. MPI Basics
Most all calls require a communicator handle as an argument.
MPI_COMM_WORLD
MPI_Init and MPI_Finalize
don’t require a communicator handle
used to begin and end and MPI program
MUST be called to begin and end

7. MPI Basics MPI_Comm_size MPI_Comm_rank
determines the number of processors in the communicator group
MPI_Comm_rank
determines the integer identifier assigned to the current process
zero based

8. MPI Basics #include <stdio.h> #include <mpi.h>
main(int argc, char *argv[]) {
int iproc, nproc;
MPI_Init(&argc, &argv);
MPI_Comm_size(MPI_COMM_WORLD, &nproc);
MPI_Comm_rank(MPI_COMM_WORLD, &iproc);
printf("I am processor %d of %d\n", iproc, nproc);
MPI_Finalize(); }

9. MPI Communication MPI_Send MPI_Recv Sends an array of a given type
Requires a destination node, size, and type
MPI_Recv
Receives an array of a given type
Same requirements as MPI_Send
Extra parameter
MPI_Status variable.

11. MPI Basics Made for both FORTRAN and C Standards for C
MPI_ prefix to all calls
First letter of function name is capitalized
Returns MPI_SUCCESS or error code
MPI_Status structure
MPI data types for each C type

12. Using MPI Based on rsh Path to compiler requires a .rhosts file
hostname login
Path to compiler
MPI_HOME /users/faculty/snell/mpich
MPI_CC MPI_HOME/bin/mpicc

13. Using MPI Write program Compile using mpicc Write process file
host nprocs full_path_to_prog
0 for nprocs on first line 1 for all others
Run program
prog -p4pg process_file args
mpirun –np #procs –machinefile machines prog

14. Example
HINT benchmark
Found at /users/faculty/snell/CS584/HINT

15. #include “mpi.h”
#include <stdio.h>
#include <math.h>
#define MAXSIZE 1000
void main(int argc, char *argv) {
int myid, numprocs;
int data[MAXSIZE], i, x, low, high, myresult, result;
char fn[255];
char *fp;
MPI_Init(&argc,&argv);
MPI_Comm_size(MPI_COMM_WORLD,&numprocs);
MPI_Comm_rank(MPI_COMM_WORLD,&myid);
if (myid == 0)
{ /* Open input file and initialize data */
strcpy(fn,getenv(“HOME”));
strcat(fn,”/MPI/rand_data.txt”);
if ((fp = fopen(fn,”r”)) == NULL) {
printf(“Can’t open the input file: %s\n\n”, fn);
exit(1); }
for(i = 0; i < MAXSIZE; i++) fscanf(fp,”%d”, &data[i]);
/* broadcast data */
MPI_Bcast(data, MAXSIZE, MPI_INT, 0, MPI_COMM_WORLD);
/* Add my portion Of data */
x = n/nproc;
low = myid * x;
high = low + x;
for(i = low; i < high; i++)
myresult += data[i];
printf(“I got %d from %d\n”, myresult, myid);
/* Compute global sum */
MPI_Reduce(&myresult, &result, 1, MPI_INT, MPI_SUM, 0, MPI_COMM_WORLD);
if (myid == 0) printf(“The sum is %d.\n”, result);
MPI_Finalize();

16. MPI
Message Passing programs are non-deterministic because of concurrency
Consider 2 processes sending messages to third
MPI does guarantee that 2 messages sent from a single process to another will arrive in order.
It is the programmer's responsibility to ensure computation determinism

17. MPI & Determinism MPI Non-Determinism MPI_ANY_SOURCE or MPI_ANY_TAG
A Process may specify the source of the message
A Process may specify the type of message
Non-Determinism
MPI_ANY_SOURCE or MPI_ANY_TAG

18. Example for (n = 0; n < nproc/2; n++) {
MPI_Send(buff, BSIZE, MPI_FLOAT, rnbor, 1,
MPI_COMM_WORLD);
MPI_Recv(buff, BSIZE, MPI_FLOAT, MPI_ANY_SOURCE,
1, MPI_COMM_WORLD, &status);
/* Process the data */ }

19. Global Operations
Coordinated communication involving multiple processes.
Can be implemented by the programmer using sends and receives
For convenience, MPI provides a suite of collective communication functions.

20. Collective Communication
Barrier
Synchronize all processes
Broadcast
Gather
Gather data from all processes to one process
Scatter
Reduction
Global sums, products, etc.

21. Collective Communication

22. MPI_Reduce
MPI_Reduce(inbuf, outbuf, count, type, op, root, comm)

23. MPI_Reduce

24. MPI_Allreduce
MPI_Allreduce(inbuf, outbuf, count, type, op, root, comm)

25. Distribute
Problem Size
Distribute
Input data
Exchange
Boundary values
Find Max Error
Collect Results

26. Other MPI Features Asynchronous Communication Modularity
MPI_ISend
MPI_Wait and MPI_Test
MPI_Probe and MPI_Get_count
Modularity
Communicator creation routines
Derived Datatypes