1. Lecture 4: Distributed-memory Computing with PVM/MPI

2. Cluster Computing What is cluster computing
Use a number of work stations or PCs to construct a virtual parallel computer.
Comparing with parallel computers, cluster processing is cheap.
Performance is worse than parallel computers, but it can be largely improved by using high performance network.

3. Global/Cloud Computing
What is global (GRID) computing
Use all the resources on internet to construct a super distributed parallel virtual computer
Super parallel
computers
Intranet +
Internet
High performance PC
User
PC cluster

4. Implement of cluster processing
Standard software for cluster processing
　PVM (Parallel Virtual Machine)
　MPI (Message Passing Interface)
History of PVM
1989　PVM1.0 released by Oak Ridge National Laboratory.
1991　PVM2.0 released by University of Tennessee.
1993　PVM3.0 released.
1994　Published with free manual
PVM released with C and Fortran.
PVM: A Users’ Guide and Tutorial for Networked Parallel Computing

5. Implement of cluster processing
History of MPI
　Specification (MP-I) was decided by the
experts from 40 companies.
1995 　Specification of MPI-2 was decided.
2001　MPI-1.1 based implementation was released.
It is used as default environment in many distributed parallel processing systems.

6. Common in PVM and MPI
　Communication is based on 1 to 1 message passing.
　They are designed for not only work stations and PCs but also parallel computers.
　Corresponding to many OS and Languages.
（OS: UNIX, Windows95/NT, Languages: C, Fortran, JAVA)
　Free
　Standard software in distributed and parallel computing.
send(5, message）
receive(3)
Processor ID: ３
Processor ID: ５

7. Construction of PVM Software of PVM
Daemon pvmd used for the communication between
processors.
Console pvm used for constructing virtual parallel computer
（xpvm is the console combined with GUI)
Library of functions such as pvm_send and pvm_receive for message sending and receiving.
A virtual parallel computer
pvmd
(opty1)
(optima)
(opty2)
(opty3)
（Constructed by connected 4 computers ）
On any computer, one can use pvm
or xpvm to construct a virtual parallel
computer, and then make and
execute programs.

8. Start of PVM
(1) On any computer, use command pvm or xpvm to start pvmd.
On optima, use pvm or xpvm
to start pvmd .

9. Start of PVM
(2)　On console pvm add computers to construct a virtual parallel computer. (At the this time, pvmd is started in these computers.)
On optima
add opty1, opty2, opty3 to virtual
parallel computer by pvmd.

10. Execution of PVM Program execution on PVM
(1)　When the program which contains communication functions of pvm is executed, the specified programs will be executed by pvmd
When program prog1 is executed on optima
which contains communication functions,
the process (in this case prog 1) will be
started in all processors by pvmd.

11. Execution of PVM
(2) Once the execution of a program started, communication and processing inside the program will be done automatically until the end.
Processing is held with
the communication between processors.

12. Using PVM Using command pvm chen[11]% pvm
pvm> add opty1 opty2 opty3（add opty1, opty2, opty3 to the virtual computer）
3 successful
HOST DTID
opty
opty2 c0000
opty
pvm> conf　(Show the configuration of the virtual parallel computer)
4 hosts, 1 data format
HOST DTID ARCH SPEED DSIG
optima X86SOL x
opty X86SOL x
opty2 c0000 X86SOL x
opty X86SOL x
pvm> spawn edm4　（Execute program edm4 on the virtual computer)
1 successful
t80001
pvm> halt　　 （Halt of the virtual computer）
Terminated
chen[12]%

13. XPVM Screen Shot

14. Using XPVM
Using xpvm (X application)

15. PVM programming – Example 1 (Greeting)
1. Example (hello.c, hello_other.c)

16. PVM Programming – Example 1(Greeting)
Start the process hello
Hello, Dr. t80004
　(Process hello_other is
started on other computers
by pvm_spawn.
pvm_spawn
　Started process has its task ID.)
（Process hello_other is started）
Hello,
Prof
. t4000a.
How are you?
pvm_send
（Send message by pvm_send）
pvm_recv
（Finish the process by pvm_exit）
（Receive the message by pvm_recv）
Fine, thank you, Dr. t80004
（Finish the process by pvm_exit）

17. PVM programming – Example 2 (Calculate π)
Equation of unit circle:
Area of unit circle: π or

18. PVM programming – Example 2 (Calculate π)
Master Program (pi_dual.c)
n=intervals/2; /* Interval for one subtask */
s=0;t=0.5; /* First subtask responsible for interval [0,0.5] */
pvm_initsend(0); /* Prepare for data sending */
pvm_pkint(&n, 1 ,1); /* Put the interval into the sending buffer */
pvm_pkdouble(&s, 1, 1); /* Put the value of s into the sending buffer */
pvm_pkdouble(&t, 1 ,1); /* Put the value of t into the sending buffer */
pvm_send(ctid[0], 1);
/* Send the content of the buffer to the first subtask */
s=0.5;t=1; /* Secong subtask responsible for interval [0.5,1] */
pvm_pkint(&n, 1 ,1); /* Put the interval into the sending buffer */
pvm_pkdouble(&s, 1, 1); /* Put the value of s into the sending buffe */
pvm_pkdouble(&t, 1 ,1); /* Put the value of t into the sending buffe */
pvm_send(ctid[1], 1);
/* Send the content of the buffer to the second subtask */
pvm_recv(-1, 1); /* Waiting for receiving the data from the subtasks,
and put the data into the receiving buffer when received */
pvm_upkdouble(&result1,1,1); /* Put the received value
(the result from the first subtask) into result1*/
pvm_upkdouble(&result2,1,1); /*Put the received value
(the result from the second subtask)into result2*/
printf("Intervals=%d, PI=%1.15f",intervals,result1+result2);
/* Show the result */
pvm_exit(); /* End */
exit(0); }
/* Calculating PI by integral (Mater Program) */
#include <stdio.h> /* For standard I/O functions¡ */
#include "pvm3.h" /* For stand pvm functions */
main(argc,argv)
int argc;
char **argv; {
int intervals; /* Number of intervals */
int ctid[2]; /* task id (subtasks) */
int n; /* Number of intervals for one subtask */
double s,t; /* Interval of tasks */
double result1,result2; /* Results of subtasks */
if(argc<=1){ /* Finish if no any parameter */
printf("Usage:pi_dual intervals\n");
pvm_exit();
exit(0); }
sscanf(argv[1],"%d",&intervals);
/* Get the first parameter and put it into intervals */
if(pvm_spawn("pi_slave",(char **) 0,0,"",2,ctid)!=2){
/* Start two subtasks for executing slave program
pi_slave¡ Save the subtask ids into arry ctid */
printf("can't start pi_slave\n"); /* Fail in starting the subtasks */

19. PVM programming – Example 2 (Calculate π)
Slave Program (pi_slave.c)
/* Program for calculating PI by integral (slave program) */
#include <math.h> /* Using numerical fuctions (sqrt, etc.) */
#include "pvm3.h" /* Using pvm functions */
main() {
int ptid; /* Master task id */
int n; /* Number of intercals */
double s,t; /* Interval of integral */
int i; /* Variable for loop */
double h; /* Width of one interval */
double sum=0; /* Accumulated value of the area. Initial value=0 */
double x; /* x coordinate used at present */
ptid = pvm_parent(); /* put master task id into ptid */
pvm_recv(ptid, 1); /* Wating for receiving the data from master task, and put the data into the receiving buffer when received */
pvm_upkint(&n, 1, 1); /*Put the interval assigned for one task into n */
pvm_upkdouble(&s, 1 ,1); /*Put the left endpoint of the interval to s*/
pvm_upkdouble(&t, 1 ,1); /* Put the right endpoint into t */
h=(t-s)/n; /* Width of small intervals */
for (i=0;i<n;++i){ /* Repeat the loop n times */
x=(i+0.5)*h+s; /* Midpoint of x coordinate */
sum+=(4*sqrt(1-x*x))*h; /* Calculate the area of one small interval and add it to sum. */ }
pvm_initsend(0); /* Prepare for sending data */
pvm_pkdouble(&sum, 1, 1); /* Put sum into the sending buffer */
pvm_send(ptid, 1); /* Send the content of the sending buffer into the master process */
pvm_exit(); /* End */
exit(0);

20. MPI programming – Example 1(Greeting)
slave
master

21. MPI programming – Example 2 (Calculate π)
#include "mpi.h"
#include <stdio.h>
#include <math.h>
int main( int argc, char *argv[] ) {
int n, myid, numprocs, i;
double PI25DT = ;
double mypi, pi, h, sum, x;
MPI_Init(&argc,&argv);
MPI_Comm_size(MPI_COMM_WORLD,&numprocs);
MPI_Comm_rank(MPI_COMM_WORLD,&myid);
while (1) {
if (myid == 0) {
printf("Enter the number of intervals: (0 quits) ");
scanf("%d",&n); }
MPI_Bcast(&n, 1, MPI_INT, 0, MPI_COMM_WORLD);
if (n == 0)
break;
else {
h = 1.0 / (double) n;
sum = 0.0;
for (i = myid + 1; i <= n; i += numprocs) {
x = h * ((double)i - 0.5);
sum += (4.0 / (1.0 + x*x));
mypi = h * sum;
MPI_Reduce(&mypi, &pi, 1, MPI_DOUBLE, MPI_SUM, 0, MPI_COMM_WORLD);
if (myid == 0) printf("pi is approximately %.16f, Error is %.16f\n", pi, fabs(pi - PI25DT));
MPI_Finalize();
return 0;
one
task
reduction function for parallel summation

22. MPI programming – Reduction

23. PVM? MPI?
PVM is easy to use, especially on a network of workstations. Its message passing API is relatively simple
MPI is a standard, has a steeper learning curve and doesn’t have a standard way to start tasks
MPICH does have an “mpirun” command
If building a new scalable, production code, should use MPI (widely supported now)
If experimenting with message passing, are interested in dynamics, use PVM.
MPI has a very rich messaging interface and designed for efficiency
PVM has a simple messaging interface +
Process control, Interoperability, Dynamics:
Perform comparably when on Ethernet
MPI outperforms when on MPP
Both are still popular, but MPI is an accepted community standard with many support chains.

24. Exercise
(1) Read the PVM program pi_dual.c and pi_slave.c which are used for calculating. Compile pi_dual.c and pi_slave.c. Learn to use xpvm.
(2) Revise program pi_dual.c to pi_single.c such that it uses only one pi_slave task. Find N (the number of the divided intervals) such that the running time of pi_single is 30 sec, 60 sec, 90 sec, and 120 sec, respectively.
(3) Compare the running time of pi_dual and pi_single using the different values of N obtained in (2). Notice that the ideal speed-up rate is
(4) Revise program pi_dual.c to pi_multi.c such that it can use any number of slave tasks. For example,
Pvm> spawn -> pi_multi
starts 4 slave tasks which do calculation in intervals [0,0.25],[0.25,0.5],[0.5,0.75],[0.75,1], respectively, where each interval is divided to smaller intervals.
(5) For each value of N obtained in (2), get the running time pi_multi when the number of slave tasks is 1,2,4,8,16, respectively, find the speed-up rate (notice that the ideal rate is, investigate the change in the speed- up rates, and discuss the reasons.
(6) Consider how to balance the work load among the host computers. (Hint: according to the work load of each computer, the time needed for running pi_slave is different. Assign more slave tasks to the computers which has lower work load. Design the PVM program to realize your idea, and discuss how much the running time and the speed-up rate are improved.
(7) Change the above PVM programs into the MPI programs.