1. 1 Lecture 4: Distributed-memory Computing with PVM/MPI

2. 2 Cluster Computing What is cluster computing  Use a number of work stations or PCs to construct a virtual parallel computer.  Comparing with parallel computers, cluster processing is cheap.  Performance is worse than parallel computers, but it can be largely improved by using high performance network.

3. 3 Global/Cloud Computing What is global (GRID) computing Use all the resources on internet to construct a super distributed parallel virtual computer Intranet + Internet Super parallel computers PC cluster User High performance PC

4. 4 Implement of cluster processing Standard software for cluster processing  PVM (Parallel Virtual Machine)  MPI (Message Passing Interface) History of PVM 1989 PVM1.0 released by Oak Ridge National Laboratory. 1991 PVM2.0 released by University of Tennessee. 1993 PVM3.0 released. 1994 Published with free manual 2001PVM 3.4.3 released with C and Fortran. PVM: A Users’ Guide and Tutorial for Networked Parallel Computing http://www.netlib.org/pvm3/book/pvm-book.html

5. 5 Implement of cluster processing History of MPI 1993-1994 Specification (MP-I) was decided by the experts from 40 companies. 1995 Specification of MPI-2 was decided. 2001 MPI-1.1 based implementation was released. It is used as default environment in many distributed parallel processing systems.

6. 6 Common in PVM and MPI  Communication is based on 1 to 1 message passing.  They are designed for not only work stations and PCs but also parallel computers.  Corresponding to many OS and Languages. （ OS: UNIX, Windows95/NT, Languages: C, Fortran, JAVA)  Free  Standard software in distributed and parallel computing. send(5, message ） receive(3) Processor ID: ３ Processor ID: ５

7. 7 Construction of PVM Software of PVM  Daemon pvmd used for the communication between processors.  Console pvm used for constructing virtual parallel computer （ xpvm is the console combined with GUI)  Library of functions such as pvm_send and pvm_receive for message sending and receiving. On any computer, one can use pvm or xpvm to construct a virtual parallel computer, and then make and execute programs. A virtual parallel computer pvmd (opty1) (optima) (opty2) (opty3) （ Constructed by connected 4 computers ）

8. 8 Start of PVM (1) On any computer, use command pvm or xpvm to start pvmd. On optima, use pvm or xpvm to start pvmd.

9. 9 Start of PVM (2) On console pvm add computers to construct a virtual parallel computer. (At the this time, pvmd is started in these computers.) On optima add opty1, opty2, opty3 to virtual parallel computer by pvmd.

10. 10 Execution of PVM Program execution on PVM (1) When the program which contains communication functions of pvm is executed, the specified programs will be executed by pvmd When program prog1 is executed on optima which contains communication functions, the process (in this case prog 1) will be started in all processors by pvmd.

11. 11 Execution of PVM (2) Once the execution of a program started, communication and processing inside the program will be done automatically until the end. Processing is held with the communication between processors.

12. 12 Using PVM Using command pvm chen[11]% pvm pvm> add opty1 opty2 opty3 （ add opty1, opty2, opty3 to the virtual computer ） 3 successful HOST DTID opty1 80000 opty2 c0000 opty3 100000 pvm> conf (Show the configuration of the virtual parallel computer) 4 hosts, 1 data format HOST DTID ARCH SPEED DSIG optima 40000 X86SOL2 1000 0x00408841 opty1 80000 X86SOL2 1000 0x00408841 opty2 c0000 X86SOL2 1000 0x00408841 opty3 100000 X86SOL2 1000 0x00408841 pvm> spawn edm4 （ Execute program edm4 on the virtual computer) 1 successful t80001 pvm> halt （ Halt of the virtual computer ） Terminated chen[12]%

13. XPVM Screen Shot

14. 14 Using XPVM Using xpvm (X application)

15. 15 PVM programming – Example 1 (Greeting) 1. Example (hello.c, hello_other.c)

16. 16 PVM Programming – Example 1(Greeting) Start the process hello Hello, Dr. t80004 （ Receive the message by pvm_recv ） Fine, thank you, Dr. t80004 （ Process hello_other is started ） Hello,Prof. t4000a. How are you? （ Send message by pvm_send ） pvm_spawn (Process hello_other is started on other computers by pvm_spawn. Started process has its task ID.) pvm_send pvm_recv （ Finish the process by pvm_exit ）

17. 17 PVM programming – Example 2 (Calculate π) Equation of unit circle: Area of unit circle: π or

18. 18 /* Calculating PI by integral (Mater Program) */ #include /* For standard I/O functions¡ */ #include "pvm3.h" /* For stand pvm functions */ main(argc,argv) int argc; char **argv; { int intervals; /* Number of intervals */ int ctid[2]; /* task id (subtasks) */ int n; /* Number of intervals for one subtask */ double s,t; /* Interval of tasks */ double result1,result2; /* Results of subtasks */ if(argc<=1){ /* Finish if no any parameter */ printf("Usage:pi_dual intervals\n"); pvm_exit(); exit(0); } sscanf(argv[1],"%d",&intervals); /* Get the first parameter and put it into intervals */ if(pvm_spawn("pi_slave",(char **) 0,0,"",2,ctid)!=2){ /* Start two subtasks for executing slave program pi_slave¡ Save the subtask ids into arry ctid */ printf("can't start pi_slave\n"); /* Fail in starting the subtasks */ pvm_exit(); exit(0); } n=intervals/2; /* Interval for one subtask */ s=0;t=0.5; /* First subtask responsible for interval [0,0.5] */ pvm_initsend(0); /* Prepare for data sending */ pvm_pkint(&n, 1,1); /* Put the interval into the sending buffer */ pvm_pkdouble(&s, 1, 1); /* Put the value of s into the sending buffer */ pvm_pkdouble(&t, 1,1); /* Put the value of t into the sending buffer */ pvm_send(ctid[0], 1); /* Send the content of the buffer to the first subtask */ s=0.5;t=1; /* Secong subtask responsible for interval [0.5,1] */ pvm_initsend(0); /* Prepare for data sending */ pvm_pkint(&n, 1,1); /* Put the interval into the sending buffer */ pvm_pkdouble(&s, 1, 1); /* Put the value of s into the sending buffe */ pvm_pkdouble(&t, 1,1); /* Put the value of t into the sending buffe */ pvm_send(ctid[1], 1); /* Send the content of the buffer to the second subtask */ pvm_recv(-1, 1); /* Waiting for receiving the data from the subtasks, and put the data into the receiving buffer when received */ pvm_upkdouble(&result1,1,1); /* Put the received value (the result from the first subtask) into result1*/ pvm_recv(-1, 1); /* Waiting for receiving the data from the subtasks, and put the data into the receiving buffer when received */ pvm_upkdouble(&result2,1,1); /*Put the received value (the result from the second subtask)into result2*/ printf("Intervals=%d, PI=%1.15f",intervals,result1+result2); /* Show the result */ pvm_exit(); /* End */ exit(0); } Master Program (pi_dual.c) PVM programming – Example 2 (Calculate π)

19. 19 /* Program for calculating PI by integral (slave program) */ #include /* Using numerical fuctions (sqrt, etc.) */ #include "pvm3.h" /* Using pvm functions */ main() { int ptid; /* Master task id */ int n; /* Number of intercals */ double s,t; /* Interval of integral */ int i; /* Variable for loop */ double h; /* Width of one interval */ double sum=0; /* Accumulated value of the area. Initial value=0 */ double x; /* x coordinate used at present */ ptid = pvm_parent(); /* put master task id into ptid */ pvm_recv(ptid, 1); /* Wating for receiving the data from master task, and put the data into the receiving buffer when received */ pvm_upkint(&n, 1, 1); /*Put the interval assigned for one task into n */ pvm_upkdouble(&s, 1,1); /*Put the left endpoint of the interval to s*/ pvm_upkdouble(&t, 1,1); /* Put the right endpoint into t */ h=(t-s)/n; /* Width of small intervals */ for (i=0;i<n;++i){ /* Repeat the loop n times */ x=(i+0.5)*h+s; /* Midpoint of x coordinate */ sum+=(4*sqrt(1-x*x))*h; /* Calculate the area of one small interval and add it to sum. */ } pvm_initsend(0); /* Prepare for sending data */ pvm_pkdouble(&sum, 1, 1); /* Put sum into the sending buffer */ pvm_send(ptid, 1); /* Send the content of the sending buffer into the master process */ pvm_exit(); /* End */ exit(0); } Slave Program (pi_slave.c) PVM programming – Example 2 (Calculate π)

20. 20 MPI programming – Example 1(Greeting) master slave

21. 21 MPI programming – Example 2 (Calculate π) #include "mpi.h" #include int main( int argc, char *argv[] ) { int n, myid, numprocs, i; double PI25DT = 3.141592653589793238462643; double mypi, pi, h, sum, x; MPI_Init(&argc,&argv); MPI_Comm_size(MPI_COMM_WORLD,&numprocs); MPI_Comm_rank(MPI_COMM_WORLD,&myid); while (1) { if (myid == 0) { printf("Enter the number of intervals: (0 quits) "); scanf("%d",&n); } MPI_Bcast(&n, 1, MPI_INT, 0, MPI_COMM_WORLD); if (n == 0) break; else { h = 1.0 / (double) n; sum = 0.0; for (i = myid + 1; i <= n; i += numprocs) { x = h * ((double)i - 0.5); sum += (4.0 / (1.0 + x*x)); } mypi = h * sum; MPI_Reduce(&mypi, &pi, 1, MPI_DOUBLE, MPI_SUM, 0, MPI_COMM_WORLD); if (myid == 0) printf("pi is approximately %.16f, Error is %.16f\n", pi, fabs(pi - PI25DT)); } MPI_Finalize(); return 0; } one task reduction function for parallel summation

22. 22 MPI programming – Reduction

23. PVM? MPI?  PVM is easy to use, especially on a network of workstations. Its message passing API is relatively simple  MPI is a standard, has a steeper learning curve and doesn’t have a standard way to start tasks MPICH does have an “mpirun” command  If building a new scalable, production code, should use MPI (widely supported now)  If experimenting with message passing, are interested in dynamics, use PVM.  MPI has a very rich messaging interface and designed for efficiency http://www-unix.mcs.anl.gov/mpi  PVM has a simple messaging interface + Process control, Interoperability, Dynamics: http://www.epm.ornl.gov/pvm  Perform comparably when on Ethernet  MPI outperforms when on MPP  Both are still popular, but MPI is an accepted community standard with many support chains.

24. Exercise 24 (1) Read the PVM program pi_dual.c and pi_slave.c which are used for calculating. Compile pi_dual.c and pi_slave.c. Learn to use xpvm. (2) Revise program pi_dual.c to pi_single.c such that it uses only one pi_slave task. Find N (the number of the divided intervals) such that the running time of pi_single is 30 sec, 60 sec, 90 sec, and 120 sec, respectively. (3) Compare the running time of pi_dual and pi_single using the different values of N obtained in (2). Notice that the ideal speed-up rate is (4) Revise program pi_dual.c to pi_multi.c such that it can use any number of slave tasks. For example, Pvm> spawn -> pi_multi 200000000 4 starts 4 slave tasks which do calculation in intervals [0,0.25],[0.25,0.5],[0.5,0.75],[0.75,1], respectively, where each interval is divided to 50000000 smaller intervals. (5) For each value of N obtained in (2), get the running time pi_multi when the number of slave tasks is 1,2,4,8,16, respectively, find the speed-up rate (notice that the ideal rate is, investigate the change in the speed- up rates, and discuss the reasons. (6) Consider how to balance the work load among the host computers. (Hint: according to the work load of each computer, the time needed for running pi_slave is different. Assign more slave tasks to the computers which has lower work load. Design the PVM program to realize your idea, and discuss how much the running time and the speed-up rate are improved. (7) Change the above PVM programs into the MPI programs.