1. Parallel Computing @ ISDS Chris Hans 29 November 2004

2. Parallel Computing @ ISDS2 Organization  Basics of Parallel Computing Structural Computational  Coding for Distributed Computing  Examples  Resources at Duke CSEM Cluster

3. 29 November 2004Parallel Computing @ ISDS3 Basics of Distributed Computing  Online Tutorial (Livermore Nat. Lab.) http://www.llnl.gov/computing/tutorials/parallel_comp/  Serial Computing: one computer, one CPU  Parallel Computing: multiple computers working at the same time

4. 29 November 2004Parallel Computing @ ISDS4 Various Setups  Collection of Workstations PVM (Parallel Virtual Machine) R (rpvm, Rmpi, SNOW) LAM-MPI (Local Area Multicomputer) Matlab  Dedicated Cluster Master/Slave model MPI

5. 29 November 2004Parallel Computing @ ISDS5 Network Layout  Basic layout: Each node has: CPU(s), memory

6. 29 November 2004Parallel Computing @ ISDS6 Designing a Parallel Program Do the nodes need to interact? “Embarrassingly Parallel” Very little communication -- Monte Carlo “Shamelessly Parallel” Communication Needed -- Heat diffusion models -- Spatial models

7. 29 November 2004Parallel Computing @ ISDS7 Message Passing Interface  “MPI” Standard Easy to use functions that manage the communication between nodes.

8. 29 November 2004Parallel Computing @ ISDS8 Master/Slave Model  Organize the layout: Slaves

9. 29 November 2004Parallel Computing @ ISDS9 Master/Slave Model  “Master” divides the task into pieces…  …while slaves “listen” to the network, waiting for work.  Master sends out the work to be done…  …slaves do the work…  …while the Master waits for the answers.  Slaves return the results.

10. 29 November 2004Parallel Computing @ ISDS10 Example: Monte Carlo

11. 29 November 2004Parallel Computing @ ISDS11 Example: Monte Carlo Master Slave 1 Draw N/3 values Compute Return Slave 2 Same Slave 3 Same

12. 29 November 2004Parallel Computing @ ISDS12 Code  Write ONE program Same for master and slaves  Run program on EACH node Each program has to figure out if it’s the master or a slave MPI

13. 29 November 2004Parallel Computing @ ISDS13 Pseudo Code LOAD DATA; GET ID; IF(ID==MASTER) { MASTER(); } ELSE { SLAVE(); }... LOAD DATA; GET ID; IF(ID==MASTER) { MASTER(); } ELSE { SLAVE(); }... LOAD DATA; GET ID; IF(ID==MASTER) { MASTER(); } ELSE { SLAVE(); }... LOAD DATA; GET ID; IF(ID==MASTER) { MASTER(); } ELSE { SLAVE(); }... MasterSlave 1Slave 2Slave 3

14. 29 November 2004Parallel Computing @ ISDS14 MASTER { // Find # of nodes GET NP; for(i in 1:NP) { “Tell process i to compute the mean for 1000 samples” } RET=RES=0; // Wait for results WHILE(RET<NP){ ANS = RECEIVE(); RES+=ANS/NP; RET++; } SLAVE { ANS = 0; // Wait for orders // Receive NREPS for(i in 1:NREPS) { ANS += DRAW(); } SEND(ANS/NREPS); RETURN TO MAIN; }; SLAVE { ANS = 0; // Wait for orders // Receive NREPS for(i in 1:NREPS) { ANS += DRAW(); } SEND(ANS/NREPS); RETURN TO MAIN; }; SLAVE { ANS = 0; // Wait for orders // Receive NREPS for(i in 1:NREPS) { ANS += DRAW(); } SEND(ANS/NREPS); RETURN TO MAIN }; MasterSlave 1Slave 2Slave 3

15. 29 November 2004Parallel Computing @ ISDS15 PRINT RES; RETURN TO MAIN ; }... FINALIZE(); }... FINALIZE(); }... FINALIZE(); }... FINALIZE(); } MasterSlave 1Slave 2Slave 3 Pseudo Code

16. 29 November 2004Parallel Computing @ ISDS16 Example: C++ Code  Large dataset N=40, p = 1,000 One outcome variable, y  Calculate R 2 for all 1-var regression  parallel_R2.cpp

17. 29 November 2004Parallel Computing @ ISDS17 CSEM Cluster @ Duke  Computational Science, Engineering and Medicine Cluster http://www.csem.duke.edu/Cluster/clustermain.htm  Shared machines Core Nodes Contributed Nodes

18. 29 November 2004Parallel Computing @ ISDS18 CSEM Cluster Details  4 Dual processing head nodes  64 Dual processing shared nodes Intel Xeon 2.8 GHz  40 Dual processing “stat” nodes Intel Xeon 3.1 GHz  161 Dual processing “other” nodes Owners get priority

19. 29 November 2004Parallel Computing @ ISDS19

20. 29 November 2004Parallel Computing @ ISDS20 Using the Cluster  Access limited ssh –l cmh27 cluster1.csem.duke.edu  Data stored locally on cluster  Compile using mpicc, mpif77, mpif90  Cluster uses SGE Queuing System

21. 29 November 2004Parallel Computing @ ISDS21 Queuing System  Submit your job with requests memory usage number of CPUs (nodes)  Assigns nodes and schedules job Least-loaded machines fitting requirements  Jobs run outside of SGE are killed

22. 29 November 2004Parallel Computing @ ISDS22 Compiling Linked libraries, etc… g++ -c parallel_R2.cpp -I/opt/mpich-1.2.5/include -L/opt/mpich-1.2.5/lib mpicc parallel_R2.o -o parallel_R2.exe -lstdc++ -lg2c -lm

23. 29 November 2004Parallel Computing @ ISDS23 Submitting a Job  Create a queue script: #!/bin/tcsh # #$ -S /bin/tcsh -cwd #$ -M username@stat.duke.edu -m b #$ -M username@stat.duke.edu -m e #$ -o parallel_R2.out -j y #$ -pe low-all 5 cd /home/stat/username/ mpirun -np $NSLOTS -machinefile $TMPDIR/machines parallel_R2.exe Email me when the job is submitted and when it finishes Output file Priority and number of nodes requested Your Home Directory on CSEM Cluster

24. 29 November 2004Parallel Computing @ ISDS24 Submitting a Job  Type: [cmh27@head1 cmh27]$ qsub parallel_R2.q Your job 93734 ("parallel_R2.q") has been submitted.  Monitoring http://clustermon.csem.duke.edu/ganglia/

25. 29 November 2004Parallel Computing @ ISDS25 Downloadable Files You can download the slides, example C++ code and queuing script at: http://www.isds.duke.edu/~hans/tutorials.html