1. 1 HPCI Presentation Kulathep Charoenpornwattana

2. March 12, 2007 2 Outline Parallel programming with MPI Running MPI applications on Azul & Itanium Running MPI applications on LONI

3. 3 Parallel Programming with MPI

4. March 12, 2007 4 Parallel Programming Parallel programming is a process of splitting task into smaller pieces and executing them simultaneously on multiple different processors to get the results faster. Applications should be written in one of the parallel programming models in order to take advantages of High Performance Computing cluster. The most commonly used parallel programming model in high performance computing environments is Message Passing Interface(MPI).

5. March 12, 2007 5 MPI Programming MPI is a library of routines that provides the functionality needed for developer to write a parallel applications. MPI has total of 126 functions but many MPI applications are written with not more than 10 basic functions.

6. March 12, 2007 6 Example of MPI application #include #include “mpi.h” int main (int argc, char * argv[]){ int processID; // Identifier of process (rank)‏ int noprocess; // Number of processes char compName[100]; // Hostname int nameSize; // Length of name MPI_Init(&argc, &argv); MPI_Comm_size(MPI_COMM_WORLD, &num_process); MPI_Comm_rank(MPI_COMM_WORLD,&processID); MPI_Get_processor_name(compName, &nameSize); printf(“Hello from process %d of %d on %s\n”,processID,noprocess,compName); MPI_Finalize(); return 0; }

7. March 12, 2007 7 MPI functions MPI_Init(*argc, **argv)‏ is used to initialize an MPI session. Every MPI application must call to MPI_Init()‏ MPI_Finalize()‏ is called to shutdown MPI session. MPI_Finalize() should be called in the end of MPI application to free memory.

8. March 12, 2007 8 MPI Programming MPI_Comm_size(comm, size)‏ is used to determine the total number of processes running in a communicator. MPI_Comm_rank(comm, rank)‏ is used to determine the unique identifier (rank) of current process.

9. March 12, 2007 9 MPI Programming 2 5 4 03 1 MPI_COMM_WORLD - MPI_COMM_WORLD is a communicator created by MPI_Init()‏ - All processes are member of MPI_COMM_WORLD by default

10. March 12, 2007 10 MPI Programming (Cont.)‏ MPI_Send(buf,count,datatype,dest,tag, comm)‏ is used to send information from one process to another process. MPI_Recv(buf,count,datatype,source,tag,comm,status)‏ is used to receive information from another process. buf = Address of the buffer count = number of element to send/receive (i.e. 1,2,3,… )‏ datatype = datatype of element (i.e. MPI_CHAR, MPI_INT, MPI_UNSIGNED_INT, etc)‏ dest/source = process id of destination/source process tag = message tag for distinguish among multiple messages comm = communicator (i.e. MPI_COMM_WORLD)‏ * MPI_Send must be matched with a corresponding call to MPI_Recv in the receiving process

11. March 12, 2007 11 Example of MPI programming // Count how many 0’s and 1’s in an array … If (rank == 0){ tsize = array_size / num_process; for(i = 1 ; i < num_process ; i++){ MPI_Send(&tsize, 1, MPI_INT, i, 0, MPI_COMM_WORLD); MPI_Send(array + (i * tsize), tsize, MPI_INT, i, 0, MPI_COMM_WORLD); … } }else{ // compute nodes // Receive size of the array MPI_Recv(&tsize, 1, MPI_INT, 0, 0, MPI_COMM_WORLD, &status); // Allocate memory to hold the array int *array = (int*) malloc(sizeof(MPI_INT)* tsize); // Receiving array MPI_Recv(array, tsize, MPI_INT, 0, 0, MPI_COMM_WORLD, &status); for(j = rank;j < tsize;j++){ // Start counting if (array[j] == 0)‏ zero += 1; } …

12. 12 Compiling & Executing MPI on Azul & Itanium

13. March 12, 2007 13 Compiling MPI Applications Azul supports 3 MPI programming environments. –LAM/MPI (default)‏ –MPICH –OpenMPI Itanium supports 2 MPI programming environments. –LAM/MPI (default)‏ –MPICH User can switch between environment by using switcher –$ switcher mpi –show // display current environment –$ switcher mpi –list// display available environments –$ switcher mpi = [lammpi, mpich, openmpi] // switch environment

14. March 12, 2007 14 Compiling MPI Applications To compile MPI use -C $ mpicc source.c –o your_app.o -C++ $ mpic++ source.cpp –o your_app.o -Fortran $ mpif77 source.f –o your_app.o

15. March 12, 2007 15 Executing MPI Applications Using LAM/MPI –Obtain host file from /home/azul/host.lst host.lst contains hostname of compute nodes in the cluster. –Booting the runtime system with lamboot $ lamboot –v host.lst –Execute application with mpirun (xx = number of processes)‏ $ mpirun –np xx your_app.o –Clean up any processes with lamclean $ lamclean –Shutdown the runtime system with lamhalt $ lamhalt

16. March 12, 2007 16 Executing MPI Applications Using MPICH –Obtain host file from /home/azul/host.lst –Execute application with mpirun $ mpirun –np xx –machinefile host.lst your_app.o

17. March 12, 2007 17 Executing MPI Applications Using OpenMPI –Obtain host file from /home/azul/host.lst –$ mpirun –np xx --hostfile host.lst your_app.o

18. 18 Submitting jobs to Azul & Itanium

19. March 12, 2007 19 Submiting job Jobs can be submitted through default scheduling system (Torque)‏ $ qsub qsubfile.qsub // submits jobs to pbs $ qstat // displays jobs status $ qdel jobID // deletes job $ pbsnodes –a // displays nodes status

20. March 12, 2007 20 Submiting job qsub command only accepts qsub scripts Example of qsub file #!/bin/sh #PBS –N Myjob// Jobs’s name #PBS –o output.txt// Output file #PBS –e error.txt// Error file #PBS –q workq// submit to workq lamboot –v /home/ter/hosts.lst mpirun –np 5 /home/ter/my_app.o exit 0

21. March 12, 2007 21 Demo …

22. 22 Compiling & Executing MPI on LONI

23. March 12, 2007 23 Compiling MPI on LONI To compile MPI use -C $ mpcc_r source.c –o your_app.o -C++ $ mpCC_r source.cpp –o your_app.o -Fortran $ mpxlf_r source.f –o your_app.o

24. March 12, 2007 24 Executing MPI Apps on LONI Currently, LONI doesn’t allow users to run jobs interactively. All of the jobs must be submitted through jobs scheduler (Loadleveler). If you have to run interactive jobs (mpirun), use Azul or Itanium.

25. 25 Submitting jobs to LONI

26. March 12, 2007 26 Submitting job to LONI Jobs can be submitted through LONI’s scheduling system (Loadleveler)‏ $ llsubmit script // submits jobs $ llq // displays jobs status $ llcancel jobID // deletes job $ llstatus // displays nodes status

27. March 12, 2007 27 Example of Loadleveler script # @ shell= /bin/ksh # @ initialdir= ~/ # @ output = ~/output.txt # @ error = ~/error.txt # @ class = workq # @ job_type = parallel # @ node = 2 # @ tasks_per_node = 8 # @ queue./helloworld

28. March 12, 2007 28 Questions/Answers Got Questions?

29. March 12, 2007 29 References LONI’s user guide: https://www.loni.org/systems/help/users_guide.php https://www.loni.org/systems/help/users_guide.php Azul & Itanium user guide http://hpci.latech.edu MPI: http://webct.ncsa.uiuc.edu:8900/public/MPI/