1 ISCM-10 Taub Computing Center High Performance Computing for Computational Mechanics Moshe Goldberg March 29, 2001

2 High Performance Computing for CM 1)Overview 2)Alternative Architectures 3)Message Passing 4)“Shared Memory” 5)Case Study Agenda:

3 1)High Performance Computing - Overview

4 * Understanding HPC concepts * Why should programmers care about the architecture? * Do compilers make the right choices? * Nowadays, there are alternatives Some Important Points

5 Trends in computer development *Speed of calculation is steadily increasing *Memory may not be in balance with high calculation speeds *Workstations are approaching speeds of especially efficient designs *Are we approaching the limit of the speed of light? * To get an answer faster, we must perform calculations in parallel

6 Some HPC concepts * HPC * HPF / Fortran90 * cc-NUMA * Compiler directives * OpenMP * Message passing * PVM/MPI * Beowulf

7

8

9

10

11

12

13

14 2) Alternative Architectures

15 Source: IDC, 2001

16 Source: IDC, 2001

17

18 IUCC (Machba) computers Cray J cpu Memory - 4 GB (500 MW) Origin cpu (R12000, 400 MHz) 28.7 GB total memory PC cluster 64 cpu (Pentium III, 550 MHz) Total memory - 9 GB Mar 2001

19 Chris Hempel, hpc.utexas.edu

20

21 Chris Hempel, hpc.utexas.edu

22 CPU Memory CPU Symmetric Multiple Processors Examples: SGI Power Challenge, Cray J90/T90 Memory Bus

23 Memory CPU Memory CPU Memory CPU Memory CPU Distributed Parallel Computing Examples: SP2, Beowulf

24

25

26

27 3) Message Passing

28 call MPI_SEND(sum,1,MPI_REAL,ito,itag, MPI_COMM_WORLD,ierror) call MPI_RECV(sum,1,MPI_REAL,ifrom,itag, MPI_COMM_WORLD,istatus,ierror) MPI commands -- examples

29 Some basic MPI functions Setup: mpi_init mpi_finalize Environment: mpi_comm_size mpi_comm_rank Communication: mpi_send mpi_receive Synchronization: mpi_barrier

30 Other important MPI functions Asynchronous communication: mpi_isend mpi_irecv mpi_iprobe mpi_wait/nowait Collective communication: mpi_barrier mpi_bcast mpi_gather mpi_scatter mpi_reduce mpi_allreduce Derived data types: mpi_type_contiguous mpi_type_vector mpi_type_indexed mpi_type_pack mpi_type_commit mpi_type_free Creating communicators: mpi_comm_dup mpi_comm_split mpi_intercomm_create mpi_comm_free

31 4) “Shared Memory”

32 CRAY: CMIC$ DO ALL do i=1,n a(i)=i enddo SGI:C$DOACROSS do i=1,n a(i)=i enddo OpenMP: C$OMP parallel do do i=1,n a(i)=i enddo Fortran directives --examples

33 OpenMP Summary OpenMP standard – first published Oct 1997 Directives Run-time Library Routines Environment Variables Versions for f77, f90, c, c++

34 OpenMP Summary Parallel Do Directive c$omp parallel do private(I) shared(a) c$omp end parallel do  optional do I=1,n a(I)= I+1 enddo

35 OpenMP Summary Defining a Parallel Region - Individual Do Loops c$omp parallel shared(a,b) do j=1,n a(j)=j enddo do k=1,n b(k)=k enddo c$omp do private(j) c$omp end do nowait c$omp do private(k) c$omp end do c$omp end parallel

36 OpenMP Summary Parallel Do Directive - Clauses shared private default(private|shared|none) reduction({operator|intrinsic}:var) if(scalar_logical_expression) ordered copyin(var)

37 OpenMP Summary Run-Time Library Routines Execution environment omp_set_num_threads omp_get_num_threads omp_get_max_threads omp_get_thread_num omp_get_num_procs omp_set_dynamic/omp_get_dynamic omp_set_nested/omp_get_nested

38 OpenMP Summary Run-Time Library Routines Lock routines omp_init_lock omp_destroy_lock omp_set_lock omp_unset_lock omp_test_lock

39 OpenMP Summary Environment Variables OMP_NUM_THREADS OMP_DYNAMIC OMP_NESTED

40 RISC memory levels CPU Main memory Cache Single CPU

41 RISC memory levels CPU Main memory Cache Single CPU

42 RISC memory levels Main memory Multiple CPU’s CPU Cache 1 CPU 0 1 Cache 0

43 RISC memory levels Main memory Multiple CPU’s CPU Cache 1 CPU 0 1 Cache 0

44 Main memory Multiple CPU’s CPU Cache 1 CPU 0 1 Cache 0 RISC Memory Levels

45 subroutine xmult (x1,x2,y1,y2,z1,z2,n) real x1(n),x2(n),y1(n),y2(n),z1(n),z2(n) real a,b,c,d do i=1,n a=x1(i)*x2(i); b=y1(i)*y2(i) c=x1(i)*y2(i); d=x2(i)*y1(i) z1(i)=a-b; z2(i)=c+d enddo end A sample program

46 subroutine xmult (x1,x2,y1,y2,z1,z2,n) real x1(n),x2(n),y1(n),y2(n),z1(n),z2(n) real a,b,c,d c$omp parallel do do i=1,n a=x1(i)*x2(i); b=y1(i)*y2(i) c=x1(i)*y2(i); d=x2(i)*y1(i) z1(i)=a-b; z2(i)=c+d enddo end A sample program

47 Run on Technion origin2000 Vector length = 1,000,000 Loop repeated 50 times Compiler optimization: low (-O1) Elapsed time, sec threads Compile No parallel Parallel Is this running in parallel? A sample program

48 Run on Technion origin2000 Vector length = 1,000,000 Loop repeated 50 times Compiler optimization: low (-O1) Elapsed time, sec threads Compile No parallel Parallel Is this running in parallel? WHY NOT? A sample program

49 c$omp parallel do do i=1,n a=x1(i)*x2(i); b=y1(i)*y2(i) c=x1(i)*y2(i); d=x2(i)*y1(i) z1(i)=a-b; z2(i)=c+d enddo Is this running in parallel? WHY NOT? Answer: by default, variables a,b,c,d are defined as SHARED A sample program

50 Elapsed time, sec threads Compile No parallel Parallel Solution: define a,b,c,d as PRIVATE: c$omp parallel do private(a,b,c,d) This is now running in parallel A sample program

51 5) Case Study

52 HPC in the Technion SGI Origin cpu (R10000) MHz Total memory GB PC cluster (linux redhat 6.1) 6 cpu (pentium II - 400MHz) Memory MB/cpu

53 Fluent test case -- Stability of a subsonic turbulent jet Source: Viktoria Suponitsky Faculty of Aerospace Engineering, Technion

54

55 Reading "Case25unstead.cas" quadrilateral cells, zone 1, binary D interior faces, zone 9, binary. 50 2D wall faces, zone 3, binary D pressure-inlet faces, zone 7, binary. 50 2D pressure-outlet faces, zone 5, binary. 50 2D pressure-outlet faces, zone 6, binary. 50 2D velocity-inlet faces, zone 2, binary D axis faces, zone 4, binary nodes, binary node flags, binary. Fluent test case 10 time steps, 20 iterations per time step

56

57

58 Host spawning Node 0 on machine "parix". ID Comm. Hostname O.S. PID Mach ID HW ID Name host net parix irix Fluent Host n7 smpi parix irix Fluent Node n6 smpi parix irix Fluent Node n5 smpi parix irix Fluent Node n4 smpi parix irix Fluent Node n3 smpi parix irix Fluent Node n2 smpi parix irix Fluent Node n1 smpi parix irix Fluent Node n0* smpi parix irix Fluent Node Fluent test case SMP command: fluent 2d -t8 -psmpi -g < inp

59 Fluent test case Cluster command: fluent 2d -cnf=clinux1,clinux2,clinux3,clinux4,clinux5,clinux6 -t6 –pnet -g < inp Node 0 spawning Node 5 on machine "clinux6". ID Comm. Hostname O.S. PID Mach ID HW ID Name n5 net clinux6 linux-ia Fluent Node n4 net clinux5 linux-ia Fluent Node n3 net clinux4 linux-ia Fluent Node n2 net clinux3 linux-ia Fluent Node n1 net clinux2 linux-ia Fluent Node host net clinux1 linux-ia Fluent Host n0* net clinux1 linux-ia Fluent Node

60

61

62 TOP500 (November 2, 2000)

63 TOP500 (November 2, 2000)