Introduction to Scientific Computing on Linux Clusters Doug Sondak Linux Clusters and Tiled Display Walls July 30 – August 1, 2002

Outline Why Clusters? Parallelization example - Game of Life performance metrics Ways to Fool the Masses summary Doug Sondak Linux Clusters and Tiled Display Walls July 30 – August 1, 2002

Why Clusters? Scientific computing has traditionionally been performed on fast, specialized machines Buzzword - Commodity Computing –clustering cheap, off-the-shelf processors –can achieve good performance at a low cost if the applications scale well Doug Sondak Linux Clusters and Tiled Display Walls July 30 – August 1, 2002

Clusters (2) 102 clusters in current Top 500 list Resonable parallel efficiency is the key generally use message passing, even if there are shared-memory CPU’s in each box Doug Sondak Linux Clusters and Tiled Display Walls July 30 – August 1, 2002

Compilers Linux Fortran compilers (F90/95) –available from many vendors, e.g., Absoft, Compaq, Intel, Lahey, NAG, Portland Group, Salford –g77 is free, but is restricted to Fortran 77, relatively slow Doug Sondak Linux Clusters and Tiled Display Walls July 30 – August 1, 2002

Compilers (2) Intel offers free unsupported Fortran compiler for non-commercial purposes –full F95 –OpenMP compilers/f60l/noncom.htm Doug Sondak Linux Clusters and Tiled Display Walls July 30 – August 1, 2002

Compilers (3)

Compilers (4) Linux C/C++ compilers –gcc/g++ seems to be the standard, usually described as a good compiler –also available from vendors, e.g., Compaq, Intel, Portland Group Doug Sondak Linux Clusters and Tiled Display Walls July 30 – August 1, 2002

Parallelization of Scientific Codes Doug Sondak Linux Clusters and Tiled Display Walls July 30 – August 1, 2002

Domain Decomposition Typically perform operations on arrays –e.g., setting up and solving system of equations domain decomposition –arrays are broken into chunks, and each chunk is handled by a separate processor –processors operate simultaneously on their own chunks of the array Doug Sondak Linux Clusters and Tiled Display Walls July 30 – August 1, 2002

Other Methods Parallelzation also possible without domain decomposition –less common –e.g., process one set of inputs while reading another set of inputs from a file Doug Sondak Linux Clusters and Tiled Display Walls July 30 – August 1, 2002

Embarrassingly Parallel if operations are completely independent of one another, this is called embarrassingly parallel –e.g., initializing an array –some Monte Carlo simulations –not usually the case Doug Sondak Linux Clusters and Tiled Display Walls July 30 – August 1, 2002

Game of Life Early simple cellular automata –created by John Conway 2-D grid of cells –each has one of 2 states (“alive” or “dead”) –cells are initialized with some distribution of alive and dead states Doug Sondak Linux Clusters and Tiled Display Walls July 30 – August 1, 2002

Game of Life (2) at each time step states are modified based on states of adjacent cells (including diagonals) Rules of the game: –3 alive neighbors - alive –2 alive neighbors - no change –other - dead Doug Sondak Linux Clusters and Tiled Display Walls July 30 – August 1, 2002

Game of Life (3) Doug Sondak Linux Clusters and Tiled Display Walls July 30 – August 1, 2002

Game of Life (4) Parallelize on 2 processors –assign block of columns to each processor Problem - What happens at split?

Game of Life (5) Solution - Overlap cells Each time step, pass overlap data processor to processor

Message Passing Largest bottleneck to good parallel efficiency is usually message passing –much slower than number crunching set up your algorithm to minimize message passing minimize surface-to-volume ratio of subdomains Doug Sondak Linux Clusters and Tiled Display Walls July 30 – August 1, 2002

Domain Decomp. For this domain: To run on 2 processors, decompose like this: Not like this: Doug Sondak Linux Clusters and Tiled Display Walls July 30 – August 1, 2002

How to Pass Msgs. MPI is the recommended method –PVM may also be used MPICH –most common –free download others also avalable, e.g., LAM Doug Sondak Linux Clusters and Tiled Display Walls July 30 – August 1, 2002

How to Pass Msgs. some MPI tutorials –Boston University –NCSA Doug Sondak Linux Clusters and Tiled Display Walls July 30 – August 1, 2002

Performance Doug Sondak Linux Clusters and Tiled Display Walls July 30 – August 1, 2002

Code Timing How well has code been parallelized? CPU time vs. wallclock time –both are seen in literature –I prefer wallclock only for dedicated processors CPU time doesn’t account for load imbalance unix time command Fortran system_clock subroutine MPI_Wtime Doug Sondak Linux Clusters and Tiled Display Walls July 30 – August 1, 2002

Parallel Speedup quantify how well we have parallelized our code S n = parallel speedup n = number of processors T 1 = time on 1 processor T n = time on n processors

Parallel Speedup (2)

Parallel Efficiency  n = parallel efficiency T 1 = time on 1 processor T n = time on n processors n = number of processors Doug Sondak Linux Clusters and Tiled Display Walls July 30 – August 1, 2002

Parallel Efficiency (2)

Parallel Efficiency (3) What is a “reasonable” level of parallel efficiency? Depends on –how much CPU time you have available –when the paper is due can think of (1-  as “wasted” CPU time my personal rule of thumb ~60% Doug Sondak Linux Clusters and Tiled Display Walls July 30 – August 1, 2002

Parallel Efficiency (4) Superlinear speedup –parallel efficiency > 1.0 –sometimes quoted in the literature –generally attributed to cache issues subdomains fit entirely in cache, entire domain does not this is very problem dependent be suspicious! Doug Sondak Linux Clusters and Tiled Display Walls July 30 – August 1, 2002

Amdahl’s Law Always some operations which are performed serially want a large fraction of code to execute in parallel Doug Sondak Linux Clusters and Tiled Display Walls July 30 – August 1, 2002

Amdahl’s Law (2) Let fraction of code that executes serially be denoted s Let fraction of code that executes in parallel be denoted p Doug Sondak Linux Clusters and Tiled Display Walls July 30 – August 1, 2002

Amdahl’s Law (3) Noting that p = (1-s) The parallel speedup is Amdahl’s Law Doug Sondak Linux Clusters and Tiled Display Walls July 30 – August 1, 2002

Amdahl’s Law (4) The parallel efficiency is Alternate version of Amdahl’s Law Doug Sondak Linux Clusters and Tiled Display Walls July 30 – August 1, 2002

Amdahl’s Law (5)

Amdahl’s Law (6) Should we despair? –No! –bigger machines solve bigger problems smaller value of s if you want to run on a large number of processors, try to minimize s Doug Sondak Linux Clusters and Tiled Display Walls July 30 – August 1, 2002

Ways to Fool the Masses full title: “Twelve Ways to Fool the Masses When Giving Performance Results on Parallel Computers” Created by David Bailey of NASA Ames in 1991 following is selection of “ways,” some paraphrased Doug Sondak Linux Clusters and Tiled Display Walls July 30 – August 1, 2002

Ways to Fool (2) Scale problem size with number of processors Project results linearly –2 proc, 1 hr proc., 1 sec. Present performance of kernel, represent as performance of application Doug Sondak Linux Clusters and Tiled Display Walls July 30 – August 1, 2002

Ways to Fool (3) Compare with old code on obsolete system Quote MFLOPS based on parallel implementation, not best serial implementation –increase no. operations rather than decreasing time Doug Sondak Linux Clusters and Tiled Display Walls July 30 – August 1, 2002

Ways to Fool (4) Quote parallel speedup making sure single-processor version is slow Mutilate the algorithm used in the parallel implementation to match the architecture –explicit vs. implicit PDE solvers Measure parallel times on dedicated system, serial times in busy environment Doug Sondak Linux Clusters and Tiled Display Walls July 30 – August 1, 2002

Ways to Fool (5) If all else fails, show pretty pictures and animated videos, and don’t talk about performance. Doug Sondak Linux Clusters and Tiled Display Walls July 30 – August 1, 2002

Summary Clusters are viable platforms for relatively low-cost scientific computing parallel considerations similar to other platforms MPI is a free, effective message passing API careful with performance timings Doug Sondak Linux Clusters and Tiled Display Walls July 30 – August 1, 2002