Performance Measurement

A Quantitative Basis for Design n Parallel programming is an optimization problem. n Must take into account several factors: –execution time –scalability –efficiency

A Quantitative Basis for Design n Parallel programming is an optimization problem. n Must take into account several factors: n Also must take into account the costs: –memory requirements –implementation costs –maintenance costs etc.

A Quantitative Basis for Design n Parallel programming is an optimization problem. n Must take into account several factors: n Also must take into account the costs: n Mathematical performance models are used to assess these costs and predict performance.

Defining Performance n How do you define parallel performance? n What do you define it in terms of? n Consider –Distributed databases –Image processing pipeline –Nuclear weapons testbed

Metrics for Performance n Efficiency n Speedup n Scalability n Others …………..

Some Terms n s(n,p) = speedup for problem size n on p processors n o(n) = serial portion of computation n p(n) = parallel portion of computation n c(n,p) = time for communication n Speed1 = o(n) + p(n) n SpeedP = o(n) + p(n)/p + c(n,p)

Efficiency pT p T1T1 E  The fraction of time a processor spends doing useful work n What about when pT p < T 1 –Does cache make a processor work at 110%? o(n) + p(n) p * o(n) + p(n) + p * c(n,p) E =

Speedup SpeedP Speed S 1  What is Speed? What algorithm for Speed1? What is the work performed? How much work?

Speedup (More Detail) n s(n,p) = speedup for problem size n on p processors n o(n) = serial portion of computation n p(n) = parallel portion of computation n c(n,p) = time for communication n Speed1 = o(n) + p(n) n SpeedP = o(n) + p(n)/p + c(n,p) o(n) + p(n) o(n) + p(n)/p + c(n,p) Speedup =

More on Speedup Computation time decreases as we add processors but communication time increases

Two kinds of Speedup n Relative –Uses parallel algorithm on 1 processor –Most common –Useful for determining algorithm scalability n Absolute –Uses best known serial algorithm –Eliminates overheads in calculation. –Useful to express absolute performance n Story: Prime Number Generation

Amdahl's Law n Every algorithm has a sequential component. n Sequential component limits speedup Sequential Component Maximum Speedup = 1/s = s ¾ can be parallelized ¼ sequential Suppose each ¼ of the program takes 1 unit of time Speedup = 1 proc time / n proc time = 4/1 = 4

Amdahl’s Law o(n) + p(n) o(n) + p(n)/p + c(n,p) Speedup = o(n) + p(n) o(n) + p(n)/p <= s = o(n)/(o(n) + p(n)) = the inherently sequential percentage Speedup <= o(n) / s o(n) + o(n) ( 1/s -1)/p Speedup <= 1 s + ( 1 - s)/p

Amdahl's Law s Speedup

Speedup n Algorithm A –Serial execution time is 10 sec. –Parallel execution time is 2 sec. n Algorithm B –Serial execution time is 2 sec. –Parallel execution time is 1 sec. n What if I told you A = B?

Speedup n Conventional speedup is defined as the reduction in execution time. n Consider running a problem on a slow parallel computer and on a faster one. –Same serial component –Speedup will be lower on the faster computer.

Logic The art of thinking and reasoning in strict accordance with the limitations and incapacities of the human misunderstanding. The basis of logic is the syllogism, consisting of a major and minor premise and a conclusion.

Example n Major Premise: Sixty men can do a piece of work sixty times as quickly as one man. n Minor Premise: One man can dig a post- hole in sixty seconds. n Conclusion: Sixty men can dig a post-hole in one second.

Speedup and Amdahl's Law n Conventional speedup penalizes faster absolute speed. n Assumption that task size is constant as the computing power increases results in an exaggeration of task overhead. n Scaling the problem size reduces these distortion effects.

Solution n Gustafson introduced scaled speedup. n Scale the problem size as you increase the number of processors. n Calculated in two ways –Experimentally –Analytical models

Traditional Speedup (Strong Scaling) )( )( 1 NT NT Speedup P  T x (y) is time taken to solve problem of size y on x processors

Scaled Speedup (weak scaling) )( )( 1 PNT T Speedup P  Traditional speedup reduces the work done by each processor as we add processors Scaled speedup keeps the work constant on each processor as we add processors.

Scaled Speedup o(n) + p(n) o(n) + p(n)/p Speedup <= can be divided into two pieces serial and parallel s = o(n) / (o(n) + p(n)/p) and (1 – s) = p(n)/p / (o(n) + p(n)/p) now solve for o(n) and p(n) respectively o(n) = (o(n) + p(n)/p) * s p(n) = (o(n) + p(n)/p) * (1 – s) * p substituting these back into Speedup Equation yeilds Speedup <= s + (1 – s) * p and Speedup <= p + (1 – p) * s where s is fraction of time doing serial code = o(n) / t(n,k) t(n,k) is time of parallel program for size n on k processors Thus, max speedup with p < k processors is Speedup <= p + (1 – p) * s

Traditional Speedup ideal measured Number of Processors Speedup

Scaled Speedup ideal Number of Processors Speedup Small problem Medium problem Large Problem

Scaled Speedup vs Amdahl’s Law n Amdahl’s Law determines speedup by taking a serial computation and predicting how quickly it could be done in parallel n Scaled speedup begins with a parallel computation and estimates how much faster the parallel computation is than the same computation on a serial processor n strong scaling is defined as how the solution time varies with the number of processors for a fixed total problem size. n weak scaling is defined as how the solution time varies with the number of processors for a fixed problem size per processor.

Determining Scaled Speedup n Time problem size n on 1 processor n Time problem size 2n on 2 processors n Time problem size 2n on 1 processor n Time problem size 4n on 4 processors n Time problem size 4n on 1 processor n etc. n Plot the curve

Performance Measurement n There is not a perfect way to measure and report performance. n Wall clock time seems to be the best. n But how much work do you do? n Best Bet: –Develop a model that fits experimental results.

A Parallel Programming Model n Goal: Define an equation that predicts execution time as a function of –Problem size –Number of processors –Number of tasks –Etc.,....),(PNfT 

A Parallel Programming Model n Execution time can be broken up into –Computing –Communicating –Idling  idlecommcomp TTTT

Computation Time n Normally depends on problem size n Also depends on machine characteristics –Processor speed –Memory system –Etc. n Often, experimentally obtained

Communication Time n The amount of time spent sending & receiving messages n Most often is calculated as –Cost of sending a single message * #messages n Single message cost –T = startuptime + time_to_send_one_word * #words

Idle Time n Difficult to determine n This is often the time waiting for a message to be sent to you. n Can be avoided by overlapping communication and computation.

Finite Difference Example n Finite Difference Code n 512 x 512 x 5 Elements n Nine-point stencil n Row-wise decomposition –Each processor gets n/p*n*z elements n 16 IBM RS6000 workstations n Connected via Ethernet z n x

Finite Difference Model n Execution Time (per iteration) –ExTime = (Tcomp + Tcomm)/P n Communication Time (per iteration) –Tcomm = 2 (lat + 2*n*z*bw) n Computation Time –Estimate using some sample code

Estimated Performance

Finite Difference Example

What was wrong? n Ethernet –Shared bus n Change the computation of Tcomm –Reduce the bandwith –Scale the message volume by the number of processors sending concurrently. –Tcomm = 2 (lat + 2*n*z*bw * P/2)

Finite Difference Example

Using analytical models n Examine the control flow of the algorithm n Find a general algebraic form for the complexity (execution time). n Fit the curve with experimental data. n If the fit is poor, find the missing terms and repeat. n Calculate the scaled speedup using formula.

Example n Serial Time = N seconds n Parallel Time = N/P + 5P seconds n Let N/P = 128 n Scaled Speedup for 4 processors is:  )4(5)4/)128(4(124 ))128(4(122    )( )( 1  PNC C P

Performance Evaluation n Identify the data n Design the experiments to obtain the data n Report data

Performance Evaluation n Identify the data –Execution time –Be sure to examine a range of data points n Design the experiments to obtain the data n Report data

Performance Evaluation n Identify the data n Design the experiments to obtain the data –Make sure the experiment measures what you intend to measure. –Remember: Execution time is max time taken. –Repeat your experiments many times –Validate data by designing a model n Report data

Performance Evaluation n Identify the data n Design the experiments to obtain the data n Report data –Report all information that affects execution –Results should be separate from Conclusions –Present the data in an easily understandable format.