1. Combining the strengths of UMIST and The Victoria University of Manchester COMP60611 Fundamentals of Parallel and Distributed Systems Lecture 7 Scalability Analysis John Gurd, Graham Riley Centre for Novel Computing School of Computer Science University of Manchester

2. October 2010 2 Scalability What do we mean by scalability? –Scalability applies to an algorithm executing on a parallel computer, not simply to an algorithm! How does an algorithm behave for a fixed problem size as the number of processors used increases? –This is known as strong scaling. How does an algorithm behave as the problem size changes, in addition to changing the number of processors? A key insight is to look at how efficiency changes.

3. October 2010 3 Efficiency and Strong Scaling Typically, for a fixed problem size, N, the efficiency of an algorithm decreases as P increases. (Why?) –Overheads typically do not get smaller as P increases. They remain ‘fixed’ or, worse, they may grow with P (e.g. the number of communications may grow – in an all-to-all communication pattern). Recall that:

4. October 2010 4 Efficiency and Strong Scaling PO P is the total overhead in the system. T ref represents the true useful work in the algorithm. Because it tends to decrease with fixed N, at some point (absolute) efficiency E abs (i.e. how well each processor is being utilised) will drop below some acceptable threshold – say, 50%(?)

5. October 2010 5 Scalability No ‘real’ algorithm scales for all possible numbers of processors solving a fixed problem size on a ‘real’ computer. Even ‘embarrassingly’ parallel algorithms will have a limit on the number of processors they can use. –For example, at the point where, with a fixed N, eventually there is only one ‘element’ of some large data structure to be operated on by each processor. So we seek another approach to scalability which applies as both problem size N and the number of processors P change.

6. October 2010 6 Isoscaling and Isoefficiency A system is said to isoscale if, for a given algorithm and parallel computer, a specific level of efficiency can be maintained by changing the problem size, N, appropriately as P increases. Not all systems isoscale! –e.g. a binary tree-based vector reduction where N = P (see later). This approach is called scaled problem analysis. The function (of P ) describing how the problem size N must change as P increases in order to maintain a specified efficiency is known as the isoefficiency function. Isoscaling does not apply to all problems. –e.g. weather modelling, where increasing problem size (resolution) is eventually not an option –or image processing with a fixed number of pixels

7. October 2010 7 Weak Scaling An alternative approach is to keep the problem size per processor fixed as P increases (total problem size N thus increases linearly with P) and see how the efficiency is affected –This is known as weak scaling. Summary: strong scaling, weak scaling and isoscaling are three different approaches to understanding the scalability of parallel systems (algorithm + machine). We will look at an example shortly, but first we need a means of comparing the behaviour of functions, e.g. performance functions and efficiency functions, over their entire domains. These concepts will be explored further in lab exercise 2.

8. October 2010 8 Comparison Functions: Asymptotic Analysis Performance models are generally functions of problem size ( N ) and the number of processors ( P ). We need relatively easy ways to compare models (functions) as N and P vary: –Model A is ‘at most’ as fast or as big as model B; –Model A is ‘at least’ as fast or as big as model B; –Model A is ‘equal’ in performance/size to model B. We will see a similar need when comparing efficiencies and in considering scalability. These are all examples of comparison functions. We are often interested in asymptotic behaviour, i.e. the behaviour as some key parameter (e.g. N or P) increases towards infinity.

9. October 2010 9 Comparison Functions – Example From ‘Introduction to Parallel Computing’, Grama. Consider the three functions below: –Think of these functions as modelling the distance travelled by three cars from time t=0. One car has fixed speed and the others are accelerating – car C makes a standing start (zero initial speed).

10. October 2010 10 Graphically

11. October 2010 11 We can see that: –For t > 45, B(t) is always greater than A(t). –For t > 20, C(t) is always greater than B(t). –For t > 0, C(t) is always less than 1.25*B(t).

12. October 2010 12 Introducing ‘big-Oh’ Notation It is often useful to express a bound on the growth of a particular function in terms of a simpler function. For example, for t > 45, B(t) is always greater than A(t), we can express the relation between A(t) and B(t) using the Ο (Omicron or ‘big-Oh’) notation: This means that A(t) is “at most” B(t) beyond some value of t. Formally, given functions f(x), g(x), f(x) = O(g(x)) if there exist positive constants c and x 0 such that f(x) ≤ cg(x) for all x ≥ x 0 [Definition from JaJa not Grama! – more transparent].

13. October 2010 13 From this definition, we can see that: –A(t) = O(t) (“at most” or “of the order t”), –B(t) = O(t 2 ) (“at most” or “of the order t 2 ”), –Finally, C(t) = O(t 2 ), too. Informally, big-Oh can be used to identify the simplest function that bounds (above) a more complex function, as the parameter gets (asymptotically) bigger.

14. October 2010 14 Theta and Omega There are two other useful symbols: –Omega (Ω) meaning “at least”: –Theta ( Θ ) “equals” or “goes as”: For formal definitions, see, for example, ‘An Introduction to Parallel Algorithms’ by JaJa or ‘Highly Parallel Computing’ by Almasi and Gottlieb. Note that the definitions in Grama et al. are a little misleading!

15. October 2010 15 Performance Modelling – Example The following slides develop performance models for the example of a vector sum reduction. The models are then used to support basic scalability analysis of the resulting parallel systems. Consider two parallel systems: –First, a binary tree-based vector sum when the number of elements (N) is equal to the number of processors (P), N=P. –Second, a version for which N >> P. Develop performance models. –Compare the models. –Consider the resulting system scalability.

16. October 2010 16 Vector Sum Reduction (N = P) Assume that: –N = P, and –N is a power of 2. Propagate intermediate values through a binary tree of ‘adder’ nodes (processors): –Takes log 2 N steps with N processors (one of the processors is busy at every step, waiting for a message then doing an addition, the other processors have some idle time). Each step thus requires time for communication of a single word (cost t s +t w ) and a single addition (cost t c ):

17. October 2010 17 Vector Sum Speedup (N = P) Speedup: Speedup is poor, but monotonically increasing. –If N=128, S abs is ~18 (E abs = S abs /P = ~0.14, i.e. 14%), –If N=1024, S abs is ~100 (E abs = ~0.1, i.e. 10%), –If N=1M, S abs is ~ 52,000 (E abs = ~0.05, i.e. 5%), –If N=1G, S abs is ~ 35M (E abs = ~ 0.035, i.e. 3.5%).

18. October 2010 18 Vector Sum Scalability (N = P) Efficiency: But, N = P in this case, so: Strong scaling not ‘good’, as we have seen (E abs << 0.5). Efficiency is monotonically decreasing –Reaches 50% point, E abs = 0.5, when log 2 P = 2, i.e. when P = 4. This does not isoscale, either! –E abs gets smaller as P (hence N) increases and P and N must change together.

19. October 2010 19 When N >> P When N >> P, each processor can be allocated N/P elements (for simplicity, assume N is exactly divisible by P). Each processor sums its local elements in a first phase. A binary tree sum of size P is then performed to sum the P partial results. The performance model is:

20. October 2010 20 Strong Scalability (N >> P) Speedup: Strong scaling?? For a given problem size N (>> P), the (log 2 P/N) term is always ‘small’ so speedup will fall off ‘slowly’. P is, of course, limited by the value of N, but we are considering the case where N >> P.

21. October 2010 21 Isoscalability (N >> P) Efficiency: Now, we can always achieve a required efficiency on P processors by a suitable choice of N.

22. October 2010 22 Isoscalability (N >> P) For example, for 50% E abs, isoefficiency function is: Or, for E abs > 50%, isoefficiency function is: –As N gets larger for a given P, E abs gets closer to 1! –The ‘good’ parallel phase (N/P work) thus dominates the log 2 P phase as N gets larger, leading to relatively good (iso)scalability.

23. October 2010 23 Summary of Performance Modelling Performance modelling provides insight into the behaviour of parallel systems (parallel algorithms on parallel machines). Performance modelling allows the comparison of algorithms and gives insight into their potential scalability. Two main forms of scalability: –Strong scaling (fixed problem size N as P varies) There is always a limit to strong scaling for real parallel systems (i.e. a value of P at which efficiency falls below an acceptable limit). –Isoscaling (the ability to maintain a specified level of efficiency by changing N as P varies). Not all parallel systems isoscale. Asymptotic (‘big-Oh’) analysis makes comparison easier, but BEWARE the constants! Weak scaling is related to isoscaling – aim to maintain a fixed problem size per processor as P changes and look at the effect on efficiency.