1. Analytical Modeling Of Parallel Programs
Dagoberto A.R.Justo
PPGMAp
UFRGS
12/3/2018
intro

2. Introduction
Problem: How can we model the behavior of a parallel program in order to predict its execution time, using the size of the problem, the number of nodes/processors, the communication network parameters ts and tw?
Clearly, we must consider the algorithm and the architecture
Issues
A serial program measures total execution time and typically depends mainly the size of its input
A parallel program has several new issues
The execution time of the program
The speedup relative to the algorithm running serially
However, is it speedup compared with the best serial algorithm or is it speed relative to the serialization of the parallel algorithm used?

3. Outline Of This Topic Sources of overhead in a parallel program
Performance metrics for parallel systems
The effect of granularity on performance
Scalability of parallel systems
Minimum execution time and minimum cost-optimal execution time
Asymptotic analysis of parallel programs
Other scalability metrics

4. Typical Parallel Execution Profile
Essential/Excess Computation
Interprocessor Communication
Idling

5. Sources Of Overhead
A profile illustrates the kinds of activities in a program execution
Essential Computation
Same computations in a serial program
Some of these are overheads (time spent not computing directly what is needed) such as:
Interprocess interaction
Idling
Excess computation
These are all activities the serial program does not perform
An efficient parallel program attempts to minimize these overheads to zero but of course this is not always possible

6. Interprocess Interaction and Idling
Usually the most significant overhead
Sometimes reduced by performing redundant computation
Idling
Caused by:
load imbalance
synchronization (waiting for collaborating processes to reach the synchronization point
serial computation that cannot be avoided

7. Excess Computation
The fastest known serial algorithm may not be easy to parallelize, especially for a large number of processes
A different serial algorithm may be necessary, which is not optimal when implemented serially
Such programs may perform more work
It may be faster for all processes to compute common intermediate results than to compute them once and broadcast them to all processes that need them
What we are thus faced with is how to measure the performance of a parallel program to tell whether the parallel program is worth using
How does performance compare with a serial implementation?
How does performance scale with adding more processes?
How does performance scale with increasing the problem size?

8. Performance Metrics Serial and Parallel runtime
TS : wall-clock time from start time to completion time (on the same processor as the parallel program)
TP : wall-clock time from the time the parallel processing starts to the end time for the last process to complete
TTP = pTP : Parallel Cost
Total parallel overhead
TO = pTP – TS
Speedup S (How well is the parallel program performing?)
The ratio of the parallel execution time with the execution of the best serial program
S = TS / TP
S is expect to be near p
S=p is called linear speedup
S<p most often. Generally, 80% of p is very good
S>p, it is called superlinear speedup

9. Speedup and Efficiency
Speedup S (How well is the parallel program performing?)
The ratio of the parallel execution time with the execution of the best serial program
S = TS / TP
S is expect to be near p
S=p is called linear speedup
S<p most often. Generally, 80% of p is very good
S>p, it is called superlinear speedup
Efficiency
The ratio of the speedup to the number of processors used
E = S/p
E=1 is ideal
E>0.8 is generally good

10. Example: Summing n Numbers in Parallel With n Processors
Using input decomposition, place each number on a different processor
The sum is performed in log n phases
Assume n is a power of 2 and assume the processors are arranged in a linear array numbered from 0
Phase 1:
Odd numbered processors send xi to the left processor
Even processor adds the two numbers Si=xi+xi+1
Phase 2:
Every second processor sends Si to the processor two positions left
This processor adds the partial sum it has to the received partial sum
Continuing for log n phases, process 0 ends up with the sum

11. A Picture Of The Parallel Sum Process15 14 13 12 11 10 9 8 7 6 5 4 3 2 1
(a) Initial data distribution and the first communication phase
(b) Second communication step 3 2 1 7 6 5 4 11 10 9 8 15 14 13 12
(c) Third communication step 3 2 1 7 6 5 4 11 10 9 8 15 14 13 12
(d) Fourth communication step 3 2 1 7 6 5 4 11 10 9 8 15 14 13 12
(e) Accumulation of the sum at processor 0 after the final communication 3 2 1 7 6 5 4 11 10 9 8 15 14 13 12

12. Modelling execution time
tC : the time to add two numbers
Serial time TS = (n-1) tC = (n)
Parallel time TP: in each of the log n phases,
computation time: tC
communication time: (ts + tw)
TP = tC log n + (ts + tw) log n = (log n)
The speedup is:
S = TS / TP = (tC n)/(tC log n + (ts + tw) log n)
= (tC /(tC + ts + tw)(n log n) = (n / log n)
The overhead (p = n) is:
TO = pTp – TS = n((ts + tw) log n + tC(log n – 1)) + tC
= (n log n)
The overhead is large -- why?
Thus, this parallel algorithm does considerably more total work than the serial algorithm

13. Misleading Speedups Faster than one deserves:
Consider a parallel implementation of bubble sort, called the odd-even sort, of 105 elements on 4 processors that completes in 40 seconds
The serial version of bubble sort takes 150 seconds
But serial quick-sort on the same set of elements takes 30 seconds
The misleading speedup is: TS / TP = 150/40 = 3.75
The correct speedup is: TS / TP = 30/40 = 0.75
The required comparison is with quick-sort, the fastest serial algorithm and the particular parallel sort algorithm
But the parallel code can run faster than expected
For example:
Because of caches effects
Because of exploratory decomposition

14. Misleading Speedups Continued
Cache effects
Consider a problem of size W words(large but small enough to fit in the memory of a single processor)
Suppose on a single processor an 80% cache hit ratio is observed for this problem
Suppose cache latency to CPU is 2 ns
Suppose the DRAM latency is 100 ns
The access time per data item on average is then 0.8* *100 = 21.6 ns
Assume the program performs at 1 floating operation per memory access
Then the performance is 1000/21.6 Mflops or 46.3 MFLOPS

15. Cache Effects Continued
Consider solving the same problem on two processors by data decomposition of all data so that two sub-problems of size W/2 are solved
The amount of data per processor is less so that we might expect the cache hit ratio to be higher, say 90%
Of the remaining 10% of the accesses, assume 8% are from each processor's memory and 2% are from the other processor's memory
Suppose the latencies are 2 ns, 100 ns, and 400 ns respectively (400 ns for access to the DRAM of the other processors memory)
The access time per data item on average is then 0.9* * *400 = 17.8 ns per processor
Assume, as before, the program performs 1 floating operation per memory access
Then the performance is 1000/17.8 Mflops or 56.2 MFLOPS per processor for a total rate of MFLOPS (2 processors used)
Speedup is then 112.6/46.3 or 2.43, faster than we deserve

16. Faster Solutions Via Exploratory Decomposition
Suppose the search tree looks like the following graph with the solution at the rightmost node
Serial search is depth first, left first
Parallel search is 2 processors
Both depth first, left first algorithms
The serial algorithm finds the solution in 14 steps
The parallel algorithm takes 5 steps
The speedup is 14/5 = 2.8 > 2 -- superlinear speedup results
Processing element 0 begins searching here
Processing element 1 begins searching here
Solution

17. Efficiency Efficiency E is defined as:
S: speedup and p: number of processors
It in essence measures the average utilization of all processors
E=1: Ideal speedup
E>1: Superlinear speedup
E>0.8: Very good speedups in practice
Example: the parallel summation process of n numbers with n processors
Notice the speedup decreases with the size of the problem
That is, there is no point in using a large number of processors for computations with speedups like this

18. A More Satisfactory Example
Edge detection of an n  n pixel image
Applies a 3  3 multiplicitive template with summation (convolution) to each pixel
The serial computation time TS is 9 tc n2 where tc is the average time for a multiply-add arithmetic operation
The parallel algorithm using p processors divides the image into p column slices with n/p columns per slice
The computation for the pixels on each processor is local except for the left and right edges which require the pixel values from the edge column of neighboring processes
Thus, the parallel time is: TP = 9 tc n2 / p + 2(ts + n tw)
The speedup and efficiency are:
E increases with increasing n
E decreases with increasing p

19. Convolution Of A Pixel Image-2 -1 1 2 2 -1 -2 1
Pixel Image
Two different convolution templates for detecting edges P0 P1 P2 P3
Image partitioning amongst processes and data sharing for process 1

20. Parallel Cost And Cost-Optimality
Defined as the number of processors p times the parallel computing time Tp
The term cost-optimal refers to a parallel system that has the following property:
The parallel system has a cost that has the same asymptotic growth as the fastest serial algorithm to solve the same problem; that is, has an efficiency of (1)
Example: Adding numbers in parallel is (n log n)
Adding the numbers serially is (n)
This parallel addition algorithm is NOT cost-optimal
Questions:
Can this algorithm or any algorithm be made cost optimal by increasing the granularity? That is, decrease p and increase the work of each proc?
Sometimes, but in general no. See the examples on the next few slides
Is there a cost-optimal parallel algorithm for summing n numbers?
Yes. But the cost-optimal algorithm is different from the algorithms so far but uses the idea of increased granularity

21. The Importance Of The Cost-Optimal Metric
Consider a sorting algorithm to sort n elements using n processors that is not cost optimal. What does this mean in practice?
Scalability performance is very poor. Let's see why
Suppose this algorithm takes (log n)2 time to sort the list
The best serial time is known to be n log n
The parallel algorithm has a speedup and efficiency of n/log n and 1/log n respectively.
The parallel algorithm is an improvement but not cost optimal because speedup is not (1)
Now, consider increasing the granularity by decreasing the number of processors from n to p < n.
The new parallel algorithm will take no more than n(log n)2/p time
Its speedup is (nlog n)/(n(log n)2/p) = p/log n and its efficiency is the same
For example, with 32 processors and n = 1024 and n = 106, the speedups are 3.2 and 1.6 respectively -- very poor scalability

22. Two Different Parallel Summation Algorithms
The second sum algorithm uses the idea of increasing the granularity of the previous parallel algorithm, also called scaling down (see the next few slides for n = 16 and p = 4)
Instead of using n processors, use p < n and place more of the addends, namely n/p, on each processor
Add the corresponding numbers on each processor as with the first algorithm (communication) until all partial sums are on one processor
Add the partial sums in pairs using the approach of the first parallel approach but on one processor
The third sum algorithm is a modification of the above
Add up all the numbers on each processor first using the usual serial algorithm and then apply the first parallel algorithm to p addends on p processors

23. Second Algorithm (n=16, p=4) -- First Step2 3 5 4 6 7 9 8 10 11 13 12 14 15
Initial distribution and first communication step P0 P1 P2 P3 5 4 6 7 9 8 10 11 13 12 14 15
Substep 2 distribution and next communication step
01
23 P0 P1 P2 P3
Data distribution after last substep
01
23
45
67
89
1011
1213
1415 P0 P1 P2 P3 9 8 10 11 13 12 14 15
Substep 3 distribution and next communication step
01
23
45
67 P0 P1 P2 P3 13 12 14 15
Substep 4 distribution and next communication step
01
23
45
67
89
1011

24. Second Algorithm (n=16, p=4) -- Second Step
Initial distribution before first substep and communication
01
23
45
67
89
1011
1213
1415 P0 P1 P2 P3
Substep 2 distribution before second substep and communication
03
45
67
89
1011
1213
1415 P0 P1 P2 P3
Substep 3 distribution before third substep and communication
03
47
89
1011
1213
1415 P0 P1 P2 P3
Substep 4 distribution before fourth substep and communication
03
47
811
1213
1415 P0 P1 P2 P3
Final distribution after last substep
03
47
811
1215

25. Second Algorithm (n=16, p=4) -- 3rd & 4th Step
Data distribution before first substep and grouping of operations
03
47
811
1215 P0 P1 P2 P3
Data distribution before second substep and grouping of operations
07
811
1215 P0 P1 P2 P3
Final distribution of data after last substep
07
815 P0 P1 P2 P3
Data distribution before first substep and grouping of operations
07
815 P0 P1 P2 P3
Final result
015

26. Third Algorithm (n=16, p=4) -- A Cost Optimal Algorithm8 12 5 1 9 13 6 2 10 14 7 3 11 15
Initial data distribution and grouping of operations P0 P1 P2 P3
Data distribution after first step and first communication step
03
811
47
1215 P0 P1 P2 P3
Data distribution after second step and second communication step
07
815 P0 P1 P2 P3
Final result after the last step
015

27. The Effect Of Granularity
Increasing the granularity of a cost-optimal algorithm maintains cost optimality
Suppose we have a cost-optimal algorithm using p processors and we increase its granularity by reducing the number of processors to q and increase the work per processor
The work per processor would increase by a factor p/q
The communication per processor should also grow by no more than a factor p/q provided the mapping is done carefully
Thus, the parallel time would increase by a factor p/q
Then parallel cost of the new algorithm would be qTp_new which is q(p/q)Tp_old = pTp_old
Thus, granularity has not changed the cost of a cost-optimal algorithm
Thus, to produce cost-optimal parallel codes from non-cost optimal parallel code, you may have to do more than increase granularity but it may help
The two new sum algorithms illustrate this point
The above proof does NOT show that increasing granularity preserves cost

28. Analysis Of The Sum Algorithms
Serial sum algorithm costs: (n)
The first parallel algorithm costs :(n log n)
The second parallel algorithm is:
n/p steps with log p sub-steps, taking ((n/p)log p)
Then, we add n/p numbers, taking (n/p)
Total time is: ((n/p)log p)
Cost is: p ((n/p)log p) = (n log p)
This is asymptotically higher than the serial algorithm and still is not cost optimal
The third parallel algorithm is:
The first step is n/p additions
The second step consists of log p sub-steps consisting of an addition and communication
Thus, the time is (n/p + log p) and the cost is (n + p log p)
As long as p is not too large, like n = (p log p) , the cost is (n), which means this algorithm is cost optimal

29. Scalability Of Parallel Systems
We typically develop parallel programs from small test cases
It is very difficult to predict scalability (performance for large problems) from small test cases, unless you have done the analysis first
We now study some tools to help in the prediction process
See the FFT case study based on observation of performance in the small and its relation to performance for large sized problems
Topics:
Scaling characteristics of parallel programs
Isoefficiency metric of scalability
Problem size
Isoefficiency function
Cost optimality and the isoefficiency function
A lower bound on the isoefficiency function
Degree of concurrency and the isoefficiency function

30. FFT Case Study Three algorithms for performing FFTs
Algorithms described in detail in Chapter 13
Binary exchange
2-D transpose
3-D transpose
Speedup data given for 64 processors, for the FFT size n varying from 1 to 18K elements
For small n (up to 7000 or so), 3-D transpose and binary exchange are best -- a lot of testing to see this
For large n, 2-D transpose outperforms the others and continues faster for n > Can you believe this remains true though for even larger n?
Not unless you have done the analysis to support the conjectured asymptotic behavior

31. Scaling Characteristics Of Parallel Programs
The efficiency is: E = S/p = TS/(pTP)
Using the expression involving TO (overhead -- slide 9)
Unfortunately, the overhead is at least linear in p unless the algorithm is completely parallel
Say, the parallel algorithm has a serial time of Tserial
Then, all but one of the processors is idle during the time one processor is performing the serial computation
Thus, the overhead is at least (p–1)Tserial
Therefore, the efficiency is bounded above by:

32. Scaling Characteristics Continued
From this expression for E (previous slide)
The efficiency E decreases with the number of processors for a given problem size
The efficiency E increases with larger problems (TS increases)
Consider the cost optimal summation algorithm
For this algorithm (assume unit time for addition and communication):
n/p is the time for adding n/p items
2log p is the time for the addition and
communication of phase 2
See the disappointing results for large p
on the next slide
See how an efficiency level of, say, 80%
can be maintained by increasing n for
each p

33. Speedup Curves
Plots of S = n/(n/p + 2 log p) for the cost-optimal addition parallel algorithm for changing p and n

34. Efficiency Tables -- A Table Of Values Of E, For Different p and n64
1.0
0.80
0.57
0.33
0.17
192
0.92
0.60
0.38
320
0.95
0.87
0.71
0.50
512
0.97
0.91
0.62
The function of p representing work (n) that keeps the efficiency fixed (with increasing p) is the isoefficiency function -- the bold entries are 80% efficiency

35. Scalability
Overhead varies with the serial time (the amount of work) and the number of processors
Clearly, overhead (communication) typically increases with the number of processors
It often increases with the amount of work to be done, usually indicated by the sequential time TS
However, as the problem size increases, overhead usually increases sublinearly as a percentage of the work
This means that the efficiency increases with the problem size even when the number of processors is fixed
For example, look at the columns of the last table
Also, an efficiency level can be maintained by increasing both the number of processors p and the amount of work
A parallel system that is able to maintain a specific efficiency in this manner is called a scalable parallel system
The scalability is a measure of a system's capacity to increase speedup in proportion to the number of processing elements

36. Scalability And Cost Optimality
Recall: Cost optimal algorithms have an efficiency of (1)
Scalable parallel systems can be always made cost-optimal
Cost-optimal algorithms are scalable
Example: the cost-optimal algorithm for adding n numbers
Its efficiency is: E = 1/(1+2p(log p)/n)
Setting E equal to a constant, say K, means that n and p must vary as n = 2(K/(1–K))p log p
Thus, for any p, the size of n can be selected to maintain efficiency K
For example, for K = 80%, then, if 32 processors are used, then the problem size of 1480 must be used (Recall: this efficiency formula was based on adding 2 numbers and communicating 1 number took unit time -- not a very realistic assumption with current hardware)

37. Isoefficiency Metric Of Scalability
Two observations:
Efficiency always decreases with increase in the number of processors, approaching 0 for a large number of processors
Efficiency often increases with the amount of work or size of the problem, approaching 1
It sometimes can then decrease for large work sizes (run out of memory, for example)
A scalable system is one for which the efficiency can be made constant by increasing both the work and the number of processors
The rate at which, for a fixed efficiency, the work or the problem size must increase with respect to the number of processors to maintain a fixed efficiency is called the degree of scalability of the parallel system
This definition depends upon a clear definition of problem size
Once problem size is defined, then the function determining problem size for varying number of processors and fixed efficiency is called the isoefficiency function

38. Problem Size
Define problem size as the number of basic operations, such as arithmetic operations, data stores and loads, etc.
It needs to be a measure that if the problem size is doubled, the computation time is doubled
Measures such as the size (order) of a matrix are misleading
Double the size of a matrix causes a computation dominated by matrix-matrix multiplication to increase by a factor of 8
Double the size of a matrix causes a computation dominated by matrix-vector multiplication to increase by a factor of 4
Double the size of vectors causes a computation dominated by vector dot product to increase by a factor of 2
In the formulas that follows, they assume the basic arithmetic operation takes 1 unit of time
Thus, the problem size W is the same as the serial time TS of the fastest known serial algorithm

39. The Isoefficiency Function -- The General Case
Parallel execution time TP is a function of:
the problem size W,
overhead function TO, and
the number number p of processors
Let's fix the efficiency to E and solve for W in terms of p and the fixed efficiency
Let K = E/(1–E) and we get:

40. Development Continued
Now solving for W gives:
In the above equation, K is a constant
For each choice of W, we solve for p
This can usually be done algebraically
Or, for each choice of p, we solve for W, possibly a non-linear equation, that has to be solved numerically or approximated somehow
The resulting function of W in terms of p is called the isoefficiency function

41. Analysis Of The Isoefficiency Function
Suppose this function has the property that:
Small changes in p give small changes in W
Then, the system is highly scalable
Small changes in p result in large changes in W
Then, the system is not very scalable
The isoefficiency function may be difficult to find
You can solve the above equation only with specific finite values and cannot solve it in general
The isoefficiency function may not exist
Such systems are not scalable

42. Two Examples Consider the cost-optimal addition algorithm
The overhead function is 2p log p
Thus, the isoefficiency function (from slide 40) is: 2Kp log p
That is, if the number of processors is doubled, the size of the problem must be increased by a factor of 2 (1+ log p)/log p
In this example, the overhead is only a function of the number of processors and not of the work W
This is unusual
Suppose we had a parallel problem with the following overhead function
TO = p3/2 + p3/4W3/4
The equation to solve for W in terms of p is: W = K p3/2 + Kp3/4W3/4
For fixed K, this is a polynomial of degree 4 in W with multiple roots
Although this case can be solved analytically, in general such equations cannot be solved analytically
See the next slide for an appropriate approximate solution that provides the relevant asymptotic growth

43. Second Example Continued -- Finding An Approximate Solution
The function W is the sum of two positive terms that increase with increasing p
Let's take each term separately, find the growth rate consistent with each, and take the maximum growth rate as the asymptotic growth rate
For just the first term, W = K p3/2 and W = (p3/2)
For just the second term, W = K p3/4 W3/4
Solving for W gives W = (p3)
The faster growth rate is (p3)
Thus, for this parallel system, the work must grow like p3 in order to maintain a constant efficiency
Thus the isoefficiency function is (p3)

44. Your Project
Perform the corresponding analysis for your implementation and determine the isoefficiency function, if it exists
It is the analysis that allows you to test the performance of your parallel code on a few test cases and then allows you to predict the performance in the large
The test cases have to be selected so that your results do not reflect initial condition or small case effects

45. Cost-Optimality And The Isoefficiency Function
Consider a cost-optimal parallel algorithm
An algorithm is cost-optimal iff the efficiency is (1)
That is, E = S/p = W/(pTP)  pTP = (W)
But pTP = W + TO(W,p) so that TO(W, p) = (W)
Thus, W = (TO(W, p))
Thus, an algorithm is cost-optimal iff its overhead does not exceed its problem size asymptotically
In addition, if there exists an isoefficiency function f(p), then the relation W = (f(p)) must be satisfied in order to ensure the cost-optimality of the parallel system

46. A Lower Bound On the Isoefficiency Function
We desire the smallest isoefficiency function (recall degrees of scalability -- eg. slides 37, 41)
How small can the isoefficiency function be?
The smallest possible function is (p)
Argument:
For W work, the maximum number of processors is W because the processors in excess of the number W will be idle -- no work to do
For a problem size growing slower than (p), if the number of processors grows like order p, then eventually there are more processors than work
Thus, such a system will not be scalable
Thus, the ideal function is (p)
This is hard to achieve -- the cost optimal add algorithm has an isoefficiency function of (p log p)

47. The Degree Of Concurrency And The Isoefficiency Function
The degree of concurrency C(W) is the maximum number of tasks that can be executed concurrently for a computation with work W
This degree of concurrency thus must limit the isoefficiency function. That is,
For a problem size W with a degree of concurrency C(W), at most C(W) processors can be used effectively
Thus, the isoefficiency function can be no better than (C(W))

48. An Example
Consider solving Ax = b (a linear system of size n) via Gaussian elimination
The total computation is (n3) time
We eliminate one variable at a time in a serial fashion, taking (n2) time
Thus, at most n2 processing elements can be used
Thus, the degree of concurrency is (W2/3)
Thus, from p = (W2/3), W = (p3/2)
which is the isoefficiency function
Thus, this algorithm cannot reach the ideal or optimal isoefficiency function (p)

49. Minimum Execution Time and Minimum Cost-Optimal Execution Time
Parallel processing time TP often decreases as the number p of processors increases until either TP approaches a minimum asymptotically or increases
The question now is what is that minimum and is it useful to know?
We can find this minimum by taking the derivative of the function of parallel time with respect to the number of processors, setting this derivative to 0, and solving for the p that satisfies this derivative equation
Let p0 be the value of p for which the minimum is attained
Let Tpmin be the minimum parallel time
Let's do this for the parallel summation system we are working with

50. An Example Consider the cost-optimal algorithm for adding n numbers
Its parallel time is (slide 32): TP = n/p + 2 log p
The equation from the first derivative set to 0 is: –n/p2 + 2/p = 0
The solution is: p0 = p = n/2
The minimum parallel time is: Tpmin = 2 log n
The processor product time (work) is p Tpmin = (n log n)
This is larger than the serial time which is (n)
Thus, for this minimum time, the problem is not being solved cost-optimally (larger than the serial time)

51. The Cost-Optimal Minimum Time Tpcost_opt
Let's characterize and find the minimum for the computation performed cost-optimally
Cost-optimality can be related to the isoefficiency function and vice versa (see slide 45)
If the isoefficiency function of a parallel system is (f(p)), then the problem of size W can solved cost-optimally iff W = (f(p))
That is, a cost-optimal solution requires p = (f- –1(p))
The parallel run-time is: TP = (W/p), (because pTP = (W))
Thus, a lower bound on the parallel runtime for solving a problem of size W cost-optimally is:
Tpcost_opt = (W/ f- –1(p))

52. The Example Continued
Estimating Tpcost_opt for the cost-optimal addition algorithm
After some algebra, we get:
Tpcost_opt = 2 log n – log log n
Notice that Tpmin and Tpcost_opt are the same asymptotically, that is, both (log n)
This is typical for most systems
It is not true in general and we can have the situation that Tpcost_opt > (Tpmin)

53. An Example Of Tpcost_opt > (Tpmin)
Consider the hypothetical system with (slide 42):
TO = p3/2 + p3/4W3/4
Parallel runtime is: TP = (W + TO)/p = W/p + p1/2 + W3/4/p1/4
Taking the derivative to find Tpmin gives: p0 = (W)
Substituting back in to give Tpmin gives: Tpmin = (W1/2)
According to slide 43, the isoefficiency function W = (p3) = f(p)
Thus, p = f -1(p) = (W1/3)
Substituting into the equation for Tpcost_opt on slide 51 gives
Tpcost_opt = (W2/3)
Thus, Tpcost_opt > (Tpmin)
This does happen often

54. Limitation By Degree Of Concurrency C(W)
Beware:
The study of asymptotic behavior is valuable and interesting, but increasing p asymptotically is unrealistic
For example, p0 larger than C(W) is meaningless
For such cases, Tpmin is:
Needless to say, for problems where W grows unendlessly, C(W) may also grow unendlessly so that considering large p is reasonable

55. Recall: the serial best time is n log n
Asymptotic Analysis Of Parallel Programs Table For 4 Parallel Sort Programs of n Numbers
Algorithm A1 A2 A3 A4 p n2
log n n n TP 1
n log n S
n log n E
(log n)/n
(log n)/ n
pTP
n1.5
Recall: the serial best time is n log n
Question: Which is the best?

56. Comments On The Table Comparison by speed TP:
A1 is best followed by A3, A4, and A2
But A1 is not practical for large n
It requires n2 processors
Let's compare via efficiency E:
A2 and A4 are best, followed by A3 and A1
Look at the costs pTP now:
A2 and A4 are cost-optimal where A3 and A1 are not
Overall, then A2 is the best …
if least number of processors is important
Overall, then A4 is the best …
if least parallel time is important

57. Other Scalability Metrics
Other metrics to handle less general cases have been developed
For example:
Metrics that deal with problems that must be solved in a specified time -- real time problems
Metrics that deal with the fact that memory may be the limiting factor and scaling of the number of processors may be necessary, not for increased performance, but because of increased memory
That is, memory scales linearly with the number of processors p
Scaled speedup
Serial fraction

58. Scaled Speedup
Analyze the speedup, increasing the problem size linearly with the number of processors
This analysis can be done by constraining either time or memory in the analysis
To see this, consider the following two examples:
an parallel algorithm for matrix-vector products
an parallel algorithm for matrix-matrix products

59. Scaled Speedup For Matrix-Vector Products
The serial time Ts performing matrix-vector product for a matrix of size nn is: tc n2
where tc is the time for a multiply-add operation
Suppose the parallel time Tp for a simple parallel algorithm (Section 8.2.1) is:
Then, the speedup is:

60. Scaled Speedup For Matrix-Vector Products Continued
Consider a memory scaling constraint
Require the memory to scale as (p)
But the memory requirement for the matrix is (n2)
Therefore n2 = (p) or n2 = c  p
Substituting into the speedup formula, we get:
Thus, the scaled speedup with a memory constraint is (p)

61. Scaled Speedup For Matrix-Vector Products Continued
Consider a time scaling constraint
Require the time to be constant as the number of processors increases
But the parallel time is (n2/p)
Therefore n2/p = c or n2 = c  p
This is the same requirement as for the memory constrained case and so the same scaled speedup results
Thus, the scaled speedup with a time constraint is also (p)

62. Scaled Speedup For Matrix-Matrix Products
The serial time Ts performing matrix-matrix product for a matrix of size nn is: tc n3
where tc is the time for a multiply-add operation
Suppose the parallel time Tp for a simple parallel algorithm (Section 8.2.1) is:
Then, the speedup is:

63. Scaled Speedup For Matrix-Matrix Products Continued
Consider a memory scaling constraint
Require the memory to scale as (p)
But the memory requirement for the matrix is (n2)
Therefore n2 = (p) or n2 = c  p
Substituting into the speedup formula, we get:
Thus, the scaled speedup with a memory constraint is (p), that is, linear

64. Scaled Speedup For Matrix-Matrix Products Continued
Consider a time scaling constraint
Require the time to be constant as the number of processors increases
But the parallel time is (n3/p)
Therefore n3/p = c or n3 = c  p
Substituting into the speedup formula, we get:
Thus, the scaled speedup with a time constraint is also (p5/6), that is, sublinear speedup

65. Serial Fraction
Used as with the other measures to indicate the nature of the scalability of a parallel algorithm
What is it?
Assume the work W be broken into two parts
The part that is totally serial, denoted as Tser
We assume this includes all the interaction time
The part that is totally parallel, denoted as Tpar
Then the work W = Tser + Tpar
Define the serial fraction as f = Tpar/W
Now, we seek an expression for f in terms of p and S in order to study how f changes with p

66. Serial Fraction Continued
From the definition of TP:
Using the relation S = W/TP and solving for f gives a formula for f in terms of S and f :
It is not clear how f varies with p here
If f increases with increasing p, the system is considered scalable
Let's look at what this formula tells us for the matrix-vector product

67. Serial Fraction Example
For the matrix vector product:
This indicates that the serial fraction f grows with increasing p and so the parallel algorithm is considered scalable