1. CS 684
Reporting Computational Experiements with Parallel Algorithms: Issues, Measures, and Experts’ Opinions
Richard S. Barr
SMU
Betty L. Hickman
University of Nebraska at Omaha

2. Introduction Algorithm performance/efficiency
Theoretical (order analysis)
Emperical testing of implementations
Order analysis is often not enough
Based on worst case
Example: Simplex method
bad worst case, but excellent on large class of problems

3. Parallel Algorithm Performance Measurement Complications
Limitations imposed by machine designs
Differences in machine architectures
The stochastic nature of some parallel algorithms
Inherent opportunities for the introduction of biases
Particular Concern: Speedup

4. Background Why use parallel processing Types of parallel computers
Absolute speed, relative speed and cost, scalability
Types of parallel computers
Instruction
Stream
Single
Multiple
SISD
SIMD
MISD
MIMD
Data
Stream

5. Message Passing Architectures
Requires some form of interconnection
The network is the bottleneck
Latency and bandwidth
Diameter
Bisection bandwidth

6. Message Passing Architectures
Line/Ring
Mesh/Torus
Tree/Fat Tree

7. Message Passing Architectures
Hypercube

8. Message Passing Computers Traditional MIMD Machines
IBM SP-2
Cray T3E
Workstation Clusters
Interconnection Network
Processors
Memory Modules

9. Shared Memory Computers
Bus
Processors
SMP Machines
SGI Origin series

10. Interconnection Network
Distributed Shared Memory
SGI Origin series
Workstation Clusters
Kendall Square Research KSR1 and KSR2
Interconnection Network
Processors
Memory Modules

11. Supercomputer Evolution
Interconnection Network
Bus
Bus
Bus
…….
Interconnection Network
…….

12. Performance Analysis Sources of Speed We want:
Machine, architecture
Solution algorithm
Algorithm implementation
We want:
A quick machine and an efficient algorithm and implementation which matches the machine
Performance Analysis Goal
Empirical evidence of an algorithms efficiency
More than just order analysis

13. Reporting of Computational Testing
An efficiency measure should reflect computational effort and solution quality
Authors should state clearly
What is being tested
What performance criteria are being considered
What performance measure is being used to draw inferences
Bear in mind:
Performance measures are summary statistics and, as much as possible, should conform to all of the accepted rules regarding the use thereof.

14. Questions What is time? What is Speedup? What is efficiency?
Rationale for parallel processing is time
CPU time? Wall-clock?
What is Speedup?
Relative vs. absolute
What is efficiency?
Traditional vs. incremental
Is superlinear speedup possible?

15. Issues in Reporting Must choose a small subset of metrics
Choosing metrics
Use absolute speedup to demonstrate performance
Use relative speedup to demonstrate scalability
What portion of system overhead should you measure?
What constitutes the base case?
For stochastic codes, multiple runs?
Sources of Bias
Metric Choices
The longer the serial time, the greater the speedup
Consider the motivational bias

17. Answers & Comments Most popular: Other
Include all data
Could overwhelm
Single number can’t capture all data
Report Std. Dev.

19. Answers & Comments Present the table More problems should be tested
Single number not enough
More problems should be tested
Include results from a well-known code

21. Answers & Comments Other
63 % - measures that reflect variability
25 % - ratio of means or medians
12 % - raw data
Should provide variance measures since that is the purpose of the study
Serial differences due to random variability - use average serial time

23. Answers & Comments Effort Statistics Non respondents said it depends
Mean = 5.8
Median & Mode = 5
Std. Dev. = 2.9
Non respondents said it depends
Be reasonable - Give justification
May be preferable to compare with well-known algorithm

25. Answers & Comments Many said both (a) and (b)
More important is to explicitly state which of the above was used

27. Answers & Comments
“I tend to be skeptical about one number measuring the goodness of an algorithm”
“Use a number of performance measures”
“There must be a better method, but I do not know it.”
“No, but it is attractive to boil performance down to a single number, so it will likely continue as the dominant measure.”

28. Overall Comments
Rating the effectiveness of a parallel algorithm by a single measure, speedup, seems analogous to describing a probability distribution by reporting only its mean
As a rule, authors of code will present data to make their code appear best. That is human nature. More important is to explicitly state how they are reporting and how testing was performed, i.e., acknowledge their biases.

29. Overall Comments
The value of parallelism is an economic issue or real time speedup issue. … Without the cost of the parallel system, the benefits of speedup are meaningless.
Also, we should report the actual times--not only speedups--and on problem sizes where speedup is of the essence.

30. Overall Comments
After the initial excitement of actual implementation of conventional or new algorithms on parallel machines, speedup factors are going to lose their allure. … However, if we show that what took an hour on a $10,000,000 superframe, now takes 15 minutes on a $500,000 multicomputer, it will have a significant impact whatever the speedup factor is.

31. Overall Comments Rules to follow
Avoid point samples, i.e., solve each problem instance several times and solve many problem instances.
Summarize the data in more than one way. Be willing to report negative results.
More people should think of these important issues.

32. Survey Conclusions Use more than just speedup
Measures of variation are important
Report as much raw data as possible
No strong consensus on how
Reference values from well-known codes are vital
Use statistical experimental design
We should focus on real time and cost

33. Additional Performance Measures for MIMD
Generalized Incremental Efficiency
Problem is too big for one processor
Scaled Speedup
Indicates the value of additional processors
Fixed-time speedup
Estimate time on one processor
Scaleup
The ability of an n-times larger system to perform an n-times larger job

34. Heuristic Code Performance
Randomization
Closeness to optimality may not be determined exactly
When do you stop?
Heuristic control parameters?
Trade-off between time and solution quality
Best: Graph of time vs. quality

35. Guidelines Thoroughly document the process
Describe the code, algorithm, design, data structures, and tuning parameters
Document the computing environment
Describe the testing environment & methodology
How were times measured
How is speedup computed
What tuning parameters were used.
Describe the quality measure used.

36. Guidelines Use a well considered experimental design
Focus on real time and cost to solve difficult problems
Try to identify the factors that contribute to the results presented, and their effects.
Provide points of reference. Use well-known codes and problems
Perform final numbers on dedicated or lightly loaded systems
Employ statistical experimental design techniques. They can highlight factors that contribute to performance results.

37. Guidelines Provide a comprehensive report of the results
For summary measures, use measures of central tendency, variability, and cost-effectiveness.
Use graphics where possible and when informative
Provide as much detail as possible
Describe the sensitivity of the code to changes in the tuning strategy.
Be courageous and include your “failures,” since they provide insight also.
Objectivity is the key.