1. Workshop on Empirical Methods for the Analysis of Algorithms
Evaluation of Parallel Metaheuristics
by Enrique Alba and Gabriel Luque
17/02/2019

2. Introduction
Parallelism is an approach that allows to reduce the execution time and also to improve the quality of the solutions.
Clear metrics are necessary that allow to measure the performance of the parallel optimization procedure and to compare with other (parallel) approaches.
Currently, there are several parallel metrics but their meaning and utilization in the metaheuristic community is not homogeneous.
We are interested in revising, proposing, and applying parallel performance metrics and guidelines to ensure the correctness of our conclusions.
17/02/2019

3. Parallel Performance Metrics (I): Speedup
The most important parallel measure is the speedup that computes the ratio between sequential and parallel execution times.
Speedup taxonomy:
Strong speedup.
Weak speedup:
Speedup with solution stop:
Versus Panmixia.
Orthodox.
Speedup with predefined effort.
17/02/2019

4. Parallel Performance Metrics (II): Other Measures
Efficiency:
Incremental efficiency:
Generalized incremental efficiency:
Scaleup:
Serial fraction:
17/02/2019

5. Inadequate Utilization of Parallel Measures (I)
Computational effort evaluation:
Eliminates effects of implementation, software and hardware.
Misleading in the field of parallel methods:
Evaluation time is not constant.
The goal of parallel methods is (only) not the reduction of the number of evaluations but the reduction of time.
Comparing means/medians:
We cannot compare two averages or medians directly but we must compare the statistical distributions of the data.
Statistical tests:
Normal data: Student t-test or ANOVA.
Otherwise: non parametric test (e.g., Kruskal-Wallis).
17/02/2019

6. Inadequate Utilization of Parallel Measures (II)
Comparing algorithms with different accuracy:
We are comparing different things.
E.g.: comparing methods solving different problems or different instances.
Comparing parallel versions vs. canonical serial ones:
We are comparing different algorithms.
E.g.: comparing different methods (e.g., pGA vs. pSA).
Using a predefined effort:
We are indirectly imposing the execution time.
It is incorrect to use it to measure speedup.
17/02/2019

7. Examples (I): Panmictic vs. Orthodox Speedup
Panmictic speedup provides superlinear values.
Orthodox speedup are worse (sublinear) than panmictic one but fair and realistic.
Both cases show a same trend but in other experiments the trends could even be contradictory.
17/02/2019

8. Examples (II): Speedup with Predefined Effort
The termination condition is based on a predefined effort (a maximum number of evaluations).
The calculation of speedup is not appropriate:
Accuracy of the solution found is different.
We have fixed the execution time, and then we can calculate a theoretical speedup:
timem = cm * evalm * teval (where cm  1/m).
Predefined effort: eval1 = evalm
sm = c1 / cm.
17/02/2019

9. Examples (III): Other Parallel Metrics
Efficiency makes easy the analysis of the speedup.
We observe a moderate loss of efficiency when we increase the number of processors.
Since the serial fraction is almost constant, that loss of efficiency is due to the limited parallelism.
17/02/2019

10. Conclusions
In this work we have considered the issue of reporting parallel experimental research with parallel metaheuristics.
We have observed that speedup is the most important metric in this field but several definition of it can be used.
Speedup can be only applied when all the methods find solutions of similar quality and in these cases, the most appropriate definition is the orthodox one.
The utilization of other metrics (efficiency and serial fraction) is an interesting complement to perform a complete and fair comparison.
17/02/2019

11. Reykjavik, Iceland, September 2006
Málaga (SPAIN)
Questions?
17/02/2019