Workshop on Empirical Methods for the Analysis of Algorithms Evaluation of Parallel Metaheuristics by Enrique Alba and Gabriel Luque 17/02/2019

Introduction Parallelism is an approach that allows to reduce the execution time and also to improve the quality of the solutions. Clear metrics are necessary that allow to measure the performance of the parallel optimization procedure and to compare with other (parallel) approaches. Currently, there are several parallel metrics but their meaning and utilization in the metaheuristic community is not homogeneous. We are interested in revising, proposing, and applying parallel performance metrics and guidelines to ensure the correctness of our conclusions. 17/02/2019

Parallel Performance Metrics (I): Speedup The most important parallel measure is the speedup that computes the ratio between sequential and parallel execution times. Speedup taxonomy: Strong speedup. Weak speedup: Speedup with solution stop: Versus Panmixia. Orthodox. Speedup with predefined effort. 17/02/2019

Parallel Performance Metrics (II): Other Measures Efficiency: Incremental efficiency: Generalized incremental efficiency: Scaleup: Serial fraction: 17/02/2019

Inadequate Utilization of Parallel Measures (I) Computational effort evaluation: Eliminates effects of implementation, software and hardware. Misleading in the field of parallel methods: Evaluation time is not constant. The goal of parallel methods is (only) not the reduction of the number of evaluations but the reduction of time. Comparing means/medians: We cannot compare two averages or medians directly but we must compare the statistical distributions of the data. Statistical tests: Normal data: Student t-test or ANOVA. Otherwise: non parametric test (e.g., Kruskal-Wallis). 17/02/2019

Inadequate Utilization of Parallel Measures (II) Comparing algorithms with different accuracy: We are comparing different things. E.g.: comparing methods solving different problems or different instances. Comparing parallel versions vs. canonical serial ones: We are comparing different algorithms. E.g.: comparing different methods (e.g., pGA vs. pSA). Using a predefined effort: We are indirectly imposing the execution time. It is incorrect to use it to measure speedup. 17/02/2019

Examples (I): Panmictic vs. Orthodox Speedup Panmictic speedup provides superlinear values. Orthodox speedup are worse (sublinear) than panmictic one but fair and realistic. Both cases show a same trend but in other experiments the trends could even be contradictory. 17/02/2019

Examples (II): Speedup with Predefined Effort The termination condition is based on a predefined effort (a maximum number of evaluations). The calculation of speedup is not appropriate: Accuracy of the solution found is different. We have fixed the execution time, and then we can calculate a theoretical speedup: timem = cm * evalm * teval (where cm  1/m). Predefined effort: eval1 = evalm sm = c1 / cm. 17/02/2019

Examples (III): Other Parallel Metrics Efficiency makes easy the analysis of the speedup. We observe a moderate loss of efficiency when we increase the number of processors. Since the serial fraction is almost constant, that loss of efficiency is due to the limited parallelism. 17/02/2019

Conclusions In this work we have considered the issue of reporting parallel experimental research with parallel metaheuristics. We have observed that speedup is the most important metric in this field but several definition of it can be used. Speedup can be only applied when all the methods find solutions of similar quality and in these cases, the most appropriate definition is the orthodox one. The utilization of other metrics (efficiency and serial fraction) is an interesting complement to perform a complete and fair comparison. 17/02/2019

Reykjavik, Iceland, September 2006 Málaga (SPAIN) Questions? 17/02/2019