Quality of Pareto set approximations Eckart Zitzler, Jörg Fliege, Carlos Fonseca, Christian Igel, Andrzej Jaszkiewicz, Joshua Knowles, Alexander Lotov, Serpil Sayin, Andrzej Wierzbicki 4.5 EMO, 4.5 non-EMO

Lessons learned This chapter is a must This chapter is a must Obvious things are not obvious Obvious things are not obvious

Outline Motivation Motivation Use cases Use cases Testing (true Pareto front is known) Testing (true Pareto front is known) Practice (true Pareto front is not known) Practice (true Pareto front is not known) Comparing approximations Comparing approximations Unary measures Unary measures Binary measures Binary measures Discussion and conclusions Discussion and conclusions

Motivation Comparison of algorithms Comparison of algorithms Pictures are valuable but not sufficient Pictures are valuable but not sufficient Design of algorithms for vector optimization Design of algorithms for vector optimization To guide the search To guide the search Stopping criteria Stopping criteria Learning Learning

Use cases Testing Testing True Pareto front is known True Pareto front is known Practice Practice True Pareto front not known in general True Pareto front not known in general Additional useful information: Additional useful information: Lower bounds through relaxation Lower bounds through relaxation Upper bounds through random search or other methods Upper bounds through random search or other methods … Pairwise comparison Pairwise comparison

Comparing approximations Unary measures Unary measures Evaluation of a single approximation Evaluation of a single approximation Binary relations Binary relations Purely ordinal Purely ordinal Binary measures Binary measures Evaluation of difference between two approximations Evaluation of difference between two approximations

Unary measures Assign a real number to a Pareto set approximation Assign a real number to a Pareto set approximation Relevant properties Relevant properties Monotonicity (strict or not) Monotonicity (strict or not) Uniqueness of optimum Uniqueness of optimum Scale invariance Scale invariance Computational requirements Computational requirements Required information, e.g. reference set, bounds Required information, e.g. reference set, bounds

Examples of unary measures Outer diameter Outer diameter Proportion of Pareto-optimal points found Proportion of Pareto-optimal points found Cardinality Cardinality Hypervolume Hypervolume ε-dominance ε-dominance D-measures D-measures D1 - mean value of the Chenycheff distance from the Pareto front D1 - mean value of the Chenycheff distance from the Pareto front D2 – asymmetric Hausdorff distance with Chebycheff metric D2 – asymmetric Hausdorff distance with Chebycheff metric R-measure - expected value of Chebycheff scalarizing function R-measure - expected value of Chebycheff scalarizing function Uniformity measures Uniformity measures Probability of improvement through random search Probability of improvement through random search Combinations of the above measures Combinations of the above measures

Binary measures Relevant properties Relevant properties Monotonicity (strict or not) Monotonicity (strict or not) Scale invariance Scale invariance Computational requirements Computational requirements Required information, e.g. bounds Required information, e.g. bounds Transformation of measures into relations Transformation of measures into relations Partial orders stronger than dominance may be constructed Partial orders stronger than dominance may be constructed Implicit introduction of preferences Implicit introduction of preferences

Examples of binary measures Difference of two unary measure values Difference of two unary measure values Pareto dominance of sets Pareto dominance of sets Binary ε-dominance Binary ε-dominance Hypervolume of difference between two Edgeworth-Pareto hulls Hypervolume of difference between two Edgeworth-Pareto hulls Coverage Coverage

Discussion and conclusions Evaluation of algorithms – not discussed here Evaluation of algorithms – not discussed here Taking into account preference information - not discussed here Taking into account preference information - not discussed here Monotonicity is important in theoretical sense Monotonicity is important in theoretical sense In practice it may be relaxed In practice it may be relaxed Learning Learning Practical optimization algorithm may benefit from this relaxation, like constraint relaxation in numerical optimization Practical optimization algorithm may benefit from this relaxation, like constraint relaxation in numerical optimization Computational complexity also influences the choice of performance indicator(s) Computational complexity also influences the choice of performance indicator(s)