1. Surrogate-based constrained multi-objective optimization Aerospace design is synonymous with the use of long running and computationally intensive simulations, which are employed in the search for optimal designs in the presence of multiple, competing objectives and constraints. The difficulty of this search is often exacerbated by numerical `noise' and inaccuracies in simulation data and the frailties of complex simulations, that is they often fail to return a result. Surrogate-based optimization methods can be employed to solve, mitigate, or circumvent problems associated with such searches. This presentation gives an overview of constrained multi-objective optimization using Gaussian process based surrogates, with an emphasis on dealing with real-world problems. Alex Forrester 3 rd July 2009

2. Coming up: Surrogate model based optimization – the basic idea Gaussian process based modelling Probability of improvement and expected improvement Missing data Noisy data Constraints Multiple objectives 2

3. Surrogate model based optimization Surrogate used to expedite search for global optimum Global accuracy of surrogate not a priority 3 SAMPLING PLAN OBSERVATIONS CONSTRUCT SURROGATE(S) design sensitivities available? multi-fidelity data? SEARCH INFILL CRITERION (optimization using the surrogate(s)) constraints present? noise in data? multiple design objectives? ADD NEW DESIGN(S) PRELIMINARY EXPERIMENTS

4. Gaussian process based modelling 4

5. Building Gaussian process models, e.g. Kriging 5 Sample the function to be predicted at a set of points

6. Correlate all points using a Gaussian type function 6

7. 7 20 Gaussian “bumps” with appropriate widths (chosen to maximize likelihood of data) centred around sample points

8. Multiply by weightings (again chosen to maximize likelihood of data) 8

9. Add together to predict function 9 Kriging predictionTrue function

10. Optimization 10

11. Polynomial regression based search (as Devil’s advocate)

12. Gaussian process prediction based optimization 12

13. Gaussian process prediction based optimization (as Devil’s advocate) 13

14. But, we have error estimates with Gaussian processes 14

15. Error estimates used to construct improvement criteria 15 Probability of improvement Expected improvement

16. Probability of improvement 16 Useful global infill criterion Not a measure of improvement, just the chance there will be one

17. Expected improvement 17 Useful metric of actual amount of improvement to be expected Can be extended to constrained and multi- objective problems

18. 18

19. Missing Data 19

20. What if design evaluations fail? No infill point augmented to the surrogate –model is unchanged –optimization stalls Need to add some information or perturb the model –add random point? –impute a value based on the prediction at the failed point, so EI goes to zero here? –use a penalized imputation (prediction + error estimate)? 20

21. Aerofoil design problem 2 shape functions (f 1,f 2 ) altered Potential flow solver (VGK) has ~35% failure rate 20 point optimal Latin hypercube max{E[I(x)]} updates until within one drag count of optimum 21

22. Results 22

23. A typical penalized imputation based optimization 23

24. Four variable problem f 1,f 2,f 3,f 4 varied 82% failure rate 24

25. A typical four variable penalized imputation based optimization Legend as for two variable Red crosses indicate imputed update points. Regions of infeasible geometries are shown as dark blue. Blank regions represent flow solver failure 25

26. ‘Noisy’ Data 26

27. ‘Noisy’ data Many data sets are corrupted by noise We are usually interested in deterministic ‘noise’ ‘Noise’ in aerofoil drag data due to discretization of Euler equations 27

28. Failure of interpolation based infill Surrogate becomes excessively snaky Error estimates increase Search becomes too global 28

29. Regression improves model Add regularization constant to correlation matrix Last plot of previous slide improved 29

30. Failure of regression based infill Regularization assumes error at sample locations (brought in through lambda in equations below) Leads to expectation of improvement here Ok for stochastic noise Search stalls for deterministic simulations 30

31. Use “re-interpolation” Error due to noise ignored using new variance formulation (equation below) Only modelling error Search proceeds as desired 31

32. Two variable aerofoil example Same parameterization as missing data problem Course mesh causes ‘noise’ 32

33. Interpolation – very global 33

34. Regression - stalls 34

35. Re-interpolation – searches local basins, but finds global optimum 35

36. Constrained EI 36

37. Probability of constraint satisfaction g(x) is the constraint function F=G(x)-g min is a measure of feasibility, where G(x) is a random variable 37

38. It’s just like the probability of improvement, but with a limit, not a minimum 38 Probability of satisfaction Prediction of constraint function Constraint function Constraint limit

39. Constrained probability of improvement Probability of improvement conditional upon constraint satisfaction Simply multiply the two probabilities: 39

40. Constrained expected improvement Expected improvement conditional upon constraint satisfaction Again, a simple multiplication: 40

41. A 1D example 41

42. After one infill point 42

43. A 2D example 43

44. 44

45. Multi-objective EI 45

46. Pareto optimization We want to identify a set of non-dominated solutions These define the Pareto front We can formulate an expectation of improvement on the current non-dominated solutions 46

47. Multi-dimensional Gaussian process Consider a 2 objective problem The random variables Y 1 and Y 2 have a 2D probability density function: 47

48. Probability of improving on one point Need to integrate the 2D pdf: 48

49. Integrating under all non-dominated solutions: The EI is the first moment of this integral about the Pareto front (see book) 49

50. A 1D example 50

51. 51

52. Matlab demo 52

53. Nowacki beam Fixed length steel cantilever beam under 5kN load Variables: –height –width Objectives: –minimize cross section area –minimize bending moment Constraints: –area ratio –Bending moment –buckling –deflection –Shear 53

54. Problem setup 10 point optimal Latin hypercube Kriging model of each objective and constraint –Parameters tuned with GA + SQP (using adjoint of likelihood) 20 points added at the maximum constrained multi- objective expected improvement 54

55. Sampling plan 55

56. Initial trade off 56

57. 5 updates 57

58. 10 updates 58

59. 15 updates 59

60. 20 updates 60

61. Final trade off 61

62. Summary Surrogate based optimization offers answers to, or ways to get round, many problems associated with real world optimization This seemingly blunt tool must, however, be used with precision as there are many traps to fall into In a multi-objective context, the use of surrogates is particularly promising There has not been time to cover new surrogate methods (e.g. blind Kriging), multi-fidelity modelling or enhancements to EI, in terms of its exploitation/exploration tradeoff properties 62

63. References A. I. J. Forrester, A. Sóbester, A. J. Keane, Engineering Design via Surrogate Modelling: A Practical Guide, John Wiley & Sons, Chichester, 240 pages, ISBN 978-0-470-06068-1. A. I. J. Forrester, A. J. Keane, Recent advances in surrogate-based optimization, Progress in Aerospace Sciences, 45, 50-79, (doi:10.1016/j.paerosci.2008.11.001) A. I. J. Forrester, A. Sóbester, A. J. Keane, Optimization with missing data, Proc. R. Soc. A, 462(2067), 935-945, (doi:10.1098/rspa.2005.1608). A. I. J. Forrester, N. W. Bressloff, A. J. Keane, Design and analysis of ‘noisy’ computer experiments, AIAA journal, 44(10), 2331-2339, (doi:10.2514/1.20068). All code at www.wiley.com/go/forrester 63

64. Gratuitous publicity 64