1. WB1440 Engineering Optimization – Concepts and Applications Engineering Optimization Concepts and Applications Fred van Keulen Matthijs Langelaar CLA H21.1 A.vanKeulen@tudelft.nl

2. WB1440 Engineering Optimization – Concepts and Applications Contents ● Optimization problem checking and simplification ● Model simplification

3. WB1440 Engineering Optimization – Concepts and Applications Model simplification ● Basic idea: Expensive model Optimizer Cheap model Optimizer ● Motivation: – Replacement of expensive function, evaluated many times – Interaction between different disciplines – Estimation of derivatives – Noise

4. WB1440 Engineering Optimization – Concepts and Applications Model simplification (2) ● Drawback: loss of accuracy ● Different ranges: local, mid-range, global ● Synonyms: – Approximation models – Metamodels – Surrogate models – Compact models – Reduced order models Extract information Construct approximation Procedure:

5. WB1440 Engineering Optimization – Concepts and Applications Model simplification (3) ● Information extraction: linked to techniques from physical experiments: “plan of experiments” / DoE ● Many approaches! Covered here: – Taylor series expansions – Exact fitting – Least squares fitting (response surface techniques) – Kriging – Reduced basis methods – Briefly: neural nets, genetic programming, simplified physical models ● Crucial: purpose, range and level of detail

6. WB1440 Engineering Optimization – Concepts and Applications Taylor series expansions ● Approximation based on local information: Truncation error! ● Use of derivative information! ● Valid in neighbourhood of x

7. WB1440 Engineering Optimization – Concepts and Applications Taylor approximation example 1 st order 2 nd order 3 rd order 4 th order5 th order 20 th order Function Approximation (x = 20) x

8. WB1440 Engineering Optimization – Concepts and Applications Exact fitting (interpolation) ● # datapoints = # fitting parameters ● Every datapoint reproduced exactly ● Example: x1x1 x2x2 f2f2 f1f1

9. WB1440 Engineering Optimization – Concepts and Applications Exact fitting (2) ● Easy for intrinsically linear functions: ● No smoothing / filtering / noise reduction ● Danger of oscillations with high-order polynomials ● Often used: polynomials, generalized polynomials:

10. WB1440 Engineering Optimization – Concepts and Applications 9 th order polynomial Oscillations ● Referred to as “Runge phenomenon” ● In practice: use order 6 or less 5 th order 9 th order

11. WB1440 Engineering Optimization – Concepts and Applications Link between FD and exact fitting ● Example: 2 nd order polynomial fit: xx+hx-h

12. WB1440 Engineering Optimization – Concepts and Applications Least squares fitting ● Less fitting parameters than datapoints ● Smoothing / filtering behaviour ● “Best fit”? Minimize sum of deviations: ● “Best fit”? Minimize sum of squared deviations: x f

13. WB1440 Engineering Optimization – Concepts and Applications Least squares fitting (2) ● Choose fitting function linear in parameters a i : ● Short notation:

14. WB1440 Engineering Optimization – Concepts and Applications LS fitting (3) ● Minimize sum of squared errors: (Optimization problem!)

15. WB1440 Engineering Optimization – Concepts and Applications Polynomial LS fitting ● Polynomial of degree m :

16. WB1440 Engineering Optimization – Concepts and Applications Polynomial LS example -0.8-0.6-0.4-0.200.20.40.60.81 -0.2 0 0.2 0.4 0.6 0.8 1 1.2 samples quadratic 6th degree

17. WB1440 Engineering Optimization – Concepts and Applications Multidimensional LS fitting ● Polynomial in multiple dimensions: ● Number of coefficients a i for quadratic polynomial in R n : Curse of dimensionality!

18. WB1440 Engineering Optimization – Concepts and Applications Fractional factorial design Response surface ● Generate datapoints through sampling: – Generate design points through Design of Experiments – Evaluate responses ● Fit analytical model ● Check accuracy 2 n full factorial design x1x1 x2x2 x3x3

19. WB1440 Engineering Optimization – Concepts and Applications Latin Hypercube Sampling (LHS) ● Popular method: LHS ● Based on idea of Latin square: ● Properties: – Space-filling – Any number of design points – Intended for box-like domains – Matlab: lhsdesign

20. WB1440 Engineering Optimization – Concepts and Applications (LS) Fit quality indicators ● Accuracy? More / fewer terms? ● Examine the residuals – Small – Random! xixi   Okay: >0.6 Okay: >>1 ● Statistical quality indicators: – R 2 correlation measure: – F-ratio (signal to noise):

21. WB1440 Engineering Optimization – Concepts and Applications Nonlinear LS ● Linear LS: intrinsically linear functions (linear in a i ): ● Nonlinear LS: more complicated functions of a i : ● More difficult to fit! (Nonlinear optimization problem) ● Matlab: lsqnonlin

22. WB1440 Engineering Optimization – Concepts and Applications LS pitfalls ● Scattered data: ● Wrong choice of basis functions: x f x f

23. WB1440 Engineering Optimization – Concepts and Applications Kriging ● Named after D.C. Krige, mining engineer, 1951 ● Statistical approach: correlation between neighbouring points – Interpolation by weighted sum: – Weights depend on distance – Certain spatial correlation function is assumed (usually Gaussian)

24. WB1440 Engineering Optimization – Concepts and Applications Kriging properties ● Kriging interpolation is “most likely” in some sense (based on assumptions of the method) ● Interpolation: no smoothing / filtering ● Many variations exist! ● Advantage: no need to assume form of interpolation function ● Fitting process more elaborate than LS procedure

25. WB1440 Engineering Optimization – Concepts and Applications Kriging example ● Results depend strongly on statistical assumptions and method used: Dataset z(x,y) Kriging interpolation

26. WB1440 Engineering Optimization – Concepts and Applications Reduced order model ● Idea: describing system in reduced basis: – Example: structural dynamics ● Select small number of “modes” to build basis – Example: eigenmodes

27. WB1440 Engineering Optimization – Concepts and Applications Reduced order model (2) ● Reduced basis: ● Reduced system equations: N1N1NkNkk1k1kNkNNkNkNNNNN1N1kNkN

28. WB1440 Engineering Optimization – Concepts and Applications Reduced order models ● Many approaches! – Selection of type and number of basis vectors – Dealing with nonlinearity / multiple disciplines ● Active research topic ● No interpolation / fitting, but approximate modeling

29. WB1440 Engineering Optimization – Concepts and Applications Aerodynamic model Example:Aircraft model Structural model Mass model

30. WB1440 Engineering Optimization – Concepts and Applications Neural nets

31. WB1440 Engineering Optimization – Concepts and Applications  (input) output Neural nets To determine internal neuron parameters, neural nets must be trained on data. x f(x)f(x)

32. WB1440 Engineering Optimization – Concepts and Applications Neural net features ● Versatile, can capture complex behavior ● Filtering, smoothing ● Many variations possible – Network – Number of neurons, layers – Transfer functions ● Many training steps might be required (nonlinear optimization) ● Matlab: see e.g. nndtoc

33. WB1440 Engineering Optimization – Concepts and Applications Radial basis functions ● Often used as transfer function in neural networks ● Can either be used for interpolation or smoothing ● Very versatile ● Check out www.aranz.com!www.aranz.com

34. WB1440 Engineering Optimization – Concepts and Applications Genetic programming ● Building mathematical functions using evolution-like approach ● Approach good fit by crossover and mutation of expressions ^2 + / x2x2 x3x3 x1x1

35. WB1440 Engineering Optimization – Concepts and Applications Genetic programming ● LS fitting with population of analytic expressions ● Selection / evolution rules ● Features: – Can capture very complex behavior – Danger of artifacts / overfitting – Quite expensive procedure

36. WB1440 Engineering Optimization – Concepts and Applications Simplified physical models ● Goal: capture trends from underlying physics through simpler model: – Lumped / Analytic / Coarse ● Parameters fitted to “high-fidelity” data Simplified model Correction function x f(x)f(x) ● Refinement: correction function, parameter functions...

37. WB1440 Engineering Optimization – Concepts and Applications Model simplification summary Many different approaches: – Local: Taylor series (needs derivatives) – Interpolation (exact fit):  (Polynomial) fitting  Kriging – Fitting: LS – Approximate modeling: reduced order / simplified models – Other: genetic programming, neural nets, etc

38. WB1440 Engineering Optimization – Concepts and Applications Response surfaces in optimization ● Popular approach for computationally expen- sive problems: 1. DoE, generate samples (expensive) in part of domain 2. Build response surface (cheap) 3. Perform optimization on response surface (cheap) 4. Update domain of interest, and repeat Expensive model Optimizer Cheap model Optimizer ● Additional advantage: smoothens noisy responses ● Easy to combine with parallel computing

39. WB1440 Engineering Optimization – Concepts and Applications Example: Multi-point Approximation Method Trust region Design domain Response surface Sub-optimal point Optimum (Expensive) simulation