1. INTRO TO OPTIMIZATION MATH-415 Numerical Analysis 1

2. Optimization is finding values of the variables that minimize or maximize the objective function while (in general) satisfying some constraints Some slides from the Princeton university course COS-323 materials are used here https://www.cs.princeton.edu/courses/archive/fall09/cos323/syll.html https://www.cs.princeton.edu/courses/archive/fall09/cos323/syll.html Optimization 2

3. Different Kinds of Optimization Figure from: Optimization Technology Center http://www-fp.mcs.anl.gov/otc/Guide/OptWeb/

4. Optimization in 1-D Let us have a function of one variable Our task is to find minimum (or maximum) of this function 4

5. Optimization in 1-D Look for analogies to bracketing in bisection method for solving nonlinear equations What does it mean to bracket a minimum? ( x left, f(x left ) ) ( x right, f(x right ) ) ( x mid, f(x mid ) ) x left < x mid < x right f(x mid ) < f(x left ) f(x mid ) < f(x right ) x left < x mid < x right f(x mid ) < f(x left ) f(x mid ) < f(x right )

6. Optimization in 1-D Once we have these properties, there is at least one local minimum between x left and x right Establishing bracket initially:  Given x initial, increment  Evaluate f(x initial ), f(x initial +increment)  If decreasing, step until find an increase  Else, step in opposite direction until find an increase  Grow increment at each step For maximization: substitute –f for f

7. Optimization in 1-D Strategy: evaluate function at some x new ( x left, f(x left ) ) ( x right, f(x right ) ) ( x mid, f(x mid ) ) ( x new, f(x new ) )

8. Optimization in 1-D Strategy: evaluate function at some x new  Here, new “bracket” points are x new, x mid, x right ( x left, f(x left ) ) ( x right, f(x right ) ) ( x mid, f(x mid ) ) ( x new, f(x new ) )

9. Optimization in 1-D Strategy: evaluate function at some x new  Here, new “bracket” points are x left, x new, x mid ( x left, f(x left ) ) ( x right, f(x right ) ) ( x mid, f(x mid ) ) ( x new, f(x new ) )

10. Optimization in 1-D Unlike with bisection method for solving nonlinear equations, in can’t always guarantee that interval will be reduced by a factor of 2 Let’s find the optimal place for x mid, relative to left and right, that will guarantee same factor of reduction regardless of outcome

11. Optimization in 1-D: Golden Section Method  b and shall be selected in such a way that is the golden section (a positive root of the equation

12. Optimization in 1-D: Golden Section Method performs the golden section of  b

13. Optimization in 1-D: Golden Section Method: Algorithm (min) Localize the interval [a, b] to bracket the min and determine - accuracy of a solution Find If then else If then is our solution 13

14. Optimization in 1-D: Golden Section Method: Algorithm (max) Localize the interval [a, b] to bracket the max and determine - accuracy of a solution Find If then else If then is our solution 14

15. Optimization in 1-D: Golden Section Method: Analysis Golden section method is one of the classical methods used in 1-D optimization Like the bisection method for solving nonlinear equations, it has linear convergence, narrowing the bracketing interval in the golden proportion

16. Error Tolerance Around minimum, derivative = 0, so Rule of thumb: pointless to ask for more accuracy than sqrt(  )  Can use double precision if you want a single- precision result (and/or have single-precision data)

17. Faster 1-D Optimization Trade off super-linear convergence for worse robustness  Combine with Golden Section search for safety Usual bag of tricks:  Fit parabola through 3 points, find minimum  Compute derivatives as well as positions, fit cubic  Use second derivatives: Newton

18. Newton’s Method

22. At each step: Requires 1 st and 2 nd derivatives Quadratic convergence

23. Multi-Dimensional Optimization (Finding min/max of functions of multiple variables) Important in many areas  Fitting a model to measured data  Finding best design in some parameter space Hard in general  Weird shapes: multiple extrema, saddles, curved or elongated valleys, etc.  Can’t bracket

24. Newton’s Method in Multiple Dimensions Replace 1 st derivative with gradient and 2 nd derivative with Hessian matrix

25. Newton’s Method in Multiple Dimensions Replace 1 st derivative with gradient and 2 nd derivative with Hessian matrix So, Tends to be extremely fragile unless function very smooth and starting close to minimum

26. Steepest Descent Methods What if you can’t / don’t want to use 2 nd derivative? “Quasi-Newton” methods estimate Hessian Alternative: walk along (negative of) gradient…  Perform 1-D minimization along line passing through current point in the direction of the gradient  Once done, re-compute gradient, iterate

27. Problem With Steepest Descent

29. Conjugate Gradient Methods Idea: avoid “undoing” minimization that’s already been done Walk along direction Polak and Ribiere formula:

30. Conjugate Gradient Methods Conjugate gradient implicitly obtains information about Hessian For quadratic function in n dimensions, gets exact solution in n steps (ignoring roundoff error) Works well in practice…

31. Value-Only Methods in Multi- Dimensions If can’t evaluate gradients, life is hard Can use approximate (numerically evaluated) gradients:

32. Generic Optimization Strategies Uniform sampling:  Cost rises exponentially with # of dimensions Simulated annealing:  Search in random directions  Start with large steps, gradually decrease  “Annealing schedule” – how fast to cool?