1. An Introduction to Optimization Theory

2. Outline Introduction Unconstrained optimization problem Constrained optimization problem

3. Introduction Mathematically speaking, optimization is the minimization of a objective function subject to constraints on its variables. Mathematically, we have

4. Introduction

5. Introduction-Linear regression

6. Introduction-Battery charger

7. Unconstrained optimization problem Definition for unconstrained optimization problem:

8. Unconstrained optimization problem

9. Gradient descent algorithm

10. Gradient descent algorithm may be trapped into the local extreme instead of the global extreme

11. Gradient descent algorithm Methodology for choosing suitable step size α k ---- Steepest descent algorithm

12. Gradient descent algorithm

13. Steepest descent algorithm with quadratic cost function:

14. Gradient descent algorithm Update equation:

15. Newton method Summary for Newton method

16. Newton method

17. Procedure for Newton method

18. Quasi-Newton method

19. What properties of F(x (k) ) -1 should it mimic ? 1. H k should be a symmetric matrix 2. H k should with secant property

20. Quasi-Newton method Typical approaches for Quasi-Newton method 1. Rank-one formula 2. DFP algorithm 3. BFGS algorithm (L-BFGS, L indicates limited-memory)

21. Constrained optimization problem Definition for constrained optimization problem

22. Problems with equality constraints ---- Lagrange multiplier

25. Suppose x * is a local minimizer

27. Karush-Kuhn-Tucker condition (KKT) From now on, we will consider the following problem

28. Karush-Kuhn-Tucker condition (KKT) Note that:

29. Image statistics & Image enhancement Illustration for gradient descent with projection Constrained set Ω Initial solution Projection

30. Useful Matlab introductions for optimization Useful instructions included in Matlab for optimization 1. fminunc: Solver for unconstrained optimization problems 2. fmincon: Solver for constrained optimization problems 3. linprog: Solver for linear programming problems 4. quadprog: Solver for quadratic programming problems