1. Engineering Optimization
Concepts and Applications
Fred van Keulen
Matthijs Langelaar
CLA H21.1

2. Covered so far … Introduction: Optimization problem: Problem checking:
Negative null form
Applications
Optimization problem:
Definition
Characteristics
Problem checking:
Boundedness
Monotonicity analysis
Optimization model:
Model simplification
Approximation

3. Upcoming topics Optimization problem Definition Checking
Negative null form
Model
Special topics
Sensitivity analysis
Topology optimization
Linear / convex problems
Solution methods
Optimality criteria
Optimization algorithms
Unconstrained problems
Constrained problems

4. Contents Unconstrained problems Transformation methods
Existence of solutions, optimality conditions
Nature of stationary points
Global optimality

5. Unconstrained Optimization
Why?
Elimination of active constraints  unconstrained problem
Develop basic understanding useful for constrained optimization
Transformation of constrained problems into unconstrained problems
Relevant engineering problems (potential energy minimization)
Unconstrained problems are rare in practice, most practical optimum design problems involve restrictions of some sort.
Transformation of constrained problems into unconstrained problems

6. Transforming constrained problem
Reformulation through barrier functions: g
Transformation:
Also called barrier functions. This is just one of the many possibilities. x f

7. Transformed problem g f

8. Transformed problem
Barrier functions result in feasible, interior optimum:
r = 100
An iterative optimization process is often used, where r is gradually increased.
Note, recent research has also produced exact barrier functions. However, this is outside the scope of this course.
r = 200
r = 400
r = 800

9. Penalization Alternative reformulation: penalty functions g
Transformation: f

10. Penalization (2)
Penalty functions result in infeasible, exerior optimum:
p = 40
p = 20
p = 10
p = 4

11. Problem transformation summary
Barrier function Penalty function
Need feasible Yes No starting point?
Nature of optimum Interior Exterior (feasible) (infeasible)
Type of constraints g g, h
Exact methods: see e.g.

12. Unconstrained Optimization
Why?
Elimination of active constraints  unconstrained problem
Develop basic understanding useful for constrained optimization
Transformation of constrained problems into unconstrained problems
Relevant engineering problems (potential energy minimization)
Unconstrained problems are rare in practice, most practical optimum design problems involve restrictions of some sort.

13. Unconstrained engineering problem
Example: displacement of loaded structure
Fx = 5N
Fy = 5N
k1 = 8N/cm
k2 = 1N/cm
10 cm x1 x2
Equilibrium: minimum potential energy

14. Unconstrained engineering problem
Potential energy: x1 x2 k1 Fy Fx k2
Equilibrium:

15. Unconstrained engineering problemx1 x2 P x2 x1

16. Contents Unconstrained problems Transformation methods
Existence of solutions, optimality conditions
Nature of stationary points
Global optimality

17. Theory for solving unc. problems
Assumptions:
Objective continuous and differentiable (C1)
Domain closed and bounded (compact)
]a, b[ f

18. Existence of minima Weierstrass Theorem:
“A continuous function on a compact set has a maximum and a minimum in that set”
Sufficient condition for existence!
Interior optima only exist for non-monotonic functions

19. One-dimensional case Calculus: conditions for minimum of f
Derivative zero:
Second derivative positive:
(necessary)
(sufficient)
Interpretation through Taylor series: f x +h
Emphasize that a Taylor series is a LOCAL approximation. The conditions apply to LOCAL optima.
Second order derivative positive means that the gradient is increasing.
Points with f’=0 are called stationary points. Stationary points with f’’>0 are local minima. -h

20. One-dimensional case (2)
Condition for minimum: f”(x) > 0
Other possibilities:
f”(x) < 0 maximum
f”(x) = 0 ? Check higher order derivatives

21. Geometrical interpretation
Positive locally increasing f x

22. One-dimensional case (3)
Possible situations for stationary points (f’ = 0): f x

23. Example Aspirin pill revisited: worst pill ever!
Maximize dissolving time  minimize surface area r h
Meaningful problems for surface minimization of a cylinder with a fixed volume could be e.g. a cooled tank (surface is proportional to heat transfer with environment) or building the cheapest container tank for e.g. oil storage.
Equality constraint active  eliminate h

24. Example (2)

25. Multidimensional case
Local approximation to multidimensional minimum by multidimensional Taylor series:
Gradient
Condition for minimum:

26. Multidimensional case (2)
For minimum, consider second-order approximation:
Hessian

27. Multidimensional case (3)
Second order approximation:
These are the optimality conditions for unconstrained optimization.
Local minimum:
First Order Necessity Condition:
Second Order Sufficiency Condition:

28. Positive definite Hessian positive definite
Test for positive definiteness:
Evaluate yTHy for all y (impractical)
All eigenvalues li of H positive
Sylvester’s rule: all determinants of H and its principal submatrices are positive

29. Example Another loaded structure: Equilibrium:Fx Fy k1 k2 ux uy
Another loaded structure:
Equilibrium:
First order necessity condition:

30. Example (2) Second order sufficiency: H positive definite?
In a separable minimization problem, the minimization can be performed separately for every variable (1-D).
Note: H diagonal constant matrix. Separable objective function:

31. Quadratic functions Polynomial terms up to 2nd order: General form:
For geometrical interpretation of optimality conditions, consider quadratic functions.
Note: 2nd order Taylor series is exact

32. Quadratic functions (2)
Transformation to symmetric matrix:
Symmetric, since:

33. Quadratic functions (2)
Optimality conditions:
Gradient:
Hessian:
The gradient is defined by matrix differentiation, see here: or other matrix calculus reference works.
Stationary points:

34. Hessian positive definite
Example
Consider
Stationary point
Hessian positive definite

35. Example (2)
Result: minimum at (1.2, -2.6): f x2 x1

36. Example: least squares
Least squares fitting: unconstrained optimization problem
Stationary point: x f

37. Contents Unconstrained problems Transformation methods
Existence of solutions, optimality conditions
Nature of stationary points
Global optimality

38. Nature of stationary points
Hessian H positive definite:
Quadratic form
Eigenvalues
Local nature: minimum
Both definitions of positive definiteness are equivalent
Note positive curvatures
Eigenvalues can be interpreted as curvatures.

39. Nature of stationary points (2)
Hessian H negative definite:
Quadratic form
Eigenvalues
Local nature: maximum

40. Nature of stationary points (3)
Hessian H indefinite:
Quadratic form
Eigenvalues
Local nature: saddle point

41. Nature of stationary points (4)
Hessian H positive semi-definite:
Quadratic form
Eigenvalues
H singular!
Local nature: valley

42. Nature of stationary points (5)
Hessian H negative semi-definite:
Quadratic form
Eigenvalues
H singular!
Local nature: ridge

43. Stationary point nature summary
Definiteness H Nature x*
Positive d. Minimum
Positive semi-d. Valley
Indefinite Saddlepoint
Negative semi-d. Ridge
Negative d. Maximum

44. Structural example
Compressed pin-jointed truss with rotational springs: F k1 k2 l
Maximum (unstable)
Minima (stable)
Saddles (unstable)

45. Contents Unconstrained problems Transformation methods
Existence of solutions, optimality conditions
Nature of stationary points
Global optimality

46. Global optimality Optimality conditions for unconstrained problem:
First order necessity: (stationary point)
Second order sufficiency: H positive definite at x*
Optimality conditions only valid locally:
local minimum
Specifically, how to be sure a global optimum has been found.
When can we be sure x* is a global minimum?

47. Convex functions
Convex function: any line connecting any 2 points on the graph lies above it (or on it):
Note difference between strictly convex and “normal” convex.
Equivalent: tangent lines/planes stay below the graph
H positive (semi-)definite  f locally convex (proof by Taylor approximation)

48. Convex domains Convex set:
“A set S is convex if for every two points x1, x2 in S, the connecting line also lies completely inside S”

49. Convexity and global optimality
If:
Objective f = (strictly) convex function
Feasible domain = convex set
(Ok for unconstrained optimization)
Stationary point = (unique) global minimum
Special case: f, g, h all linear  linear optimization
programming
More general class: convex optimization

50. Example
Quadratic functions with A positive definite are strictly convex: f x1 x2
Stationary point (1.2, -2.6) must be unique global optimum

51. Summary optimality conditions
Conditions for local minimum of unconstrained problem:
First Order Necessity Condition:
Second Order Sufficiency Condition: H positive definite
For convex f in convex feasible domain: condition for global minimum:
Sufficiency Condition: