1. Optimization Problems 虞台文 大同大學資工所 智慧型多媒體研究室

2. Content Introduction Definitions Local and Global Optima Convex Sets and Functions Convex Programming Problems

3. Optimization Problems Introduction 大同大學資工所 智慧型多媒體研究室

4. General Nonlinear Programming Problems objective function constraints

5. Local Minima vs. Global Minima objective function constraints local minimum global minimum

6. Convex Programming Problems objective function constraints f (x) gi (x)gi (x) hj (x)hj (x) convex concave linear Local optimality  Global optimality

7. Linear Programming Problems objective function constraints f (x) gi (x)gi (x) hj (x)hj (x) linear Local optimality  Global optimality a special case of convex programming problems

8. Linear Programming Problems objective function constraints f (x) gi (x)gi (x) hj (x)hj (x) linear Local optimality  Global optimality

9. Integer Programming Problems objective function constraints f (x) gi (x)gi (x) hj (x)hj (x) linear

10. The Hierarchy of Optimization Problems Nonlinear Programs Convex Programs Linear Programs (Polynomial) Integer Programs (NP-Hard) Flow and Matching

11. Optimization Problems General Nonlinear Programming Problems Convex Programming Problems Linear Programming Problems Integer Linear Programming Problems

12. Optimization Techniques General Nonlinear Programming Problems Convex Programming Problems Linear Programming Problems Integer Linear Programming Problems Continuous Variables Discrete Variables Continuous Optimization Combinatorial Optimization

13. Optimization Problems Definitions 大同大學資工所 智慧型多媒體研究室

14. Optimization Problems

15. Define the set of feasible points F Minimize cost c: F  R 1

16. Definition: Instance of an Optimization Problem (F, c) F: the domain of feasible points c: F  R 1 cost function Goal: To find f  F such that c( f )  c(g) for all g  F. A global optimum

17. Definition: Optimization Problem A set of instances of an optimization problem, e.g. – Traveling Salesman Problem (TSP) – Minimal Spanning Tree (MST) – Shortest Path (SP) – Linear Programming (LP)

18. Traveling Salesman Problem (TSP)

19. Instance of the TSP – Given n cities and an n  n distance matrix [d ij ], the problem is to find a Hamiltonian cycle with minimal total length.

20. Minimal Spanning Tree (MST)

21. Instance of the MST – Given an integer n > 0 and an n  n symmetric distance matrix [d ij ], the problem is to find a spanning tree on n vertices that has minimum total length of its edge.

22. Linear Programming (LP) minimize Subject to

23. Linear Programming (LP) minimize Subject to

24. Linear Programming (LP) minimize Subject to

25. Example: Linear Programming (LP) minimize Subject to

26. Example: Linear Programming (LP) minimize Subject to x1x1 x2x2 x3x3 v1v1 v2v2 v3v3 c(v 1 ) = 8 c(v 2 ) = 4 c(v 3 ) = 6 The optimum The optimal point is at one of the vertices.

27. Example: Minimal Spanning Tree (3 Nodes) minimize Subject to c1=4c1=4 c3=3c3=3 c2=2c2=2 x 1  {0, 1} x 2  {0, 1} x 3  {0, 1} Integer Programming x1x1 x2x2 x3x3

28. Example: Minimal Spanning Tree (3 Nodes) minimize Subject to c1=4c1=4 c3=3c3=3 c2=2c2=2 x 1  {0, 1} x 2  {0, 1} x 3  {0, 1} Linear Programming x1x1 x2x2 x3x3 Some integer programs can be transformed into linear programs.

29. Optimization Problems Local and Global Optima 大同大學資工所 智慧型多媒體研究室

30. Neighborhoods Given an optimization problem with instance (F, c), a neighborhood is a mapping defined for each instance. For combinatorial optimization, the choice of N is critical.

31. TSP (2-Change) f  F g  N 2 (f )

32. TSP (k-Change)

33. MST f  F g  N(f ) 1.Adding an edge to form a cycle. 2.Deleting any edge on the cycle.

34. LP minimize Subject to

35. Local Optima Given (F, c) N an instance of an optimization problem neighborhood f  F is called locally optimum with respect to N (or simply locally optimum whenever N is understood by context) if c(f )  c(g) for all g  N(f ).

36. 0 1 F c small Local Optima F = [0, 1]  R 1 C B A Local minimum Global minimum

37. Decent Algorithm f = initial feasible solution While Improve(f )   do f = any element in Improve(f ) return f Decent algorithm is usually stuck at a local minimum unless the neighborhood N is exact.

38. Exactness of Neighborhood Neighborhood N is said to be exact if it makes Local minimum  Global Minimum

39. Exactness of Neighborhood 0 1 F c F = [0, 1]  R 1 C B A Local minimum Global minimum N  is exact if   1.

40. TSP N 2 : not exact N n : exact

41. MST N is exact f  F g  N(f ) 1.Adding an edge to form a cycle. 2.Deleting any edge on the cycle.

42. Optimization Problems Convex Sets and Functions 大同大學資工所 智慧型多媒體研究室

43. Convex Combination x, y  R n 0   1 z = x +(1  )y A convex combination of x, y. A strict convex combination of x, y if  0, 1.

44. Convex Sets S  RnS  Rn z = x +(1  )y is convex if it contains all convex combinations of pairs x, y  S. convex nonconvex 0   1

45. Convex Sets S  RnS  Rn z = x +(1  )y is convex if it contains all convex combinations of pairs x, y  S. n = 1 S is convex iff S is an interval. 0   1

46. Convex Sets Fact: The intersection of any number of convex sets is convex.

47. c Convex Functions xy x +(1  )y c(x)c(x) c(y)c(y) c(x) + (1  )c(y) c( x +(1  )y) S  RnS  Rn a convex set c:S  Rc:S  R a convex function if c( x +(1  )y)  c(x) + (1  )c(y),0   1 Every linear function is convex.

48. Lemma S c(x)c(x) t a convex set a convex function on S a real number is convex. Pf) Let x, y  S t x +(1  )y  S c( x +(1  )y)  c(x) + (1  )c(y)  t + (1  )t = t x +(1  )y  S t

49. Level Contours c = 1 c = 2 c = 3 c = 4 c = 5

50. Concave Functions S  RnS  Rn a convex set c:S  Rc:S  R a concave function if  c is a convex Every linear function is concave as well as convex.

51. Optimization Problems Convex Programming Problems 大同大學資工所 智慧型多媒體研究室

52. Theorem (F, c) an instance of optimization problem a convex set a convex function Define is exact for every  > 0.

53. Let x be a local minimum w.r.t. N  for any fixed  > 0. Let y  F be any other feasible point. Theorem Pf) x F y Next, we now want to show that c(y)  c(x).

54. Let x be a local minimum w.r.t. N  for any fixed  > 0. Let y  F be any other feasible point.   < <1 such that Since c is convex, we have Therefore, Theorem Pf) x F y z

55. Convex Programming Problems (F, c) Defined by Convex function an instance of optimization problem Important property: Local minimum  Global Minimum Concave functions

56. Convexity of Feasible Set (F, c) Defined by Convex function an instance of optimization problem Important property: Local minimum  Global Minimum Concave functions

57. Convex Programming Problems (F, c) Defined by Convex function an instance of optimization problem Important property: Local minimum  Global Minimum Concave functions Convex

58. Theorem In a convex programming problem, every point locally optimal with respect to the Euclidean distance neighborhood N  is also global optimal.