Optimization Problems 虞台文 大同大學資工所 智慧型多媒體研究室

Content Introduction Definitions Local and Global Optima Convex Sets and Functions Convex Programming Problems

Optimization Problems Introduction 大同大學資工所 智慧型多媒體研究室

General Nonlinear Programming Problems objective function constraints

Local Minima vs. Global Minima objective function constraints local minimum global minimum

Convex Programming Problems objective function constraints f (x) gi (x)gi (x) hj (x)hj (x) convex concave linear Local optimality  Global optimality

Linear Programming Problems objective function constraints f (x) gi (x)gi (x) hj (x)hj (x) linear Local optimality  Global optimality a special case of convex programming problems

Linear Programming Problems objective function constraints f (x) gi (x)gi (x) hj (x)hj (x) linear Local optimality  Global optimality

Integer Programming Problems objective function constraints f (x) gi (x)gi (x) hj (x)hj (x) linear

The Hierarchy of Optimization Problems Nonlinear Programs Convex Programs Linear Programs (Polynomial) Integer Programs (NP-Hard) Flow and Matching

Optimization Problems General Nonlinear Programming Problems Convex Programming Problems Linear Programming Problems Integer Linear Programming Problems

Optimization Techniques General Nonlinear Programming Problems Convex Programming Problems Linear Programming Problems Integer Linear Programming Problems Continuous Variables Discrete Variables Continuous Optimization Combinatorial Optimization

Optimization Problems Definitions 大同大學資工所 智慧型多媒體研究室

Optimization Problems

Define the set of feasible points F Minimize cost c: F  R 1

Definition: Instance of an Optimization Problem (F, c) F: the domain of feasible points c: F  R 1 cost function Goal: To find f  F such that c( f )  c(g) for all g  F. A global optimum

Definition: Optimization Problem A set of instances of an optimization problem, e.g. – Traveling Salesman Problem (TSP) – Minimal Spanning Tree (MST) – Shortest Path (SP) – Linear Programming (LP)

Traveling Salesman Problem (TSP)

Instance of the TSP – Given n cities and an n  n distance matrix [d ij ], the problem is to find a Hamiltonian cycle with minimal total length.

Minimal Spanning Tree (MST)

Instance of the MST – Given an integer n > 0 and an n  n symmetric distance matrix [d ij ], the problem is to find a spanning tree on n vertices that has minimum total length of its edge.

Linear Programming (LP) minimize Subject to

Linear Programming (LP) minimize Subject to

Linear Programming (LP) minimize Subject to

Example: Linear Programming (LP) minimize Subject to

Example: Linear Programming (LP) minimize Subject to x1x1 x2x2 x3x3 v1v1 v2v2 v3v3 c(v 1 ) = 8 c(v 2 ) = 4 c(v 3 ) = 6 The optimum The optimal point is at one of the vertices.

Example: Minimal Spanning Tree (3 Nodes) minimize Subject to c1=4c1=4 c3=3c3=3 c2=2c2=2 x 1  {0, 1} x 2  {0, 1} x 3  {0, 1} Integer Programming x1x1 x2x2 x3x3

Example: Minimal Spanning Tree (3 Nodes) minimize Subject to c1=4c1=4 c3=3c3=3 c2=2c2=2 x 1  {0, 1} x 2  {0, 1} x 3  {0, 1} Linear Programming x1x1 x2x2 x3x3 Some integer programs can be transformed into linear programs.

Optimization Problems Local and Global Optima 大同大學資工所 智慧型多媒體研究室

Neighborhoods Given an optimization problem with instance (F, c), a neighborhood is a mapping defined for each instance. For combinatorial optimization, the choice of N is critical.

TSP (2-Change) f  F g  N 2 (f )

TSP (k-Change)

MST f  F g  N(f ) 1.Adding an edge to form a cycle. 2.Deleting any edge on the cycle.

LP minimize Subject to

Local Optima Given (F, c) N an instance of an optimization problem neighborhood f  F is called locally optimum with respect to N (or simply locally optimum whenever N is understood by context) if c(f )  c(g) for all g  N(f ).

0 1 F c small Local Optima F = [0, 1]  R 1 C B A Local minimum Global minimum

Decent Algorithm f = initial feasible solution While Improve(f )   do f = any element in Improve(f ) return f Decent algorithm is usually stuck at a local minimum unless the neighborhood N is exact.

Exactness of Neighborhood Neighborhood N is said to be exact if it makes Local minimum  Global Minimum

Exactness of Neighborhood 0 1 F c F = [0, 1]  R 1 C B A Local minimum Global minimum N  is exact if   1.

TSP N 2 : not exact N n : exact

MST N is exact f  F g  N(f ) 1.Adding an edge to form a cycle. 2.Deleting any edge on the cycle.

Optimization Problems Convex Sets and Functions 大同大學資工所 智慧型多媒體研究室

Convex Combination x, y  R n 0   1 z = x +(1  )y A convex combination of x, y. A strict convex combination of x, y if  0, 1.

Convex Sets S  RnS  Rn z = x +(1  )y is convex if it contains all convex combinations of pairs x, y  S. convex nonconvex 0   1

Convex Sets S  RnS  Rn z = x +(1  )y is convex if it contains all convex combinations of pairs x, y  S. n = 1 S is convex iff S is an interval. 0   1

Convex Sets Fact: The intersection of any number of convex sets is convex.

c Convex Functions xy x +(1  )y c(x)c(x) c(y)c(y) c(x) + (1  )c(y) c( x +(1  )y) S  RnS  Rn a convex set c:S  Rc:S  R a convex function if c( x +(1  )y)  c(x) + (1  )c(y),0   1 Every linear function is convex.

Lemma S c(x)c(x) t a convex set a convex function on S a real number is convex. Pf) Let x, y  S t x +(1  )y  S c( x +(1  )y)  c(x) + (1  )c(y)  t + (1  )t = t x +(1  )y  S t

Level Contours c = 1 c = 2 c = 3 c = 4 c = 5

Concave Functions S  RnS  Rn a convex set c:S  Rc:S  R a concave function if  c is a convex Every linear function is concave as well as convex.

Optimization Problems Convex Programming Problems 大同大學資工所 智慧型多媒體研究室

Theorem (F, c) an instance of optimization problem a convex set a convex function Define is exact for every  > 0.

Let x be a local minimum w.r.t. N  for any fixed  > 0. Let y  F be any other feasible point. Theorem Pf) x F y Next, we now want to show that c(y)  c(x).

Let x be a local minimum w.r.t. N  for any fixed  > 0. Let y  F be any other feasible point.   < <1 such that Since c is convex, we have Therefore, Theorem Pf) x F y z

Convex Programming Problems (F, c) Defined by Convex function an instance of optimization problem Important property: Local minimum  Global Minimum Concave functions

Convexity of Feasible Set (F, c) Defined by Convex function an instance of optimization problem Important property: Local minimum  Global Minimum Concave functions

Convex Programming Problems (F, c) Defined by Convex function an instance of optimization problem Important property: Local minimum  Global Minimum Concave functions Convex

Theorem In a convex programming problem, every point locally optimal with respect to the Euclidean distance neighborhood N  is also global optimal.