Optimization - Lecture 4, Part 1 M. Pawan Kumar Slides available online

“Recap” of Linear Programming

Polyhedron Ax ≤ b A : m x n matrix b: m x 1 vector

Bounded Polyhedron = Polytope Ax ≤ b A : m x n matrix b: m x 1 vector

Vertex z is a vertex of P = {x, Ax ≤ b} z is not a convex combination of two points in P There does not exist x, y ∈ P and 0 < λ < 1 x ≠ z and y ≠ z such that z = λ x + (1-λ) y

Vertex z is a vertex of P = {x, Ax ≤ b} A z is a submatrix of A Contains all rows of A such that a i T z = b i Recall A is an m x n matrix

Vertex z is a vertex of P Rank of A z = n Proof? See “hidden” slides

Proof Sketch: Necessity Let z be a vertex of P Suppose rank(A z ) < nA z c = 0 for some c ≠ 0 Then there exists a d > 0 such that z - dc ∈ Pz + dc ∈ P Contradiction

Proof Sketch: Sufficiency Suppose rank(A z ) = n but z is not a vertex of P z = (x+y)/2 for some x, y ∈ P, x ≠ y ≠ z For each a in A z Implies A z (x-y) = 0Contradiction a T x ≤ b = a T za T y ≤ b = a T z

Linear Programming Duality Solving the LP Outline

Linear Program Maximize a linear function Over a polyhedral feasible region s.t. A x ≤ b max x c T x A: m x n matrix b: m x 1 vector c: n x 1 vector Objective function Constraints x: n x 1 vector

Example 4x 1 – x 2 ≤ 8 max x x 1 + x 2 2x 1 + x 2 ≤ 10 5x 1 - 2x 2 ≥ -2 x 1 ≥ 0 x 2 ≥ 0 s.t. What is c? A?b?

Example x 1 ≥ 0 x 2 ≥ 0

Example 4x 1 – x 2 = 8 x 1 ≥ 0 x 2 ≥ 0

Example 4x 1 – x 2 ≤ 8 x 1 ≥ 0 x 2 ≥ 0

Example 4x 1 – x 2 ≤ 8 2x 1 + x 2 ≤ 10 x 1 ≥ 0 x 2 ≥ 0

Example 4x 1 – x 2 ≤ 8 2x 1 + x 2 ≤ 10 5x 1 - 2x 2 ≥ -2 x 1 ≥ 0 x 2 ≥ 0

Example 4x 1 – x 2 ≤ 8 2x 1 + x 2 ≤ 10 5x 1 - 2x 2 ≥ -2 x 1 ≥ 0 x 2 ≥ 0 max x x 1 + x 2 x 1 + x 2 = 0

Example 4x 1 – x 2 ≤ 8 2x 1 + x 2 ≤ 10 5x 1 - 2x 2 ≥ -2 x 1 ≥ 0 x 2 ≥ 0 max x x 1 + x 2 x 1 + x 2 = 8 Optimal solution

Linear Programming Duality Solving the LP Outline

Example 2x 1 + 2x 2 + 5x 3 ≤ 24 max x 3x 1 + x 2 + 2x 3 4x 1 + x 2 + 2x 3 ≤ 36 -x 1 ≤ 0, -x 2 ≤ 0, -x 3 ≤ 0 x 1 + x 2 + 3x 3 ≤ 30 s.t. 3 x 2 x 7 x Scale the constraints, add them up 3x 1 + x 2 + 2x 3 ≤ 90 Upper bound on solution

Example 2x 1 + 2x 2 + 5x 3 ≤ 24 max x 3x 1 + x 2 + 2x 3 4x 1 + x 2 + 2x 3 ≤ 36 -x 1 ≤ 0, -x 2 ≤ 0, -x 3 ≤ 0 x 1 + x 2 + 3x 3 ≤ 30 s.t. 1 x Scale the constraints, add them up 3x 1 + x 2 + 2x 3 ≤ 36 Upper bound on solution

Example 2x 1 + 2x 2 + 5x 3 ≤ 24 max x 3x 1 + x 2 + 2x 3 4x 1 + x 2 + 2x 3 ≤ 36 -x 1 ≤ 0, -x 2 ≤ 0, -x 3 ≤ 0 x 1 + x 2 + 3x 3 ≤ 30 s.t. 1 x Scale the constraints, add them up 3x 1 + x 2 + 2x 3 ≤ 36 Tightest upper bound?

Example 2x 1 + 2x 2 + 5x 3 ≤ 24 max x 3x 1 + x 2 + 2x 3 4x 1 + x 2 + 2x 3 ≤ 36 -x 1 ≤ 0, -x 2 ≤ 0, -x 3 ≤ 0 x 1 + x 2 + 3x 3 ≤ 30 s.t. y4y4 y1y1 y2y2 y3y3 y5y5 y6y6 y 1, y 2, y 3, y 4, y 5, y 6 ≥ 0 We should be able to add up the inequalities

Example 2x 1 + 2x 2 + 5x 3 ≤ 24 max x 3x 1 + x 2 + 2x 3 4x 1 + x 2 + 2x 3 ≤ 36 -x 1 ≤ 0, -x 2 ≤ 0, -x 3 ≤ 0 x 1 + x 2 + 3x 3 ≤ 30 s.t. y4y4 y1y1 y2y2 y3y3 y5y5 y6y6 -y 1 + y 4 + 2y 5 + 4y 6 = 3 Coefficient of x 1 should be 3

Example 2x 1 + 2x 2 + 5x 3 ≤ 24 max x 3x 1 + x 2 + 2x 3 4x 1 + x 2 + 2x 3 ≤ 36 -x 1 ≤ 0, -x 2 ≤ 0, -x 3 ≤ 0 x 1 + x 2 + 3x 3 ≤ 30 s.t. y4y4 y1y1 y2y2 y3y3 y5y5 y6y6 -y 2 + y 4 + 2y 5 + y 6 = 1 Coefficient of x 2 should be 1

Example 2x 1 + 2x 2 + 5x 3 ≤ 24 max x 3x 1 + x 2 + 2x 3 4x 1 + x 2 + 2x 3 ≤ 36 -x 1 ≤ 0, -x 2 ≤ 0, -x 3 ≤ 0 x 1 + x 2 + 3x 3 ≤ 30 s.t. y4y4 y1y1 y2y2 y3y3 y5y5 y6y6 -y 3 + 3y 4 + 5y 5 + 2y 6 = 2 Coefficient of x 3 should be 2

Example 2x 1 + 2x 2 + 5x 3 ≤ 24 max x 3x 1 + x 2 + 2x 3 4x 1 + x 2 + 2x 3 ≤ 36 -x 1 ≤ 0, -x 2 ≤ 0, -x 3 ≤ 0 x 1 + x 2 + 3x 3 ≤ 30 s.t. y4y4 y1y1 y2y2 y3y3 y5y5 y6y6 min y 30y y y 6 Upper bound should be tightest

Dual min y 30y y y 6 s.t. y 1, y 2, y 3, y 4, y 5, y 6 ≥ 0 -y 1 + y 4 + 2y 5 + 4y 6 = 3 -y 2 + y 4 + 2y 5 + y 6 = 1 -y 3 + 3y 4 + 5y 5 + 2y 6 = 2 Original problem is called primal Dual of dual is primal

Dual s.t. A x ≤ b max x c T x

Dual - y T (A x – b)max x c T x min y≥0 KKT Condition?A T y = c min y≥0 bTybTy s.t. A T y = c

min y≥0 bTybTy s.t. A T y = c s.t. A x ≤ b max x c T x Primal Dual

Strong Duality min y≥0 bTybTy s.t. A T y = c s.t. A x ≤ b max x c T x Primal Dual p = d = If p ≠ ∞ or d ≠ ∞, then p = d. Skipping the proof Think back to the intuition of dual

Question s.t. A 1 x ≤ b 1 max x c T x A 2 x ≥ b 2 A 3 x = b 3 Dual?

Weak Duality Let x be a feasible solution of the primal c T x ≤ b T y Objective of any primal ≤ Objective of any dual Think back to the intuition of dual Let y be a feasible solution of the dual

Linear Programming Duality Solving the LP Outline

Graphical Solution 4x 1 – x 2 ≤ 8 2x 1 + x 2 ≤ 10 5x 1 - 2x 2 ≥ -2 x 1 ≥ 0 x 2 ≥ 0 max x x 1 + x 2 x 1 + x 2 = 8 Optimal solution at a vertex

Graphical Solution 4x 1 – x 2 ≤ 8 2x 1 + x 2 ≤ 10 5x 1 - 2x 2 ≥ -2 x 1 ≥ 0 x 2 ≥ 0 max x x 1 x 1 = 3 Optimal solution at a vertex

Graphical Solution 4x 1 – x 2 ≤ 8 2x 1 + x 2 ≤ 10 5x 1 - 2x 2 ≥ -2 x 1 ≥ 0 x 2 ≥ 0 max x x 2 x 2 = 6 Optimal solution at a vertex

Graphical Solution 4x 1 – x 2 ≤ 8 2x 1 + x 2 ≤ 10 5x 1 - 2x 2 ≥ -2 x 1 ≥ 0 x 2 ≥ 0 An optimal solution always at a vertex Proof? max x c T x

Dantzig (1951): Simplex Method –Search over vertices of the polyhedra –Worst-case complexity is exponential –Smoothed complexity is polynomial Khachiyan (1979, 1980): Ellipsoid Method –Polynomial time complexity –LP is a P optimization problem Karmarkar (1984): Interior-point Method –Polynomial time complexity –Competitive with Simplex Method Solving the LP

Plenty of standard software available Mosek ( –C++ API –Matlab API –Python API –Free academic license Solving the LP

Dantzig (1951): Simplex Method –Search over vertices of the polyhedra –Worst-case complexity is exponential –Smoothed complexity is polynomial Khachiyan (1979, 1980): Ellipsoid Method –Polynomial time complexity –LP is a P optimization problem Karmarkar (1984): Interior-point Method –Polynomial time complexity –Competitive with Simplex Method Solving the LP

Optimization vs. Feasibility s.t. A x ≤ b max x c T x Optimization Feasibility asks if there exists an x such that c T x ≥ K A x ≤ b Optimization via binary search on K Feasible solution For a given K

Feasibility via Ellipsoid Method Feasible region of LP

Feasibility via Ellipsoid Method Ellipsoid containing feasible region of LP

Feasibility via Ellipsoid Method Centroid of ellipsoid

Feasibility via Ellipsoid Method Separating hyperplane for centroid

Feasibility via Ellipsoid Method Smallest ellipsoid containing “truncated” ellipsoid

Feasibility via Ellipsoid Method Centroid of ellipsoid

Feasibility via Ellipsoid Method Separating hyperplane for centroid

Feasibility via Ellipsoid Method Smallest ellipsoid containing “truncated” ellipsoid

Feasibility via Ellipsoid Method Centroid of ellipsoid

Feasibility via Ellipsoid Method Terminate when feasible solution is found

Separating hyperplane in polynomial time –Check each of the ‘m’ LP constraints in O(n) time New ellipsoid in polynomial time –Shor (1971), Nemirovsky and Yudin (1972) Polynomial iterations (Khachiyan) –Volume of ellipsoid reduces exponentially Only requires a separation oracle –Constraint matrix A can be very large Ellipsoid Method Useful Fact !!

Questions?