Polyhedral Optimization Lecture 1 – Part 2 M. Pawan Kumar Slides available online

Computational Complexity Convex Sets and Functions Linear Programming Outline

f(n)  O(g(n)) There exists k > 0 and n 0 such that f(n) ≤ k g(n) for all n ≥ n 0 f(n) = 5n 2 + 3nlog(n) g(n) = n 2 f(n) = 5n 3 + 3nlog(n) g(n) = n 2 ✔✗

f(n)  Ω(g(n)) There exists k > 0 and n 0 such that f(n) ≥ k g(n) for all n ≥ n 0 f(n) = 5n 2 + 3nlog(n) g(n) = n 2 f(n) = 5n 3 + 3nlog(n) g(n) = n 2 ✔✔

f(n)  Θ(g(n)) There exists k 1 > 0, k 2 > 0 and n 0 such that k 1 g(n) ≤ f(n) ≤ k 2 g(n) for all n ≥ n 0 f(n) = 5n 2 + 3nlog(n) g(n) = n 2 ✔

Computational Complexity –Decision Problems (Answered by ‘yes’ or ‘no’) –P and NP –NP-Complete –NP-Hard –Combinatorial Optimization Problems Convex Sets and Functions Linear Programming Outline

Finite set Σ called alphabet of size ≥ 2 –{0,1} –{a,b,c,d,e,…,x,y,z} Set Σ* of all finite length strings (called words) –0, 1, 00, 01, 10, 11, 000,…. –discrete, optimization Size of word size(w) = number of letters –size(00) = 2 –size(discrete) = 8 Decision Problem

Problem Π is a subset of Σ* –All words with the answer “yes” Informal problem –Given input word x  Σ*, does x  Π ? Polynomial-time solvable problem Π –There exists a polynomial-time algorithm for the informal problem –Polynomial in size(x) Decision Problem

Minimum Sum Subset Given a set of ‘n’ real numbers S Is there a non-empty subset X ⊆ S such that ∑ x ∈ X x ≤ C

Minimum Sum Subset Real numbers  Σ {  Σ }  Σ,  Σ {1, 2, 3, 4, 5, 6, 7}, -5  Σ* ∉ Π∉ Π

Minimum Sum Subset Real numbers  Σ {  Σ }  Σ,  Σ {1, 2, 3, 4, 5, 6, 7}, 5  Σ*  Π Π

Subset Sum Given a set of ‘n’ real numbers S Is there a non-empty subset X ⊆ S such that ∑ x ∈ X x = 0

Subset Sum Real numbers  Σ {  Σ }  Σ,  Σ {1, 2, 3, 4, 5, 6, 7}  Σ* ∉ Π∉ Π

Subset Sum Real numbers  Σ {  Σ }  Σ,  Σ {1, 2, 3, 4, 5, 6, -7}  Σ*  Π Π

Shortest Path Given graph G and and a path P from u to v Is the path a shortest u-v path, which minimizes ∑ e ∈ P length(e)

Shortest Path v 0,v 1,v 2 … v n  Σ {  Σ 0,1,2, … N  Σ }  Σ,  Σ {v 0,v 1,v 2,v 3,v 4,v 5 }, {{v 0,v 1 },{v 0,v 3 },{v 1,v 2 },{v 2,v 3 },{v 2,v 4 },{v 3,v 4 }}, {3,5,5,4,6,1}, v1v1 v1v1 v0v0 v0v0 v2v2 v2v2 v3v3 v3v3 v4v4 v4v {v 0,v 3,v 4,v 2 }  Σ*

Shortest Path v 0,v 1,v 2 … v n  Σ {  Σ 0,1,2, … N  Σ }  Σ,  Σ {v 0,v 1,v 2,v 3,v 4,v 5 }, {{v 0,v 1 },{v 0,v 3 },{v 1,v 2 },{v 2,v 3 },{v 2,v 4 },{v 3,v 4 }}, {3,5,5,4,6,1}, v1v1 v1v1 v0v0 v0v0 v2v2 v2v2 v3v3 v3v3 v4v4 v4v {v 0,v 3,v 2 }  Σ*

Shortest Path v 0,v 1,v 2 … v n  Σ {  Σ 0,1,2, … N  Σ }  Σ,  Σ {v 0,v 1,v 2,v 3,v 4,v 5 }, {{v 0,v 1 },{v 0,v 3 },{v 1,v 2 },{v 2,v 3 },{v 2,v 4 },{v 3,v 4 }}, {3,5,5,4,6,1}, v1v1 v1v1 v0v0 v0v0 v2v2 v2v2 v3v3 v3v3 v4v4 v4v {v 0,v 1,v 2 }  Σ*

Shortest Path v 0,v 1,v 2 … v n  Σ {  Σ 0,1,2, … N  Σ }  Σ,  Σ {v 0,v 1,v 2,v 3,v 4,v 5 }, {{v 0,v 1 },{v 0,v 3 },{v 1,v 2 },{v 2,v 3 },{v 2,v 4 },{v 3,v 4 }}, {3,5,5,4,6,1}, v1v1 v1v1 v0v0 v0v0 v2v2 v2v2 v3v3 v3v3 v4v4 v4v {v 0,v 1,v 2 }  Π

Computational Complexity –Decision Problems –P and NP –NP-Complete –NP-Hard –Combinatorial Optimization Problems Convex Sets and Functions Linear Programming Outline

P Polynomial-time solvable problem Π –There exists a polynomial-time algorithm that decides whether x  Σ* belongs to Π or not. –Polynomial in size(x) P = {all Π, Π is polynomial time solvable} For example –Minimum Sum Subset Problem

NP NP is not “not polynomial” NP stands for non-deterministic polynomial Easy to verify “yes” words

NP Π  NP –There exists a problem Π’  P –There exists a polynomial p –There exists an x, size(x) ≤ p(size(w)) such that –w  Π if and only if wx  Π’ Polynomial-time checkable ‘certificate’ For example –Subset Sum Problem

NP Relationship between P and NP? P is a subset of NP Is P = NP? –Open Problem –One of Clay Institute’s Millennium Prize problems

If P = NP … If P=NP, then the world would be a profoundly different place than we usually assume it to be. There would be no special value in "creative leaps," no fundamental gap between solving a problem and recognizing the solution once it's found. Everyone who could appreciate a symphony would be Mozart; everyone who could follow a step-by-step argument would be Gauss; … Scott Aaronson

Computational Complexity –Decision Problems –P and NP –NP-Complete –NP-Hard –Combinatorial Optimization Problems Convex Sets and Functions Linear Programming Outline

NP-Complete Hardest problems in NP –All problems in NP can be reduced to an NP- complete problem Π is reducible to Λ –There exists a polynomial time algorithm –Given w, returns x –w  Π if and only if x  Λ If Λ  P then Π  P

NP-Complete Hardest problems in NP –All problems in NP can be reduced to an NP- complete problem Π is reducible to Λ –There exists a polynomial time algorithm –Given w, returns x –w  Π if and only if x  Λ If Λ  NP then Π  NP

NP-Complete Hardest problems in NP –All problems in NP can be reduced to an NP- complete problem Π is reducible to Λ –There exists a polynomial time algorithm –Given w, returns x –w  Π if and only if x  Λ If Λ  NP-complete and Λ  P, then P = NP

NP-Complete Is there an NP-Complete problem? SAT: Is a Boolean expression satisfiable? Cook’s Theorem: SAT is NP-Complete In more detail …

Alphabet Σ containing variables x 1,x 2,…,x n And special symbols (, ), ∧, ∨,~ And not containing 0 and 1 A variable is a Boolean expression Boolean Expression

Alphabet Σ containing variables x 1,x 2,…,x n And special symbols (, ), ∧, ∨,~ And not containing 0 and 1 If v and w are Boolean expressions then –(v ∧ w) is a Boolean expression Boolean Expression

Alphabet Σ containing variables x 1,x 2,…,x n And special symbols (, ), ∧, ∨,~ And not containing 0 and 1 If v and w are Boolean expressions then –(v ∨ w) is a Boolean expression Boolean Expression

Alphabet Σ containing variables x 1,x 2,…,x n And special symbols (, ), ∧, ∨,~ And not containing 0 and 1 If v and w are Boolean expressions then –~v is a Boolean expression –~w is a Boolean expression Boolean Expression

Alphabet Σ containing variables x 1,x 2,…,x n And special symbols (, ), ∧, ∨,~ And not containing 0 and 1 For example, f(x 1,x 2,…,x n ) –((x 2 ∧ x 3 ) ∨ ~(x 3 ∨ x 5 ) ∧ x 2 ) ∨ ~(x 2 ∧ x 5 ) –~(~x 2 ∨ ~x 3 ) ∧ ~x 1 Boolean Expression

f(x 1,x 2,…,x n ) is satisfiable if –there exists an assignment x 1 = α 1, …, x n = α n –where α i  {0,1} –such that f(x 1,x 2,…,x n ) = 1 The following identities hold –0 ∧ 0 = 0, 0 ∧ 1 = 0, 1 ∧ 0 = 0, 1 ∧ 1 = 1 –0 ∨ 0 = 0, 0 ∨ 1 = 1, 1 ∨ 0 = 1, 1 ∨ 1 = 1 –~0 = 1, ~1 = 0 –(0) = 0, (1) = 1 Satisfiability

Given the alphabet Σ Given the identities for 0 and 1 SAT is a subset of Σ* that is satisfiable Informal Problem –Given a Boolean expression w, is w satisfiable SAT

Karp’s 21 NP-Complete Problems If SAT is reducible to Π, Π is NP-complete SAT, 0-1 integer programming, Clique, Set Packing, Vertex Cover, Set Covering, Feedback node set, Feedback arc set, Directed Hamilton circuit, Undirected Hamilton circuit, 3-SAT, Chromatic number, Clique cover, Exact cover, Hitting set, Steiner tree, 3- dimensional matching, Knapsack, Job sequencing, Partition, Max cut

A special case of SAT Given variables x 1,x 2,…,x n  Σ –B 1 consists of x 1,~x 1,…,x n,~x n –B 2 consists of (w 1 ∨ … ∨ w k ), w i  B 1, 1≤k≤3 –B 3 consists of w 1 ∧ w 2 ∧ … ∧ w m, w i  B 2 –e.g., (x 1 ∨ ~x 2 ∨ x 3 ) ∧ (x 2 ∨ ~x 3 ∨ x 4 ) ∧ (~x 1 ∨ ~x 2 ) 3-SAT is the subset of B 3 that is satisfiable 3-SAT

x 1 = x 2 ∨ x 3 –(x 1 ∨ ~x 2 ) ∧ (x 1 ∨ ~x 3 ) ∧ (~x 1 ∨ x 2 ∨ x 3 ) x 1 = x 2 ∧ x 3 –(~x 1 ∨ x 2 ) ∧ (~x 1 ∨ x 3 ) ∧ (x 1 ∨ ~x 2 ∨ ~x 3 ) x 1 = ~x 2 –(x 1 ∨ x 2 ) ∧ (~x 1 ∨ ~x 2 ) SAT is reducible to 3-SAT

NP-Complete Problems Today, we know several such problems complete_problemshttp://en.wikipedia.org/wiki/List_of_NP- complete_problems No proof if they are in P No proof if they are not in P

Computational Complexity –Decision Problems –P and NP –NP-Complete –NP-Hard –Combinatorial Optimization Problems Convex Sets and Functions Linear Programming Outline

NP-hard Need not be in NP –No polynomial-time checkable certificate –Not even a decision problem (e.g. optimization) At least as hard as NP-complete problems Λ  NP-hard –There exists Π  NP-complete –Π is reducible to Λ

NP-hard Image Courtesy of Wikipedia

Computational Complexity –Decision Problems –P and NP –NP-Complete –NP-Hard –Combinatorial Optimization Problems Convex Sets and Functions Linear Programming Outline

Combinatorial Optimization Problem Given a discrete set of feasible solutions And a measure of the quality of each solution Find the best quality solution

Combinatorial Optimization Problem I = set of instances of a problem Given an instance i ∈ I Discrete set of feasible solutions F i Measure of quality m i : F i → Real max x m i (x), x ∈ F i

Minimum Sum Subset Given a set of ‘n’ real numbers S Find a non-empty subset X ⊆ S that minimizes ∑ x ∈ X x

Minimum Sum Subset Given a set of ‘n’ real numbers S Find a non-empty subset X ⊆ S that maximizes -∑ x ∈ X x

Minimum Sum Subset Instance Given a set of ‘n’ real numbers S Find a non-empty subset X ⊆ S that maximizes -∑ x ∈ X x {-28, 53, -58, -99, 13, 27, -55, -31, -91, 12, -87, -68} Feasible solutions F i ? Measure m i ?

Subset Sum Given a set of ‘n’ real numbers S Find a non-empty subset X ⊆ S that minimizes | ∑ x ∈ X x |

Subset Sum Instance Given a set of ‘n’ real numbers S Find a non-empty subset X ⊆ S that minimizes | ∑ x ∈ X x | {-28, 53, -58, -99, 13, 27, -55, -31, -91, 12, -87, -68} Feasible solutions F i ? Measure m i ?

Shortest Path Given graph G and vertices u and v Find the path from u to v that minimizes ∑ e ∈ P length(e)

Shortest Path Instance v1v1 v1v1 v0v0 v0v0 v2v2 v2v2 v3v3 v3v3 v4v4 v4v Feasible solutions F i ? Measure m i ?

Complexity Easy problems have polynomial-time solution P optimization problems Hard problems ∉ P optimization problems NP-hard problems

Computational Complexity Convex Sets and Functions –Convex Set –Convex Hull and Convex Cone –Polyhedron and Polytope –Convex Function Linear Programming Outline

Convex Set x1x1 x2x2 λ x 1 + (1 - λ) x 2 λ  [0,1] Line Segment Endpoints

Convex Set x1x1 x2x2 All points on the line segment lie within the set For all line segments with endpoints in the set

Non-Convex Set x1x1 x2x2

Examples of Convex Sets x1x1 x2x2 Line Segment

Examples of Convex Sets x1x1 x2x2 Line

Examples of Convex Sets Hyperplane a T x - b = 0

Examples of Convex Sets Halfspace a T x - b ≤ 0

Examples of Convex Sets Second-order Cone ||x|| ≤ t t x2x2 x1x1

Examples of Convex Sets Semidefinite Matrices {X | X 0} a T Xa ≥ 0, for all a  R n All eigenvalues of X are non-negative a T X 1 a ≥ 0 a T X 2 a ≥ 0 a T (λX 1 + (1-λ)X 2 )a ≥ 0

Operations that Preserve Convexity Intersection

Operations that Preserve Convexity Affine Transformation x  Ax + b

Separating Hyperplane Closed Convex Set C z ∉ C

Property For any closed convex set C, and z ∉ C There exists a separating hyperplane Proof?

Proof Sketch Let y = argmin x ∈ C ||z-x|| Define a = z - yb = (||z|| 2 - ||y|| 2 )/2 We can show that a T z > b We can show that a T x < b, for all x ∈ C

Proof Sketch a T z = (z-y) T z > (z-y) T z - ||z-y|| 2 /2 = b

Proof Sketch Let a T x ≥ b, for all x ∈ C a T y < a T y + ||a|| 2 /2 = b Therefore, there exists 0 < λ ≤ 1 λ < 2a T (x-y)/||x-y|| 2 Let w = λx + (1-λ)y We can show ||w-z|| 2 < ||y-z|| 2 Contradiction

Closed Convex Set C is a closed convex set C is an intersection of halfspaces The two statements are equivalent Proof?

Computational Complexity Convex Sets and Functions –Convex Set –Convex Hull and Convex Cone –Polyhedron and Polytope –Convex Function Linear Programming Outline

Convex Hull Intersection preserves convexity Given a set X Smallest convex set containing X exists Convex Hull

Convex Hull of a Convex Set

Convex Hull of Vectors

Convex Hull x belongs to the convex hull of X There exists a natural number t x = λ 1 x 1 + λ 2 x 2 + … + λ t x t xi ∈ Xxi ∈ X λ 1 + λ 2 + … + λ t = 1λ i > 0 Proof?

Convex Cone Convex Set C If x 1 and x 2 belong to C λ 1 x 1 + λ 2 x 2 ∈ C For all λ 1, λ 2 ≥ 0

Cone x belongs to the cone(X) There exists a natural number t x = λ 1 x 1 + λ 2 x 2 + … + λ t x t xi ∈ Xxi ∈ X λ i > 0 Given a set X cone(X): Smallest convex cone containing X

Computational Complexity Convex Sets and Functions –Convex Set –Convex Hull and Convex Cone –Polyhedron and Polytope –Convex Function Linear Programming Outline

Polyhedron a 1 T x ≤ b 1 a 2 T x ≤ b 2 a m T x ≤ b m......

Polyhedron Ax ≤ b A : m x n matrix b: n x 1 vector

Bounded Polyhedron

Vertex Formal definition?

Vertex z is a vertex of P = {x, Ax ≤ b} z is not a convex combination of two points in P There does not exist x, y ∈ P and 0 < λ < 1 x ≠ z and y ≠ z such that z = λ x + (1-λ) y

Vertex z is a vertex of P = {x, Ax ≤ b} A z is a submatrix of A Contains all rows of A such that a i T z = b i Recall A is an m x n matrix

Vertex z is a vertex of P Rank of A z = n Proof?

Proof Sketch: Necessity Let z be a vertex of P Suppose rank(A z ) < nA z c = 0 for some c ≠ 0 Then there exists a d > 0 such that z - dc ∈ Pz + dc ∈ P Contradiction

Proof Sketch: Sufficiency Suppose rank(A z ) = n but z is not a vertex of P z = (x+y)/2 for some x, y ∈ P, x ≠ y ≠ z For each a in A z Implies A z (x-z) = 0Contradiction a T x ≤ b = a T za T y ≤ b = a T z

Polytope Convex hull of a finite number of vectors

Polytope and Polyhedron P is a polytope P is a bounded polyhedron Skipping the proof (available on webpage)

Computational Complexity Convex Sets and Functions –Convex Set –Convex Hull and Convex Cone –Polyhedron and Polytope –Convex Function Linear Programming Outline

Convex Function x g(x) Blue point always lies above red point x1x1 x2x2

Convex Function x g(x) g( c x 1 + (1 - c) x 2 ) ≤ c g(x 1 ) + (1 - c) g(x 2 ) x1x1 x2x2 Domain of g(.) has to be convex

Convex Function x g(x) x1x1 x2x2 -g(.) is concave g( c x 1 + (1 - c) x 2 ) ≤ c g(x 1 ) + (1 - c) g(x 2 )

Convex Function Once-differentiable functions g(y) +  g(y) T (x - y) ≤ g(x) x g(x) (y,g(y)) g(y) +  g(y) T (x - y) Twice-differentiable functions  2 g(x) 0

Convex Function and Convex Sets x g(x) Epigraph of a convex function is a convex set

Examples of Convex Functions Linear function a T x p-Norm functions (x 1 p + x 2 p + x n p ) 1/p, p ≥ 1 Quadratic functions x T Q x Q 0

Operations that Preserve Convexity Non-negative weighted sum x g 1 (x) w1w1 x g 2 (x) + w 2 + …. x T Q x + a T x + b Q 0

Operations that Preserve Convexity Pointwise maximum x g 1 (x) max x g 2 (x), Pointwise minimum of concave functions is concave

Computational Complexity Convex Sets and Functions Linear Programming Outline Examples taken from the slides of Ziv Bar-Yossef (Technion)

Linear Program Maximize a linear function Over a polyhedral feasible region s.t. A x ≤ b max x c T x A: m x n matrix b: m x 1 vector c: n x 1 vector Objective function Constraints x: n x 1 vector

Example 4x 1 – x 2 ≤ 8 max x x 1 + x 2 2x 1 + x 2 ≤ 10 5x 1 - 2x 2 ≥ -2 x 1 ≥ 0 x 2 ≥ 0 s.t. What is c? A?b?

Example x 1 ≥ 0 x 2 ≥ 0

Example 4x 1 – x 2 = 8 x 1 ≥ 0 x 2 ≥ 0

Example 4x 1 – x 2 ≤ 8 x 1 ≥ 0 x 2 ≥ 0

Example 4x 1 – x 2 ≤ 8 2x 1 + x 2 ≤ 10 x 1 ≥ 0 x 2 ≥ 0

Example 4x 1 – x 2 ≤ 8 2x 1 + x 2 ≤ 10 5x 1 - 2x 2 ≥ -2 x 1 ≥ 0 x 2 ≥ 0

Example 4x 1 – x 2 ≤ 8 2x 1 + x 2 ≤ 10 5x 1 - 2x 2 ≥ -2 x 1 ≥ 0 x 2 ≥ 0 max x x 1 + x 2 x 1 + x 2 = 0

Example 4x 1 – x 2 ≤ 8 2x 1 + x 2 ≤ 10 5x 1 - 2x 2 ≥ -2 x 1 ≥ 0 x 2 ≥ 0 max x x 1 + x 2 x 1 + x 2 = 8 Optimal solution

Computational Complexity Convex Sets and Functions Linear Programming –Duality –Complementary Slackness –Solving the LP Outline

Example 2x 1 + 2x 2 + 5x 3 ≤ 24 max x 3x 1 + x 2 + 2x 3 4x 1 + x 2 + 2x 3 ≤ 36 -x 1 ≤ 0, -x 2 ≤ 0, -x 3 ≤ 0 x 1 + x 2 + 3x 3 ≤ 30 s.t. 3 x 2 x 7 x Scale the constraints, add them up 3x 1 + x 2 + 2x 3 ≤ 90 Upper bound on solution

Example 2x 1 + 2x 2 + 5x 3 ≤ 24 max x 3x 1 + x 2 + 2x 3 4x 1 + x 2 + 2x 3 ≤ 36 -x 1 ≤ 0, -x 2 ≤ 0, -x 3 ≤ 0 x 1 + x 2 + 3x 3 ≤ 30 s.t. 1 x Scale the constraints, add them up 3x 1 + x 2 + 2x 3 ≤ 36 Upper bound on solution

Example 2x 1 + 2x 2 + 5x 3 ≤ 24 max x 3x 1 + x 2 + 2x 3 4x 1 + x 2 + 2x 3 ≤ 36 -x 1 ≤ 0, -x 2 ≤ 0, -x 3 ≤ 0 x 1 + x 2 + 3x 3 ≤ 30 s.t. 1 x Scale the constraints, add them up 3x 1 + x 2 + 2x 3 ≤ 36 Tightest upper bound?

Example 2x 1 + 2x 2 + 5x 3 ≤ 24 max x 3x 1 + x 2 + 2x 3 4x 1 + x 2 + 2x 3 ≤ 36 -x 1 ≤ 0, -x 2 ≤ 0, -x 3 ≤ 0 x 1 + x 2 + 3x 3 ≤ 30 s.t. y4y4 y1y1 y2y2 y3y3 y5y5 y6y6 y 1, y 2, y 3, y 4, y 5, y 6 ≥ 0 We should be able to add up the inequalities

Example 2x 1 + 2x 2 + 5x 3 ≤ 24 max x 3x 1 + x 2 + 2x 3 4x 1 + x 2 + 2x 3 ≤ 36 -x 1 ≤ 0, -x 2 ≤ 0, -x 3 ≤ 0 x 1 + x 2 + 3x 3 ≤ 30 s.t. y4y4 y1y1 y2y2 y3y3 y5y5 y6y6 -y 1 + y 4 + 2y 5 + 4y 6 = 3 Coefficient of x 1 should be 3

Example 2x 1 + 2x 2 + 5x 3 ≤ 24 max x 3x 1 + x 2 + 2x 3 4x 1 + x 2 + 2x 3 ≤ 36 -x 1 ≤ 0, -x 2 ≤ 0, -x 3 ≤ 0 x 1 + x 2 + 3x 3 ≤ 30 s.t. y4y4 y1y1 y2y2 y3y3 y5y5 y6y6 -y 2 + y 4 + 2y 5 + y 6 = 1 Coefficient of x 2 should be 1

Example 2x 1 + 2x 2 + 5x 3 ≤ 24 max x 3x 1 + x 2 + 2x 3 4x 1 + x 2 + 2x 3 ≤ 36 -x 1 ≤ 0, -x 2 ≤ 0, -x 3 ≤ 0 x 1 + x 2 + 3x 3 ≤ 30 s.t. y4y4 y1y1 y2y2 y3y3 y5y5 y6y6 -y 3 + 3y 4 + 5y 5 + 2y 6 = 2 Coefficient of x 3 should be 2

Example 2x 1 + 2x 2 + 5x 3 ≤ 24 max x 3x 1 + x 2 + 2x 3 4x 1 + x 2 + 2x 3 ≤ 36 -x 1 ≤ 0, -x 2 ≤ 0, -x 3 ≤ 0 x 1 + x 2 + 3x 3 ≤ 30 s.t. y4y4 y1y1 y2y2 y3y3 y5y5 y6y6 min y 30y y y 6 Upper bound should be tightest

Dual min y 30y y y 6 s.t. y 1, y 2, y 3, y 4, y 5, y 6 ≥ 0 -y 1 + y 4 + 2y 5 + 4y 6 = 3 -y 2 + y 4 + 2y 5 + y 6 = 1 -y 3 + 3y 4 + 5y 5 + 2y 6 = 2 Original problem is called primal Dual of dual is primal

Dual s.t. A x ≤ b max x c T x

Dual - y T (A x – b)max x c T x min y≥0 KKT Condition?A T y = c min y≥0 bTybTy s.t. A T y = c

min y≥0 bTybTy s.t. A T y = c s.t. A x ≤ b max x c T x Primal Dual

Strong Duality min y≥0 bTybTy s.t. A T y = c s.t. A x ≤ b max x c T x Primal Dual p = d = If p ≠ ∞ or d ≠ ∞, then p = d. Skipping the proof (available on webpage) Think back to the intuition of dual

Question s.t. A 1 x ≤ b 1 max x c T x A 2 x ≥ b 2 A 3 x = b 3 Dual?

Computational Complexity Convex Sets and Functions Linear Programming –Duality –Complementary Slackness –Solving the LP Outline

Weak Duality Let x be a feasible solution of the primal c T x ≤ b T y Objective of any primal ≤ Objective of any dual Think back to the intuition of dual Proof? Let y be a feasible solution of the dual

Proof Sketch cTxcTx = (A T y) T x When does equality hold? Optimal primal and dual solutions: x* and y* = y T Ax ≤ y T b = b T y

Equality (y*) T Ax* - (y*) T b = 0 Recall that y* ≥ 0 Recall that Ax* ≤ b If y* j > 0 a j T x* = b j Complementary slackness Necessary and sufficient for optimality

Computational Complexity Convex Sets and Functions Linear Programming –Duality –Complementary Slackness –Solving the LP Outline

Graphical Solution 4x 1 – x 2 ≤ 8 2x 1 + x 2 ≤ 10 5x 1 - 2x 2 ≥ -2 x 1 ≥ 0 x 2 ≥ 0 max x x 1 + x 2 x 1 + x 2 = 8 Optimal solution at a vertex

Graphical Solution 4x 1 – x 2 ≤ 8 2x 1 + x 2 ≤ 10 5x 1 - 2x 2 ≥ -2 x 1 ≥ 0 x 2 ≥ 0 max x x 1 x 1 = 3 Optimal solution at a vertex

Graphical Solution 4x 1 – x 2 ≤ 8 2x 1 + x 2 ≤ 10 5x 1 - 2x 2 ≥ -2 x 1 ≥ 0 x 2 ≥ 0 max x x 2 x 2 = 6 Optimal solution at a vertex

Graphical Solution 4x 1 – x 2 ≤ 8 2x 1 + x 2 ≤ 10 5x 1 - 2x 2 ≥ -2 x 1 ≥ 0 x 2 ≥ 0 An optimal solution always at a vertex Proof? max x c T x

Dantzig (1951): Simplex Method –Search over vertices of the polyhedra –Worst-case complexity is exponential –Smoothed complexity is polynomial Khachiyan (1979, 1980): Ellipsoid Method –Polynomial time complexity –LP is a P optimization problem Karmarkar (1984): Interior-point Method –Polynomial time complexity –Competitive with Simplex Method Solving the LP

Optimization vs. Feasibility s.t. A x ≤ b max x c T x Optimization Feasibility asks if there exists an x such that c T x ≥ K A x ≤ b Optimization via binary search on K Feasible solution For a given K

Feasibility via Ellipsoid Method Feasible region of LP

Feasibility via Ellipsoid Method Ellipsoid containing feasible region of LP

Feasibility via Ellipsoid Method Centroid of ellipsoid

Feasibility via Ellipsoid Method Separating hyperplane for centroid

Feasibility via Ellipsoid Method Smallest ellipsoid containing “truncated” ellipsoid

Feasibility via Ellipsoid Method Centroid of ellipsoid

Feasibility via Ellipsoid Method Separating hyperplane for centroid

Feasibility via Ellipsoid Method Smallest ellipsoid containing “truncated” ellipsoid

Feasibility via Ellipsoid Method Centroid of ellipsoid

Feasibility via Ellipsoid Method Terminate when feasible solution is found

Separating hyperplane in polynomial time –Check each of the ‘m’ LP constraints in O(n) time New ellipsoid in polynomial time –Shor (1971), Nemirovsky and Yudin (1972) Polynomial iterations (Khachiyan) –Volume of ellipsoid reduces exponentially Only requires a separation oracle –Constraint matrix A can be very large Ellipsoid Method Useful Fact !!

Questions?