Convex Programming Brookes Vision Reading Group

Huh? What is convex ??? What is programming ??? What is convex programming ???

Huh? What is convex ??? What is programming ??? What is convex programming ???

Convex Function f(t x + (1-t) y) <= t f(x) + (1-t) f(y)

Convex Function Is a linear function convex ???

Convex Set Region above a convex function is a convex set.

Convex Set Is the set of all positive semidefinite matrices convex??

Huh? What is convex ??? What is programming ??? What is convex programming ???

Programming Objective function to be minimized/maximized. Constraints to be satisfied. Example Objective function Constraints

Example Feasible region Vertices Objective function Optimal solution

Huh? What is convex ??? What is programming ??? What is convex programming ???

Convex Programming Convex optimization function Convex feasible region Why is it so important ??? Global optimum can be found in polynomial time. Many practical problems are convex Non-convex problems can be relaxed to convex ones.

Convex Programming Convex optimization function Convex feasible region Examples ??? Linear Programming Refer to Vladimir/Pushmeets reading group Second Order Cone Programming What ??? Semidefinite Programming All this sounds Greek and Latin !!!!

Outline Convex Optimization – Second Order Cone Programming (SOCP) – Semidefinite Programming (SDP) Non-convex optimization – SDP relaxations – SOCP relaxations Optimization Algorithms –Interior Point Method for SOCP –Interior Point Method for SDP 2 out of 3 is not bad !!!

Outline Convex Optimization – Second Order Cone Programming (SOCP) – Semidefinite Programming (SDP) Non-convex optimization – SDP relaxations – SOCP relaxations Optimization Algorithms –Interior Point Method for SOCP –Interior Point Method for SDP

Second Order Cone || u || < t u - vector of dimension d-1 t - scalar Cone lies in d dimensions Second Order Cone defines a convex set Example: Second Order Cone in 3D x 2 + y 2 <= z 2

Hmmm ICE CREAM !!

Second Order Cone Programming Minimize f T x Subject to || A i x+ b i || <= c i T x + d i i = 1, …, L Linear Objective Function Affine mapping of SOC Constraints are SOC of n i dimensions Feasible regions are intersections of conic regions

Example

Why SOCP ?? A more general convex problem than LP – LP SOCP Fast algorithms for finding global optimum – LP - O(n 3 ) – SOCP - O(L 1/2 ) iterations of O(n 2 n i ) Many standard problems are SOCP-able

SOCP-able Problems Convex quadratically constrained quadratic programming Sum of norms Maximum of norms Problems with hyperbolic constraints

SOCP-able Problems Convex quadratically constrained quadratic programming Sum of norms Maximum of norms Problems with hyperbolic constraints

QCQP Minimize x T P 0 x + 2 q 0 T x + r 0 Subject to x T P i x + 2 q i T x + r i P i >= 0 || P 0 1/2 x + P 0 -1/2 x || 2 + r 0 -q 0 T P 0 -1 p 0

QCQP Minimize x T P 0 x + 2 q 0 T x + r 0 Subject to x T P i x + 2 q i T x + r i Minimize t Subject to || P 0 1/2 x + P 0 -1/2 x || < = t || P 0 1/2 x + P 0 -1/2 x || < = (r 0 -q 0 T P 0 -1 p 0 ) 1/2

SOCP-able Problems Convex quadratically constrained quadratic programming Sum of norms Maximum of norms Problems with hyperbolic constraints

Sum of Norms Minimize || F i x + g i || Minimize t i Subject to || F i x + g i || <= t i Special Case: L-1 norm minimization

SOCP-able Problems Convex quadratically constrained quadratic programming Sum of norms Maximum of norms Problems with hyperbolic constraints

Maximum of Norms Minimize max || F i x + g i || Minimize t Subject to || F i x + g i || <= t Special Case: L-inf norm minimization

You werent expecting a question, were you ??

SOCP-able Problems Convex quadratically constrained quadratic programming Sum of norms Maximum of norms Problems with hyperbolic constraints

Hyperbolic Constraints w 2 <= xy x >= 0, y >= 0 || [2w; x-y] || <= x+y

Lets see if everyone was awake !

Outline Convex Optimization – Second Order Cone Programming (SOCP) – Semidefinite Programming (SDP) Non-convex optimization – SDP relaxations – SOCP relaxations Optimization Algorithms –Interior Point Method for SOCP –Interior Point Method for SDP

Semidefinite Programming Minimize C X Subject to A i X = b i X >= 0 Linear Objective Function Linear Constraints Linear Programming on Semidefinite Matrices

Why SDP ?? A more general convex problem than SOCP – LP SOCP SDP Generality comes at a cost though – SOCP - O(L 1/2 ) iterations of O(n 2 n i ) – SDP - O((n i ) 1/2 ) iterations of O(n 2 n i 2 ) Many standard problems are SDP-able

SDP-able Problems Minimizing the maximum eigenvalue Class separation with ellipsoids

SDP-able Problems Minimizing the maximum eigenvalue Class separation with ellipsoids

Minimizing the Maximum Eigenvalue Matrix M(z) To find vector z* such that max is minimized. Let max (M(z)) <= n max (M(z)-nI) <= 0 min (nI - M(z)) >= 0 nI - M(z) >= 0

Minimizing the Maximum Eigenvalue Matrix M(z) To find vector z* such that max is minimized. Max -n nI - M(z) >= 0

SDP-able Problems Minimizing the maximum eigenvalue Class separation with ellipsoids

Outline Convex Optimization – Second Order Cone Programming (SOCP) – Semidefinite Programming (SDP) Non-convex optimization – SDP relaxations – SOCP relaxations Optimization Algorithms –Interior Point Method for SOCP –Interior Point Method for SDP

Non-Convex Problems Minimize x T Q 0 x + 2q 0 T x + r 0 Subject to x T Q i x + 2q i T x + r i < = 0 Q i >= 0 => Convex Redefine x in homogenous coordinates. y = (1; x) Non-Convex Quadratic Programming Problem !!!

Non-Convex Problems Minimize x T Q 0 x + 2q 0 T x + r 0 Subject to x T Q i x + 2q i T x + r i < = 0 Minimize y T M 0 y Subject to y T M i y < = 0 M i = [ r i q i T ; q i Q i ] Lets solve this now !!!

Non-Convex Problems Problem is NP-hard. Lets relax the problem to make it convex. Pray !!!

Outline Convex Optimization – Second Order Cone Programming (SOCP) – Semidefinite Programming (SDP) Non-convex optimization – SDP relaxations – SOCP relaxations Optimization Algorithms –Interior Point Method for SOCP –Interior Point Method for SDP

SDP Relaxation Minimize y T M 0 y Subject to y T M i y < = 0 Minimize M 0 Y Subject to M i Y < = 0 Y = yy T Bad Constraint !!!! No donut for you !!!

SDP Relaxation Minimize y T M 0 y Subject to y T M i y < = 0 Minimize M 0 Y Subject to M i Y < = 0 Y >= 0 SDP Problem Nothing left to do …. but Pray Note that we have squared the number of variables.

Example - Max Cut Graph: G=(V,E) Maximum-Cut

Graph: G=(V,E) Maximum-Cut Example - Max Cut - x i = -1 - x i = +1

Graph: G=(V,E) Maximum-Cut Example - Max Cut Alright !!! So its an integer programming problem !!! Doesnt look like quadratic programming to me !!!

Max Cut as an IQP Max Cut problem can be written as Naah !! Lets get it into the standard quadratic form.

Max Cut as an IQP Max Cut problem can be written as Naah !! Lets get it into the standard quadratic form.

Solving Max Cut using SDP Relaxations To the white board. (You didnt think Ill prepare slides for this, did you??)

Outline Convex Optimization – Second Order Cone Programming (SOCP) – Semidefinite Programming (SDP) Non-convex optimization – SDP relaxations – SOCP relaxations Optimization Algorithms –Interior Point Method for SOCP –Interior Point Method for SDP

SOCP Relaxation Minimize y T M 0 y Subject to y T M i y < = 0 Minimize M 0 Y Subject to M i Y < = 0 Y >= 0 Remember Y = [1 x T ; x X] X - xx T >= 0

SOCP Relaxation Say youre given C = { C 1, C 2, … C n } such that C j >= 0 C j (X - xx T ) >= 0 (Ux) T (Ux) <= C j X Wait.. Isnt this a hyperbolic constraint Therefore, its SOCP-able.

SOCP Relaxation Minimize y T M 0 y Subject to y T M i y < = 0 Minimize Q 0 X + 2q 0 T x + r 0 Subject to Q i X + 2q i T x + r i < = 0 C j (X - xx T ) >= 0 C j C

SOCP Relaxation If C is the infinite set of all semidefinite matrices SOCP Relaxation = SDP Relaxation If C is finite, SOCP relaxation is looser than SDP relaxation. Then why SOCP relaxation ??? Efficiency - Accuracy Tradeoff

Choice of C Remember we had squared the number of variables. Lets try to reduce them with our choice of C. For a general problem - Kim and Kojima Using the structure of a specific problem - e.g. Muramatsu and Suzuki for Max Cut

Choice of C Minimize c T x Subject to Q i X + 2q i T x + r i < = 0 Q X + 2q T x + r <= 0 Q = n i u i u i T Let 1 >= 2 >= …. k >= 0 >= k+1 >= n

Choice of C C = Q + = k i u i u i T Q X + 2q T x + r <= 0 x T Q + x - Q + X <= 0 x T Q + x + k+1 i u i u i T X + 2q T x + r <= 0 zizi

Choice of C C = Q + = k i u i u i T x T Q + x + k+1 i z i + 2q T x + r <= 0 u i u i T i = k+1, k+2, … n x T u i u i T x - u i u i T X <= 0 Q X + 2q T x + r <= 0

Choice of C C = Q + = k i u i u i T x T Q + x + k+1 i z i + 2q T x + r <= 0 u i u i T i = k+1, k+2, … n x T u i u i T x - z i <= 0 Q X + 2q T x + r <= 0

Specific Problem Example - Max Cut e i = [0 0 …. 1 0 …0] u ij = e i + e j v ij = e i - e j C = e i e i T i = 1, …, |V| u ij u ij T (i,j) E v ij v ij T (i,j) E

Specific Problem Example - Max Cut Warning: Scary equations to follow.

Outline Convex Optimization – Second Order Cone Programming (SOCP) – Semidefinite Programming (SDP) Non-convex optimization – SDP relaxations – SOCP relaxations Optimization Algorithms –Interior Point Method for SOCP –Interior Point Method for SDP

Outline Convex Optimization – Second Order Cone Programming (SOCP) – Semidefinite Programming (SDP) Non-convex optimization – SDP relaxations – SOCP relaxations Optimization Algorithms –Interior Point Method for SOCP –Interior Point Method for SDP

Back to work now !!!