A Combinatorial, Primal-Dual Approach to Semidefinite Programs Satyen Kale Microsoft Research Joint work with Sanjeev Arora Princeton University

The Ubiquity of Semidefinite Programming Max Cut [GW’95] Balanced Partitioning [ARV’04] Control Theory dx(t)/dt = Ax(t) Graph Coloring [KMS’98, ACC’06] SDP (a Ç :b) Æ (:a Ç c) Æ (a Ç b) Æ (:c Ç b) Constraint Satisfaction [ACMM’05] 1 0 0 0 1 1 0 1

Primal-Dual algorithms Algorithms for SDP Combinatorial, Primal-Dual algorithms MISSING Interior point methods [NN’90, A’95]: O(√m(n3 + m)) time Ellipsoid method [GLS’81]: O(n8) iterations A1 ² X ¸ b1  Am ² X ¸ bm i wi(Ai ² X – bi) ¸ 0 Lagrangian Relaxation [KL’95], [AHK’05]: Reduction to eigenvectors poly(1/) dependence on , limits applicability (e.g. Sparsest Cut)

Primal-Dual algorithms for LP: Multicommodity Flows Unit capacities Minimize colors Objective: Maximize total flow, while respecting edge capacities

Primal-Dual algorithms for LP: Multicommodity Flows 1 +  1 Edge weights we = 1 Repeat until some e reaches capacity: Find shortest path p Route 2/log(m) flow on p Update we for all e 2 p as we à we ¢ (1 + ) Output flow. 1 f_e explanation, shortest path over all commodities Thm [GK’98]: Stops in Õ(m) rounds with (1 – 2)¢OPT flow. Total running time is Õ(m2).

Primal-Dual algorithms for LP: Multicommodity Flows Combinatorial Multiplicative Weights Update Rule Edge weights we = 1 Repeat until some e reaches capacity: Find shortest path p Route 2/log(m) flow on p Update we for all e 2 p as we à we ¢ (1 + ) Output flow. f_e explanation, shortest path over all commodities Thm [GK’98]: Stops in Õ(m) rounds with (1 – 2)¢OPT flow. Total running time is Õ(m2).

Starting point for our work: Analogous primal-dual algorithms for SDP? Difficulties: Positive semidefiniteness hard to maintain Rounding algorithms exploit geometric structure (e.g. negative-type metrics) Matrix operations inefficient to implement Our work: a generic scheme that yields fast, combinatorial, primal-dual algorithms for various optimization problems using SDP

Our Results: Primal-Dual algorithms Problem Previous best: O(√log n) apx Our alg.: O(log n) apx Undirected Sparsest Cut Õ(n2) Õ(m + n1.5) Balanced Sep. Directed Õ(n4.5) Õ(m1.5 + n2+) Õ(m1.5) Min UnCut Õ(n3) --- Min 2CNF Deletion Õ(nm1.5 + n3) [AHK’04] [AHK’04]

Our Results: Primal-Dual algorithms Problem Previous best: O(√log n) apx Our alg.: O(log n) apx Undirected Sparsest Cut Õ(n2) Õ(m + n1.5) Balanced Sep. Directed Õ(n4.5) Õ(m1.5 + n2+) Õ(m1.5) Min UnCut Õ(n3) --- Min 2CNF Deletion Õ(nm1.5 + n3) [AHK’04] Cannot be achieved by LPs even! [AHK’04]

Our Results: Near-linear time algorithm for Max Cut Õ(n + m) time algorithm to approximate Max Cut SDP to (1 + o(1)) factor Previous best: Õ(n2) time algorithm by Klein, Lu ‘95

Prototype MW for LPs: Winnow [L’88] a1 ¢ x ¸ a2 ¢ x ¸  am ¢ x ¸ x ¸ 0 i xi = 1 -   Thm: Finds x in O(2log(n)/2) iterations.  = maxij |aij| MW for LP: Init: x = (1/n, …, 1/n) Update: xj = xj ¢ exp(- aij)/  = norm. factor x Oracle Find i s.t. ai ¢ x < - ai Convex comb. of constraints allowed: i yi ai ¢ x < -, where yi ¸ 0, i yi = 1. Hence Primal-Dual. MW algorithm: boosting, hard-core sets, zero-sum games, flows, portfolio, … Analysis uses  as potential fn.

Prototype Matrix MW for SDPs A1 ² X ¸ A2 ² X ¸  Am ² X ¸ X º 0 Tr(X) = 1 -   Thm: Finds X in O(2log(n)/2) iterations.  = maxi kAik Matrix MW for SDP: Init: X = I/n Update: X = exp(- ts=1 Ais)/  = norm. factor X Oracle Find it s.t. Ait ² X < - Ait Convex comb. of constraints allowed: i yi Ai²X < -, where yi ¸ 0, i yi = 1. Hence Primal-Dual. Analysis uses  as potential fn.

The Matrix Exponential exp(A) = I + A + A2/2! + A3/3! + … Always PSD: exp(A) º 0 Tricky beast: exp(A+B) = exp(A) exp(B) Computation: O(n3) time Usually, can use J-L lemma Only need exp(A)v products: Õ(m) time Golden-Thompson inequality: Tr(exp(A+B)) · Tr(exp(A)exp(B)) Matrix MW for SDP: Init: X = I/n Update: X = exp(- ts=1 Ais)/  = norm. factor

Approximation via SDP Relaxations Input Reduction 0-1 Quadratic Program Relaxation Rounding Algorithm Primal-Dual SDP SDP Opt v1 vn v2 SDP: Vector vars SDP Solver

Primal-Dual SDP Framework Input Reduction 0-1 Quadratic Program Rounding Algorithm Relaxation Primal-Dual SDP v1 vn v2 y1,…, ym Matrix MW SDP: Vector vars X

Approximating Balanced Separator G = (V, E) Cut (S, S’) is c-balanced if |S|, |S’| ¸ cn Min c-Balanced Separator: c-balanced cut of min capacity Numerous applications: Divide-and-conquer algorithms Markov chains Geometric embeddings Clustering Layout problems … S S’

SDP for Balanced Separator G = (V, E) ¼kvi – vjk2 = 0 if i, j on same side, = 1 otherwise SDP: min i,j 2 E ¼kvi – vjk2 8i: kvik2 = 1 8ijk: kvi–vjk2 + kvj–vkk2 ¸ kvi–vkk2 i, j ¼kvi – vjk2 ¸ c(1-c)n2 vi = -1 vi = 1

8ijk: kvi–vjk2 + kvj–vkk2 ¸ kvi–vkk2 Implementing Oracle Primal: min i,j 2 E ¼kvi – vjk2 8i: kvik2 = 1 8ijk: kvi–vjk2 + kvj–vkk2 ¸ kvi–vkk2 i, j ¼kvi – vjk2 ¸ c(1-c)n2 Oracle: given X (thus, vi), : check first and third constraints  inequalities: dual vars , multicommodity flow s.t. total flow from any node is · d no capacity is exceeded i,j fijkvi – vjk2 ¸  Bounds eigenvalues via demand graph Laplacian Thm: Given vi, : Max-flow ) desired flow or cut of value O(log(n)¢). Multicommodity flow ) desired flow or cut of value O(√log(n)¢). fij = total flow between i, j

Conclusions and Future Work Matrix MW algorithm: more applications in Solving SDPs: e.g. Min Linear Arrangement Quantum algorithms: density matrix is a central concept Learning: e.g. online variance minimization [WK COLT’06], online PCA [WK NIPS’06] Other applications? Linear time algorithm for Sparsest Cut? Our algorithm runs in Õ(m + n1.5) time for an O(log n) approximation

Thank you!