Arora: SDP + Approx Survey Semidefinite Programming and Approximation Algorithms for NP-hard Problems: A Survey Sanjeev Arora Princeton University

Arora: SDP + Approx Survey NP-completeness Pragmatic Researcher Why the fuss? I am perfectly content with approximately optimal solutions. (e.g., cost within 10% of optimum) Bad News: NP-hard for many problems. (PCPs) Good news: Possible for a few problems. (Approximation Algorithms) Thousands of problems are NP-complete (TSP, Scheduling, Circuit layout, Machine Learning,..)

Arora: SDP + Approx Survey Approximation Algorithms MAX-3SAT: Given 3-CNF formula, find assignment maximizing the number of satisfied clauses. An -approximation algorithm is one that for every formula, produces in polynomial time an assignment that satisfies at least OPT/ clauses. ( > 1). Good News: [KarloffZwick97] 8/7-approximation algorithm. Bad News: [Hastad97] If P NP then for every > 0, an (8/7 - )-approximation algorithm does not exist. Many similar results...

Arora: SDP + Approx Survey Example: 2-approximation for Min Vertex Cover G= (V, E) Vertex Cover = Set of vertices that touches every edge LP Relaxation Claim: Value at least OPT/2 Proof: Rounding most Proof: On Complete Graph K n, OPT = n-1 but setting all x i = 1/2 gives feasible LP soln 0 · x i · 1

Arora: SDP + Approx Survey General Philosophy… Interested in: NP-hard Minimization Problem Value = OPT Write tractable relaxation value= Round to get a solution of cost = Approximation ratio = Integrality gap

Arora: SDP + Approx Survey SDP = Generalization of linear programming; vector programming Graph Vector Representation. (Inner products satisfy some linear constraints) Developed in 1970s as one of many flavors of nonlinear optimization. Can be solved in poly time (GLS81). Has many applications in operations research, control theory, approximation algorithms for NP-hard problems.

Arora: SDP + Approx Survey Main Idea in SDP: Simulate nonlinear programming by convex program Nonlinear program for Vertex Cover Homogenized SDP relaxation: New variable intended to stand for Vector Programs.

Arora: SDP + Approx Survey Take home message… SDP gives best approximation known for host of NP-hard problems (and algorithms can be made highly efficient): Vertex Cover Sparsest Cut and most graph partitioning problems Graph coloring Max-cut, and every Constraint Satisfaction Problem…. Analysis of these algorithms used interesting geometric ideas, which have had other applications. Compelling evidence from complexity theory that no poly-time algorithm can do better than many of these SDP-based algorithms.(Novel interplay between SDP, reductions, high-dimension geometry….)

Arora: SDP + Approx Survey Outline: SDPs & Approximation SDP and its use in approximation: Generations 1 & 2 Understanding SDPs high dimensional geometry Faster algorithms (multiplicative update rule) Limitations of SDPs, Unique Games Conjecture Future directions Open problems

Arora: SDP + Approx Survey How do you understand these vector programs? Ans. Interesting geometric analysis

Arora: SDP + Approx Survey Understanding SDPs Understanding phenomena in high-dimensional geometry computes c-approximation for c < 2 iff following is true Vertex Cover SDP Every graph in this family has an independent set of size Thm [Frankl-Rodl87] False. Vertices: n unit vectors Edges: almost-antipodal pairs RnRn [GK96]

Arora: SDP + Approx Survey SDP rounding: The two generations Generation 1: *Uses random hyperplane as in [GW]; * Edge-by-edge analysis Max-2SAT and Max-CUT [GW94] ;Graph coloring [KMS95]; MAX-3SAT [KZ97]; Algorithms for Unique Games;.. Generation 2: Global rounding and analysis Graph partitioning problems [ARV04], Graph deletion and directed partitioning problems [ACMM05], New analysis of graph coloring [ACC06] Disproof of UGC for expanding constraints [AKKSTV08] Recently, generation 1.5: Squish n solve rounding. k-CSPs[RS09]

Arora: SDP + Approx Survey 1 st Generation Rounding: Ratio for MAX-CUT [Goemans- Williamson93] G = (V,E) Find that maximizes capacity. Quadratic Programming Formulation Semidefinite Relaxation [DP 91, GW 93]

Arora: SDP + Approx Survey Randomized Rounding (1 st Gen) [GW 93] v6v6 v2v2 v3v3 v5v5 RnRn v1v1 Form a cut by partitioning v 1,v 2,...,v n around a random hyperplane. SDP OPT vjvj vivi ij Old math rides to the rescue...

Arora: SDP + Approx Survey Surely this bizarre algorithm is not the right way to solve max cut??

Arora: SDP + Approx Survey Fact 1: No rounding algorithm can produce a better solution out of this SDP [Feige-Schechtman] Fact 2: If P NP then impossible to get 1.06-approximation in poly time [Hastad97] Fact 3: If unique games conjecture is true, no better than 1.13-approximation possible in poly time.[KKMO05] (i.e., algorithm on prev. slide is optimal) Edges between all pairs of vectors making an angle 138 degrees.

Arora: SDP + Approx Survey 2 nd Generation: for c-balanced separator G= (V, E); constant c >0 Goal: Find cut s.t. each side contains at least c fraction of nodes and minimized 1 SDP: Triangle inequality Angle subtended by the line joining two of them on the third is non-obtuse; condition.

Arora: SDP + Approx Survey Rounding algorithm for -approximation [ARV04] 1. Pick random hyperplane 2. Remove points in slab of width 3. Remove any pair (i, j) that lie on opp. sides of slab but 4. Call remaining sets S, T. Do BFS from S to T according to distance 5. Output level of BFS tree with least # of edges. S T ST Heart of analysis: Shows |S|, |T| = (n); Large well-separated sets

Arora: SDP + Approx Survey Geometric fact underlying the analysis ( restatement of [ARV04] Structure Theorem by [AL06]) (expander : | (S)| (|S|) ) Vertices: unit vectors satisfying triangle inequality Edges: If then no graph in this family is an expander. Proof: Difficult chaining argument. (Aside: Has been used to prove that l 1 embeds into l 2 with distortion [CGR05,ALN06] ) =

Arora: SDP + Approx Survey Next few slides: Results showing Approximation is hard assuming Unique-Games-Conjecture (UGC) Recall: Integrality gap of an SDP = min c st <= c Let tough instance = problem instance with integrality gap c These play crucial role in above reductions!!

Arora: SDP + Approx Survey Unique Games Given: Number p, and m equations in n vars of the form: Promise: Either there is a solution that satisfies fraction of constraints or no solution satisfies even fraction. UGC [Khot02]: Deciding which case holds is NP-hard. [Raghavendra08; building upon [KKMO04][MOO05]] UGC For every MAX-CSP, the simplest SDP relaxation is the best possible poly-time approximation.

Arora: SDP + Approx Survey Anatomy of a UGC-based hardness result ( eg Khot-Regev, KKMO, Raghavendra08) Interpret as a graph Equations Variables Replace edges/vertices with gadgets involving integrality gap instance Equations Variables Prove using harmonic analysis that near- -optimum solns correspond to good solution to the unique game

Generation 1.5 rounding: Squish-n-solve ( Raghavendra-Steurer09; provably optimal approximation for all k-CSPs if UGC is true ) Arora: SDP + Approx Survey Project to random t-dim subspace; t =O(1) Merge nodes whose vectors are close together; get instance of size exp(t) and solve it optimally. If g= approx. ratio for this algorithm then it is also the integrality gap for SDP, and also the best possible approx. ratio (assuming UGC).

Arora: SDP + Approx Survey Solving SDPs with m constraints takes time. Issue of Running Time m =n 3 in some of these SDPs! Next slide: Often, can reduce running time: O(n 2 ) or O(n 3 ) [AHK05], [AK07] even O(n) for CSPs! [St10]; O(n 1.5 +m) for sparsest cut [S09] Main idea: Primal-dual schema. Solve to approximate optimality; using insights from the rounding algorithms. Multiplicative Weight-Update Rule for psd matrices

Arora: SDP + Approx Survey Primal-dual approach for SDP relaxations (contd.) [A., Kale07] At step t: Primal player: PSD matrix X t ; candidate primal Dual player: Let me run the rounding algorithm on P t, get a primal integer candidate and point out how pitiful it is. Candidate slack matrix M t Primal player: X t+1 = exp(- t M t ) (Analysis uses formal analogy between real #s and symmetric matrics: [Other ingredients: flow computations, eigenvalues, dimension reduction tricks, etc.; simplified by [KRV07], [KRVV08]]

Arora: SDP + Approx Survey Future directions….

Direction 1: How to soup up the relaxation Arora: SDP + Approx Survey Other forms of convex relaxations instead of SDP? (e.g. geometric programming, convex programming) For a start, re-derive existing approximation ratios (eg For MAX-CUT) using the above.

Arora: SDP + Approx Survey Direction 2: Understand Lifted SDP relaxations Recall: SDP tries to simulate nonlinear programming; Variable for Why not take it to the next level? Variables for products of up to k variables. This is the main idea of Lovasz-Schrijver91, Sherali-Adams, Lasserre etc. Yields better approx for hypergraph I.S. [CS08] Dont seem to help for some problems, even for k =n/100: MAX-k-SAT [BGHMP06], [AAT05], Vertex Cover[ABLT06],[STT07a+b] MAX-CUT, Vertex Cover etc. [CMM08] MAX-3SAT (Sch09) (v. subtle and beautiful ideas!)

Direction 3: graph expansion Arora: SDP + Approx Survey Expansion Approx Ratio [ARV]; SDP w/ triangle ineq. [Cheeger72]; eigenvalue O(1) approx is UGC hard; as is improving Cheeger for constt Lots of space for improvement even if UGC holds Also, small set expansion problem is v. close to UG and progress on it would possibly give progress on UG. [RS10]

Direction 4: Subexponential algorithms Inspiration: Unique Games with completeness 1- can be approximately solved in exp(n ) time [ABS10] Arora: SDP + Approx Survey (For problems like MAX3SAT, no such algorithms exist if 3SAT has no subexp. algorithms) Intriguing possibility: Many of the UGC-hard problems have subexponential algorithms. Another interesting idea: derive [ABS10] algorithm for UG using SDPs or Lasserre relaxations.

Arora: SDP + Approx Survey Direction 5: SDP in Avg. Case Complexity Recent development: Interreducibility among some average case problems of interest. [Feige01]; e.g Easy optimal 7/8-hardness of MAX-3SAT. Problems like 3SAT seem difficult not only in the worst case but also on average. (Needs careful definition!) Theory of Avg Case complexity [Levin84] doesnt usually apply to problems of practical interest (e.g., random 3SAT). SDP is used in the reduction! (Used to weed out uninteresting cases)

Direction 6: Applications to quantum computing PSD matrix of trace 1 = density matrix, a way to describe a mixed quantum state Arora: SDP + Approx Survey Recent result QIP=PSPACE (JUW10) uses the [AK07] Primal-dual framework. (Fast NC computation of near-optimum quantum state) Expertise in designing specialized matrices for SDP integrality gaps may prove useful in QC…

Arora: SDP + Approx Survey Open problems Can Lasserre relaxations compute nontrivial approximations to Vertex Cover, MAX-CUT, etc? (ruled out already for MAX-3SAT [Sch08]) Generation 3 rounding? Iterative rounding for SDPs? Resolve UGC (eg disprove by giving truly subexp. algorithms) SDP as a proof technique---apply to open problems of circuit complexity, communication complexity etc. Looking forward to many developments THANK YOU!

Arora: SDP + Approx Survey Classical MW update rule (Example: predicting the market) N experts on TV Can we perform as good as the best expert in hindsight? 1$ won for correct prediction 1$ lost for incorrect prediction Thm[Going back to Hannan, 1950s] Yes.

Weighted Majority Algorithm (LW94) Arora: SDP + Approx Survey For each expert, weight w i. Initially w i 1 Follow expert i w/ prob. proportional to w i t Claim: Expected per-round loss of our algorithm Update weights according to Losses M 1 t M 2 t M 3 t …

Lagrangian method to approximately solve LPs (PST91, many others) Arora: SDP + Approx Survey For each expert, weight w i. Initially w i 1 Follow expert i w/ prob. proportional to w i t Claim: Expected per round loss of our algorithm Update weights according to Losses M 1 t M 2 t M 3 t … Experts = Dual LP constraints ; maintain weighting Loss vector = Slack vector of candidate dual soln Claim: After a few rounds; the average of all the loss vectors is an approximately feasible dual soln.

Lagrangian method to approximately solve LPs (PST91, many others since) Arora: SDP + Approx Survey Claim: Expected per round loss of our algorithm x = weighting of n experts; updated via multiplicative update Loss vector Expected loss ! Average per-round loss of expert i = ith coordinate of Only 1 constraint !

Lagrangian method to approximately solve SDPs (A.,Kale 07) Arora: SDP + Approx Survey Claim: Expected per round loss of our algorithm x = weighting of n experts; updated via multiplicative update Loss vector Expected loss ! x= PSD matrix; updated according to psd

Arora: SDP + Approx Survey SDPs and MW Updates: Primal-dual algorithm Known: MW Update rule --> Approx. solutions to LPs [PST91, Y95, GK97,..etc.] [AK07] Matrix MW update rule that uses formal analogy between psd matrices and nonnegative real #s. (Spl. Case: LPs= SDPs with 0s on offdiagonals) [Other ingredients: flow computations, eigenvalues, dimension reduction tricks, etc.] experts constraints payoffs slack in constraint [Golden-Thompson]

Arora: SDP + Approx Survey Embeddings and Cuts Thm[LLR94, AR94]: Integrality gap for SDP for Nonuniform Sparsest Cut = Min distortion of any embedding of into Rounding algorithm of [ARV04] gives insight into structure of ; basis of new embeddings Hardness results for sparsest cut yielded insights at the heart of the embedding impossibility results.

Arora: SDP + Approx Survey Limitations of SDPs For many problems, we know neither an NP-hardness result (via PCPs) nor a good SDP-based approach. 2nd Generation results: Large families of LPs or SDPs dont work [ABL02], [ABLT06]: Proving integrality gaps without knowing the LP. Much subsequent work, especially on families obtained from lift and project ideas) 1st generation results: Specific SDPs dont work Can we show that known SDPs dont work??

Arora: SDP + Approx Survey Example: 2-approximation for Min Vertex Cover G= (V, E) Vertex Cover = Set of vertices that touches every edge LP Relaxation Claim: Value at least OPT/2 Proof: Rounding most Proof: On Complete Graph K n, OPT = n-1 but setting all x i = 1/2 gives feasible LP soln

Next, briefly Arora: SDP + Approx Survey Connection between analysis of SDPs and Geometric Embedding of Metric Spaces

[CGR05,ALN05]: Yes, C possible for X = Arora: SDP + Approx Survey Geometric embeddings of metric spaces x (X, d): metric space y d(x, y) f f(x) f(y) C = distortion Thm (Bourgain85) For every X, there is f s.t. C= O(log n). Open qs since then: is it possible to achieve smaller C for concrete X, say X = ? Via [LLR94,AR94] implies approx for general sparsest cut

Arora: SDP + Approx Survey Main issue: Local versus Global Example: [Erdos] There are graphs on n vertices that cannot be colored with 100 colors yet every subgraph on 0.01 n vertices is 3-colorable. LP relaxations or SDP relaxations concern local conditions. How well do such local conditions capture global property in question? Results for MAX-k-SAT [], [AAT05], Vertex Cover[ABLT06], [STT07a+b] MAX-CUT, Vertex Cover etc.[CMM08] Lifted SDPs. Connections to Proof Complexity.