Boaz Barak – Microsoft Research Partially based on joint work with Jonathan Kelner (MIT) and David Steurer (Cornell) Sum of Squares Proofs and The Quest Towards Optimal Algorithms SURGEON GENERAL’S WARNING: Talk contains only “pseudo theorems” Antidote: 10-week seminar series starting 9/29

The space of algorithms is very rich - a different algorithm for every problem

A pipe dream… One algorithm for all problems? Answer 2: Maybe there is a deeper reason.. Few unifying principles underlie many algorithms. Convexity, matroid structure, submodularity, algebraic identities,.. Answer 1: Yes for non-interesting reasons. Proof: Enumerate over all Turing machines and try them all..

Hope: An optimal meta algorithm: Why care? A pipe dream… One algorithm for all problems? Approach to getting many hardness/algorithmic results at once. Principled approach to generate hardness conjectures: Cryptosystem with principled arguments for security? Design hard problems to demonstrate quantum speedups? Understand why a problem is easy or hard.. Find computational “phase transitions” … not just a “laundry list” of algorithms/reductions “Computational information theory”

A generic “meta algorithm” for polynomial optimization [Shor’87,Nesterov’00,Parillo’00,Lasserre’01] Algorithmic version of works related to Hilbert’s 17 th problem [Artin 27,Krivine64,Stengle74] Used in many applications including quantum information theory, control theory, automated theorem proving, game theory, and more… Image credit: Chakraborty et al The Sum of Squares Algorithm Generalizes many known algorithms: Empirically seems to work well. Theoretical analysis lags far behind. Linear Programming Semidefinite Programming Exhaustive search ???

Is Sum-of-Squares an Optimal Algorithm? No. SOS doesn’t solve linear equations mod 2 [Grigoriev’01] One (vague) interpretation: SOS doesn’t do half-measures. Can’t distinguish between solving perfectly satisfiable vs noisy equations. But solving noisy linear equations is NP hard [Håstad ‘95] Is SOS optimal for an interesting class of problems? Don’t really know…... has interesting relation to the Unique Games Conjecture see rest of this talk

This talk: Dual views of SOS alg: Positivstellensatz/SOS proofs “Pseudo distributions” Example application: Dictionary Learning / Sparse Coding Relation to Khot’s Unique Games Conjecture Image credits: Georghiades et al, Mairal et al, Singer, Kindler et al

Polynomial Equations Extremely general framework: All poly’s of degree 1: Linear programming Single quadratic: Least squares (eigenvalue*) problem Poly’s of degree >1: Captures a great many problems, some NP hard Can similarly encode many other problems, including SAT, 3COL, Max-Cut, etc…

Empirically often converges much faster, still not very well understood. Some evidence and intuition, still not at the level of a precise conjecture. Next: What is a “proof” and what is a “pseudo-solution” SOS Algorithm – high level view:

SOS Proofs An SOS proof deduces polynomial inequalities using the following rules: SOS proofs surprisingly powerful: Capture many standard tools such as Cauchy-Schwarz, Hölder, etc.. and even more advanced notions (e.g., Isoperimetric results on Boolean cube) Only examples of assertions that robustly* require large degree to prove are non-constructive, shown using probabilistic method.

What is a “pseudo-solution”? Next: 1) Detour: defining statistical knowledge via distributions 2) Definition of pseudo solutions. SOS Algorithm – high level view:

Detour: Knowledge as a distribution As you learn more information, you adjust your distribution accordingly.

Detour: Knowledge as a distribution As you learn more information, you adjust your distribution accordingly. i.e., all information is known, but we are computationally bounded and can’t make all possible logical inferences from it. Can we do the same for computational knowledge?

Computational knowledge as a pseudo-distribution Notes:

An SOS proof deduces polynomial inequalities using the following rules:

This talk: Dual views of SOS alg: Positivstellensatz/SOS proofs “Pseudo distributions” Example application: Dictionary Learning / Sparse Coding Relation to Khot’s Unique Games Conjecture Image credits: Georghiades et al, Mairal et al, Singer, Kindler et al

Find the “right” representation of observed data LOTS of work: important primitive in Machine Learning, Vision, Neuroscience... Example Application: Dictionary Learning / Sparse Coding [Olhausen-Field ’96] [Mairal-Bach-Ponce-Sapiro ’10]

Example Application: Dictionary Learning / Sparse Coding (have no control over local maxima) Proof of (*) uses low degree SOS arguments Can show (*) using Hölder-type inequalities.

This talk: Dual views of SOS alg: Positivstellensatz/SOS proofs “Pseudo distributions” Example application: Dictionary Learning / Sparse Coding Relation to Khot’s Unique Games Conjecture Image credits: Georghiades et al, Mairal et al, Singer, Kindler et al

Computational Phase Transitions Running Time exp(n) Quality parameter Proven phase transitions for several worst case problems [Håstad ’95, Moshkovitz-Raz’08] Conjectured for many more and for several average-case problems Complexity vs quality for some problem X

Unique Games Conjecture Conjecture [Khot ‘02] : Certain problem known as “Unique Games” is NP hard. Many implications to complexity: Implies hardness results problems across many domains including constraint satisfaction, cut and routing, scheduling, algebra,.. Fascinating connections to areas including probability, geometry, metric embeddings, social choice theory, etc.. see surveys [Khot, ‘10 ‘10 ‘14], [Trevisan ‘12] vs. The $64K question: Is the UGC true? Yields optimality (= computational phase transition) of many canonical algorithms (e.g. Grothendieck, Cheeger-Alon-Milman, Geomans-Williamson, etc..).

Main reasons to believe UGC: Can’t refute it: Don’t know of an algorithm that solves it. Want it to be true: Gives very clean picture of complexity landscape. Algorithmic attacks on UGC: “Eigenspace enumeration”: Brute force search in top eigenspace of adj. matrix SOS Algorithm: Generalizes both Solves random instances … [Arora-Khot-Kolla-Steurer-Tulsiani-Vishnoi’05] but not all instances [Khot-Vishnoi’04] Solves KV instance [Kolla’10] [Arora-B-Steurer-’10]..and every instance in subexp time [B-Gopalan-Håstad-Meka-Raghavendra-Steurer’12]..but not much better know an algorithm, don’t have proof Solves KV+BGHMRS instances [B-Brandao-Harrow-Kelner-Steurer-Zhou ‘12] [B-Kelner-Steurer ‘14].. candidate algorithm to solve all instances sort of

SOS and the Unique Games Conjecture UGC implies that UGC: UGC with poly (depending on parameter) time reduction false [Arora-B-Steurer’10] Running Time Quality parameter Plausible scenarios for complexity of a “typical” CSP (e.g. Max-Cut) SOS is optimal for many problems, including all CSP’s

Conclusions