Recovering structured signals: Precise performance analysis Christos Thrampoulidis Joint ITA Workshop, La Jolla, CA February 3, 2016

Let’s start “simple”…

Given y and A can you find x 0 ? Noiseless linear measurements

Does it work? (if so, when?) Let A be random (e.g. Gaussian, random DFT). [Candes,Romberg,Tao’04] then, x 0 is the unique solution with high probability. If,

Suppose A has entries iid Gaussian. Please, be precise… Then, [Donoho’06; Donoho,Tanner’09; Stojnic’09; Donoho et.al.’09; Amelunxen et.al.’13; Stojnic’13] In fact, much more is known… Generalization to other structures (e.g. group-sparse, low-rank, binary entries, smooth) and convex regularizer functions f (e.g. l 1,2, nuclear-norm, atomic-norms). Gaussian-width, statistical dimension [Donoho et. al.’09; Chandrasekaran et.al.’12; Amelunxen et.al.’13]

Crystal clear understanding!

One step further…

Add some noise!

Can you estimate x 0 ? Noisy linear measurements

What is the quality of the estimate? LASSO Suppose A has randomness. then w.h.p.: and If e.g. [Belloni’11]

Precise results?

Precise results? Yes! [Donoho, Maleki, Montanari’09] and [Bayati, Montanari’11]. Approach is based on the AMP framework, inspired by statistical physics. Over the past couple of year, we have developed a new framework the CGMT that is based on Gaussian process methods. Framework is inspired by [Stojnic’13]. Gives results that are very general. Natural way of analysis… CGMT

Precise results? Yes! [C.T., Panahi, Guo, Hassibi’14]

Precise results? Yes! [C.T., Panahi, Guo, Hassibi’14]

Precise results? Yes! [C.T., Panahi, Guo, Hassibi’14]

Precise results? Yes! [C.T., Panahi, Guo, Hassibi’14]

Precise results? Yes! [C.T., Panahi, Guo, Hassibi’14]

Precise results? Yes! Expected Moreau Envelope summary functional: captures the role of regularizer and of p X0. generalizes the “Gaussian-width” of noiseless CS. [Thrampoulidis, Panahi, Guo, Hassibi’14]

Optimal tuning There is value to be precise! Noiseless CS results follow as special case:

Generalizations General structures and convex regularizers. Other performance metrics (e.g. support recovery, BER) – Massive MIMO (BER of convex relaxation decoders) General noise distributions and loss functions! Regularized M-estimators

Master Theorem [Thrampoulidis, Abbasi, Hassibi ’16]

Optimal tuning Precise is good!

Optimal loss function? 23 Precise & general is better!

Consistent Estimators? 24 sparse noise

Robustness to outliers? Huber loss function + Cauchy noise

Final challenge

measurement device has nonlinearities & uncertainties. can also arise by design Non-linear measurements (or single-index model)

measurement device has nonlinearities & uncertainties. can also arise by design Non-linear measurements (or single-index model)

What if we use the LASSO? 1.the link function might be unknown/misspecified (robustness) 2.widely available LASSO solvers, used in practice 3.turns out it performs well

Theorem (Non-linear=linear) For example, for 1-bit measurements: [Thrampoulidis, Abbasi, Hassibi. NIPS’15] Previous work of Plan & Vershynin ‘15 derives order-wise bounds for the constrained LASSO. Our precise result explicitly captures equivalence & the role of σ.

Application: optimal quantization q-bit measurements LASSO Lloyd-Max (LM) algorithm is optimal A LASSO A

One last word: How it all works?

Gaussian min-max Theorem (Gordon’88) Convex Gaussian Min-max Theorem [Thrampoulidis, Oymak, Hassibi. COLT ’15] CGMT framework

Gaussian min-max Theorem (Gordon’88) Convex Gaussian Min-max Theorem [Thrampoulidis, Oymak, Hassibi. COLT ’15] CGMT framework GMT

Thrampoulidis, Abbasi, Hassibi: “Precise Error Analysis of Regularized M-estimators in High-dimensions”, available on arXiv, Jan More… Thank you!