1. Compressed Sensing meets Information Theory Dror Baron drorb@ee.technion.ac.il www.ee.technion.ac.il/people/drorb Duarte Wakin Sarvotham Baraniuk Guo Shamai

2. Sensing, Computation, Communication –fast, readily available, cheap Progress in individual disciplines (computing, networks, comm, DSP, …) Technology Breakthroughs

3. The Data Deluge

4. Challenges: –Exponentially increasing amounts of data  myriad different modalities (audio, image, video, financial, seismic, weather …)  global scale acquisition –Analysis/processing hampered by slowing Moore’s law  finding “needle in haystack” –Energy consumption Opportunities (today)

5. From Sampling to Compressed Sensing

6. Sensing by Sampling Sample data at Nyquist rate (2x highest frequency in signal) Compress data using model (e.g., sparsity) Lots of work to throw away >90% of the coefficients Most computation at sensor (asymmetrical) Brick wall to performance of modern acquisition systems compress transmit/store receivedecompress sample sparse wavelet transform

7. Sparsity / Compressibility pixels large wavelet coefficients wideband signal samples large Gabor coefficients Many signals are sparse in some representation/basis (Fourier, wavelets, …)

8. Compressed Sensing Shannon/Nyquist sampling theorem –must sample at 2x highest frequency in signal –worst case bound for any bandlimited signal –too pessimistic for some classes of signals –does not exploit signal sparsity/compressibility Seek direct sensing of compressible information Compressed Sensing (CS) –sparse signals can be recovered from a small number of nonadaptive (fixed) linear measurements –[Candes et al.; Donoho; Rice,…]

9. Measure linear projections onto random basis where data is not sparse –mild “over-sampling” in analog Decode (reconstruct) via optimization Highly asymmetrical (most computation at receiver) Compressed Sensing via Random Projections projecttransmit/store receivedecode

10. CS Encoding Replace samples by more general encoder based on a few linear projections (inner products) measurements sparse signal # non-zeros

11. Random projections Universal for any compressible/sparse signal class measurements sparse signal Universality via Random Projections # non-zeros

12. Optical Computation of Random Projections [Rice DSP 2006] CS measurements directly in analog Single photodiode

13. First Image Acquisition ideal 64x64 image (4096 pixels) 400 wavelets image on DMD array 1600 random meas.

14. Goal: find x given y Ill-posed inverse problem Decoding approach –search over subspace of explanations to measurements –find “most likely” explanation –universality accounted for during optimization Linear program decoding [Candes et al., Donoho] –small number of samples –computationally tractable Variations –greedy (matching pursuit) [Tropp et al., Needell et al.,...] –optimization [Hale et al., Figueiredo et al.] CS Signal Decoding

15. CS Hallmarks CS changes rules of data acquisition game –exploits a priori sparsity information to reduce #measurements Hardware/software:Universality –same random projections for any compressible signal class –simplifies hardware and algorithm design Processing:Information scalability –random projections ~ sufficient statistics –same random projections for range of tasks  decoding > estimation > recognition > detection –far fewer measurements required to detect/recognize Next generation data acquisition new imaging devices –new distributed source coding algorithms [Baron et al.]

16. CS meets Information Theoretic Bounds [Sarvotham, Baron, & Baraniuk 2006] [Guo, Baron, & Shamai 2009]

17. Fundamental Goal: Minimize Compressed sensing aims to minimize resource consumption due to measurements Donoho: “Why go to so much effort to acquire all the data when most of what we get will be thrown away?”

18. Signal Model Signal entry X n = B n U n iid B n » Bernoulli()  sparse iid U n » P U PUPU Bernoulli() Multiplier PXPX

19. Non-Sparse Input Can use =1  X n = U n PUPU

20. Measurement Noise Measurement process is typically analog Analog systems add noise, non-linearities, etc. Assume Gaussian noise for ease of analysis Can be generalized to non-Gaussian noise

21. Noiseless measurements denoted y 0 Noise Noisy measurements Unit-norm columns  SNR= Noise Model noiseless SNR

22. Model process as measurement channel Measurements provide information! channel CS measurement CS decoding source encoder channel encoder channel decoder source decoder CS Analog to Communication System [Sarvotham, Baron, & Baraniuk 2006]

23. Theorem: [Sarvotham, Baron, & Baraniuk 2006] For sparse signal with rate-distortion function R(D), lower bound on measurement rate s.t. SNR and distortion D Numerous single-letter bounds –[Aeron, Zhao, & Saligrama] –[Akcakaya & Tarokh] –[Rangan, Fletcher, & Goyal] –[Gastpar & Reeves] –[Wang, Wainwright, & Ramchandran] –[Tune, Bhaskaran, & Hanly] –… Single-Letter Bounds

24. Goal: Precise Single-letter Characterization of Optimal CS [Guo, Baron, & Shamai 2009]

25. What Single-letter Characterization? Ultimately what can one say about X n given Y? (sufficient statistic) Very complicated Want a simple characterization of its quality Large-system limit:  channelposterior

26. Main Result: Single-letter Characterization [Guo, Baron, & Shamai 2009] Result1: Conditioned on X n =x n, the observations (Y,) are statistically equivalent to  easy to compute… Estimation quality from (Y,) just as good as noisier scalar observation degradation  channelposterior

27.  2 (0,1) is fixed point of Take-home point: degraded scalar channel Non-rigorous owing to replica method w/ symmetry assumption –used in CDMA detection [Tanaka 2002, Guo & Verdu 2005] Related analysis [Rangan, Fletcher, & Goyal 2009] –MMSE estimate (not posterior) using [Guo & Verdu 2005] –extended to several CS algorithms particularly LASSO Details

28. Decoupling

29. Result2: Large system limit; any arbitrary (constant) L input elements decouple: Take-home point: individual posteriors statistically independent Decoupling Result [Guo, Baron, & Shamai 2009]

30. Sparse Measurement Matrices

31. Why is Decoding Expensive? measurements sparse signal nonzero entries Culprit: dense, unstructured

32. Sparse Measurement Matrices [Baron, Sarvotham, & Baraniuk 2009] LDPC measurement matrix (sparse) Mostly zeros in ; nonzeros » P  Each row contains ¼ Nq randomly placed nonzeros Fast matrix-vector multiplication  fast encoding / decoding sparse matrix

33. CS Decoding Using BP [Baron, Sarvotham, & Baraniuk 2009] Measurement matrix represented by graph Estimate real-valued input iteratively Implemented via nonparametric BP [Bickson,Sommer,…] measurements y signal x

34. Identical Single-letter Characterization w/BP [Guo, Baron, & Shamai 2009] Result3: Conditioned on X n =x n, the observations (Y,) are statistically equivalent to Sparse matrices just as good Result4: BP is asymptotically optimal! identical degradation

35. Decoupling Between Two Input Entries (N=500, M=250, =0.1, =10) density

36. CS-BP vs Other CS Methods (N=1000, =0.1, q=0.02) MM MMSE CS-BP

37. CS-BP is O(Nlog 2 (N)) (M=0.4N, =0.1, =100, q=0.04) Runtime [seconds] NN

38. Fast CS Decoding [Sarvotham, Baron, & Baraniuk 2006]

39. Setting measurements sparse signal nonzero entries LDPC measurement matrix (sparse) Fast matrix-vector multiplication Assumptions: –noiseless measurements –strictly sparse signal

40. Example 01140114 0 1 1 0 0 0 0 0 0 1 1 0 1 1 0 0 1 0 0 0 0 0 1 1 ????????????

41. Example 01140114 0 1 1 0 0 0 0 0 0 1 1 0 1 1 0 0 1 0 0 0 0 0 1 1 ???????????? What does zero measurement imply? Hint: x strictly sparse

42. Example 01140114 0 1 1 0 0 0 0 0 0 1 1 0 1 1 0 0 1 0 0 0 0 0 1 1 ?00????00??? Graph reduction!

43. Example 01140114 0 1 1 0 0 0 0 0 0 1 1 0 1 1 0 0 1 0 0 0 0 0 1 1 ?00????00??? What do matching measurements imply? Hint: non-zeros in x are real numbers

44. Example 01140114 0 1 1 0 0 0 0 0 0 1 1 0 1 1 0 0 1 0 0 0 0 0 1 1 00001?00001? What is the last entry of x?

45. Main Results [Sarvotham, Baron, & Baraniuk 2006] # nonzeros per row # measurements Fast encoder and decoder –sub-linear decoding complexity Can be used for distributed content distribution –measurements stored on different servers –any M measurements suffice Strictly sparse signals, noiseless measurements

46. Related Direction: Linear Measurements unified theory for linear measurement systems

47. Linear Measurements in Finance Fama and French three factor model (1993) –stock returns explained by linear exposure to factors  e.g., “market” (change in stock market index) –numerous factors can be used (e.g., earnings to price) Noisy linear measurements stock returns unexplained (typically big) exposures factor returns

48. Financial Prediction Explanatory power ¼ prediction (can invest on this) Goal: estimate x to explain y well Financial prediction vs CS: longer y, shorter x Sounds easy, nonetheless challenging –NOISY data  need lots of measurements –nonlinear, nonstationary compressed sensing financial prediction

49. Application Areas for Linear Measurements DSP (CS) Finance Medical imaging (tomography) Information retrieval Seismic imaging (oil industry)

50. Unified Theory of Linear Measurement Common goals –minimal resources –robustness –computationally tractable Inverse problems Striving toward theory and efficient processing in linear measurement systems

51. THE END