1. ECE Department Rice University dsp.rice.edu/cs Measurements and Bits: Compressed Sensing meets Information Theory Shriram Sarvotham Dror Baron Richard Baraniuk

2. CS encoding Replace samples by more general encoder based on a few linear projections (inner products) Matrix vector multiplication measurements sparse signal # non-zeros

3. The CS revelation – Of the infinitely many solutions seek the one with smallest L 1 norm

4. If then perfect reconstruction w/ high probability [Candes et al.; Donoho] Linear programming The CS revelation –

5. Compressible signals Polynomial decay of signal components Recovery algorithms –reconstruction performance: –also requires –polynomial complexity (BPDN) [Candes et al.] Cannot reduce order of [Kashin,Gluskin] squared of best term approximation constant

6. Fundamental goal: minimize Compressed sensing aims to minimize resource consumption due to measurements Donoho: “Why go to so much effort to acquire all the data when most of what we get will be thrown away?”

7. Measurement reduction for sparse signals Ideal CS reconstruction of -sparse signal Of the infinitely many solutions seek sparsest one If M · K then w/ high probability this can’t be done If M ¸ K+1 then perfect reconstruction w/ high probability [Bresler et al.; Wakin et al.] But not robust and combinatorial complexity number of nonzero entries

8. Why is this a complicated problem?

9. Rich design space What performance metric to use? –Wainwright: determine support set of nonzero entries  this is distortion metric  but why let tiny nonzero entries spoil the fun? – metric?? What complexity class of reconstruction algorithms? –any algorithms? –polynomial complexity? –near-linear or better? How to account for imprecisions? –noise in measurements? –compressible signal model?

10. How many measurements do we need?

11. Measurement noise Measurement process is analog Analog systems add noise, non-linearities, etc. Assume Gaussian noise for ease of analysis

12. Setup Signal is iid Additive white Gaussian noise Noisy measurement process

13. Measurement and reconstruction quality Measurement signal to noise ratio Reconstruct using decoder mapping Reconstruction distortion metric Goal: minimize CS measurement rate

14. Measurement channel Model processas measurement channel Capacity of measurement channel Measurements are bits!

15. Main result Theorem: For a sparse signal with rate-distortion function, lower bound on measurement rate subject to measurement qualityand reconstruction distortionsatisfies Direct relationship to rate-distortion content

16. Main result Theorem: For a sparse signal with rate-distortion function, lower bound on measurement rate subject to measurement qualityand reconstruction distortionsatisfies Proof sketch: –each measurement providesbits –information content of sourcebits –source-channel separation for continuous amplitude sources –minimal number of measurements –Obtain measurement ratevia normalization by

17. Example Spike process - spikes of uniform amplitude Rate-distortion function Lower bound Numbers: –signal of length 10 7 –10 3 spikes –SNR=10 dB  –SNR=-20 dB 

18. Upper bound (achievable) in progress…

19. CS reconstruction meets channel coding

20. Why is reconstruction expensive? measurements sparse signal nonzero entries Culprit: dense, unstructured

21. Fast CS reconstruction measurements sparse signal nonzero entries LDPC measurement matrix (sparse) Only 0/1 in Each row of contains randomly placed 1’s Fast matrix multiplication  fast encoding and reconstruction

22. Ongoing work: CS using BP [Sarvotham et al.] Considering noisy CS signals Application of Belief Propagation –BP over real number field –sparsity is modeled as prior in graph Low complexity Provable reconstruction with noisy measurements using Success of LDPC+BP in channel coding carried over to CS!

23. Summary Determination of measurement rates in CS –measurements are bits: each measurement provides bits –lower bound on measurement rate –direct relationship to rate-distortion content Compressed sensing meets information theory Additional research directions –promising results with LDPC measurement matrices –upper bound (achievable) on number of measurements dsp.rice.edu/cs