ECE Department Rice University dsp.rice.edu/cs Measurements and Bits: Compressed Sensing meets Information Theory Shriram Sarvotham Dror Baron Richard Baraniuk

CS encoding Replace samples by more general encoder based on a few linear projections (inner products) Matrix vector multiplication measurements sparse signal # non-zeros

The CS revelation – Of the infinitely many solutions seek the one with smallest L 1 norm

If then perfect reconstruction w/ high probability [Candes et al.; Donoho] Linear programming The CS revelation –

Compressible signals Polynomial decay of signal components Recovery algorithms –reconstruction performance: –also requires –polynomial complexity (BPDN) [Candes et al.] Cannot reduce order of [Kashin,Gluskin] squared of best term approximation constant

Fundamental goal: minimize Compressed sensing aims to minimize resource consumption due to measurements Donoho: “Why go to so much effort to acquire all the data when most of what we get will be thrown away?”

Measurement reduction for sparse signals Ideal CS reconstruction of -sparse signal Of the infinitely many solutions seek sparsest one If M · K then w/ high probability this can’t be done If M ¸ K+1 then perfect reconstruction w/ high probability [Bresler et al.; Wakin et al.] But not robust and combinatorial complexity number of nonzero entries

Why is this a complicated problem?

Rich design space What performance metric to use? –Wainwright: determine support set of nonzero entries  this is distortion metric  but why let tiny nonzero entries spoil the fun? – metric?? What complexity class of reconstruction algorithms? –any algorithms? –polynomial complexity? –near-linear or better? How to account for imprecisions? –noise in measurements? –compressible signal model?

How many measurements do we need?

Measurement noise Measurement process is analog Analog systems add noise, non-linearities, etc. Assume Gaussian noise for ease of analysis

Setup Signal is iid Additive white Gaussian noise Noisy measurement process

Measurement and reconstruction quality Measurement signal to noise ratio Reconstruct using decoder mapping Reconstruction distortion metric Goal: minimize CS measurement rate

Measurement channel Model processas measurement channel Capacity of measurement channel Measurements are bits!

Main result Theorem: For a sparse signal with rate-distortion function, lower bound on measurement rate subject to measurement qualityand reconstruction distortionsatisfies Direct relationship to rate-distortion content

Main result Theorem: For a sparse signal with rate-distortion function, lower bound on measurement rate subject to measurement qualityand reconstruction distortionsatisfies Proof sketch: –each measurement providesbits –information content of sourcebits –source-channel separation for continuous amplitude sources –minimal number of measurements –Obtain measurement ratevia normalization by

Example Spike process - spikes of uniform amplitude Rate-distortion function Lower bound Numbers: –signal of length 10 7 –10 3 spikes –SNR=10 dB  –SNR=-20 dB 

Upper bound (achievable) in progress…

CS reconstruction meets channel coding

Why is reconstruction expensive? measurements sparse signal nonzero entries Culprit: dense, unstructured

Fast CS reconstruction measurements sparse signal nonzero entries LDPC measurement matrix (sparse) Only 0/1 in Each row of contains randomly placed 1’s Fast matrix multiplication  fast encoding and reconstruction

Ongoing work: CS using BP [Sarvotham et al.] Considering noisy CS signals Application of Belief Propagation –BP over real number field –sparsity is modeled as prior in graph Low complexity Provable reconstruction with noisy measurements using Success of LDPC+BP in channel coding carried over to CS!

Summary Determination of measurement rates in CS –measurements are bits: each measurement provides bits –lower bound on measurement rate –direct relationship to rate-distortion content Compressed sensing meets information theory Additional research directions –promising results with LDPC measurement matrices –upper bound (achievable) on number of measurements dsp.rice.edu/cs