Compressive Signal Processing Richard Baraniuk Rice University

Better, Stronger, Faster

Sense by Sampling sample Example: Large Hadron Collider will produce 10 peta bytes of data per second; need to triage that down to 100/sec “potentially interesting events” in real time

Sense by Sampling too sample much data! Example: Large Hadron Collider will produce 10 peta bytes of data per second; need to triage that down to 100/sec “potentially interesting events” in real time

Accelerating Data Deluge 1250 billion gigabytes generated in 2010 # digital bits > # stars in the universe growing by a factor of 10 every 5 years Total data generated > total storage Increases in generation rate >> increases in comm rate Available transmission bandwidth

Sense then Compress sample compress JPEG JPEG2000 … decompress

Sparsity large wavelet coefficients (blue = 0) pixels

Sparsity large wavelet coefficients pixels wideband signal samples (blue = 0) pixels wideband signal samples large Gabor (TF) coefficients frequency time

Concise Signal Structure Sparse signal: only K out of N coordinates nonzero sparse signal nonzero entries sorted index

Concise Signal Structure Sparse signal: only K out of N coordinates nonzero model: union of K-dimensional subspaces aligned w/ coordinate axes sparse signal nonzero entries sorted index

Concise Signal Structure Sparse signal: only K out of N coordinates nonzero model: union of K-dimensional subspaces Compressible signal: sorted coordinates decay rapidly with power-law approximately sparse power-law decay sorted index

Concise Signal Structure Sparse signal: only K out of N coordinates nonzero model: union of K-dimensional subspaces Compressible signal: sorted coordinates decay rapidly with power-law model: ball: power-law decay sorted index

What’s Wrong with this Picture? Why go to all the work to acquire N samples only to discard all but K pieces of data? sample compress decompress

What’s Wrong with this Picture? linear processing linear signal model (bandlimited subspace) nonlinear processing nonlinear signal model (union of subspaces) sample compress decompress

Compressive Sensing Directly acquire “compressed” data via dimensionality reduction Replace samples by more general “measurements” compressive sensing recover

Sampling Signal is -sparse in basis/dictionary WLOG assume sparse in space domain sparse signal nonzero entries

Sampling Signal is -sparse in basis/dictionary Sampling WLOG assume sparse in space domain Sampling sparse signal measurements nonzero entries

Compressive Sampling When data is sparse/compressible, can directly acquire a condensed representation with no/little information loss through linear dimensionality reduction sparse signal measurements nonzero entries

How Can It Work? Projection not full rank… … and so loses information in general Ex: Infinitely many ’s map to the same (null space)

How Can It Work? Projection not full rank… … and so loses information in general But we are only interested in sparse vectors columns

How Can It Work? Projection not full rank… … and so loses information in general But we are only interested in sparse vectors is effectively MxK columns

How Can It Work? Projection not full rank… … and so loses information in general But we are only interested in sparse vectors Design so that each of its MxK submatrices are full rank (ideally close to orthobasis) Restricted Isometry Property (RIP) see also phase transition approach of Donoho et al. columns

RIP = Stable Embedding An information preserving projection preserves the geometry of the set of sparse signals RIP ensures that K-dim subspaces 24

RIP = Stable Embedding An information preserving projection preserves the geometry of the set of sparse signals RIP ensures that 25

How Can It Work? Projection not full rank… … and so loses information in general Design so that each of its MxK submatrices are full rank (RIP) Unfortunately, a combinatorial, NP-Hard design problem columns

Insight from the 70’s [Kashin, Gluskin] Draw at random iid Gaussian iid Bernoulli … Then has the RIP with high probability provided columns

Randomized Sensing Measurements = random linear combinations of the entries of No information loss for sparse vectors whp sparse signal measurements nonzero entries

CS Signal Recovery Goal: Recover signal from measurements Problem: Random projection not full rank (ill-posed inverse problem) Solution: Exploit the sparse/compressible geometry of acquired signal

CS Signal Recovery Random projection not full rank Recovery problem: given find Null space Search in null space for the “best” according to some criterion ex: least squares (N-M)-dim hyperplane at random angle

Signal Recovery Recovery: given (ill-posed inverse problem) find (sparse) Optimization: Closed-form solution:

Signal Recovery Recovery: given (ill-posed inverse problem) find (sparse) Optimization: Closed-form solution: Wrong answer! 32

Signal Recovery Recovery: given (ill-posed inverse problem) find (sparse) Optimization: Closed-form solution: Wrong answer! 33

Signal Recovery Recovery: given (ill-posed inverse problem) find (sparse) Optimization: “find sparsest vector in translated nullspace” 34

Signal Recovery Recovery: given (ill-posed inverse problem) find (sparse) Optimization: Correct! “find sparsest vector in translated nullspace” 35

Signal Recovery Recovery: given (ill-posed inverse problem) find (sparse) Optimization: Correct! But NP-Complete alg “find sparsest vector in translated nullspace” 36

Signal Recovery Recovery: given (ill-posed inverse problem) find (sparse) Optimization: Convexify the optimization Donoho Candes Romberg Tao 37

Signal Recovery Recovery: given (ill-posed inverse problem) find (sparse) Optimization: Convexify the optimization Correct! Polynomial time alg (linear programming) Much recent alg progress greedy, Bayesian approaches, … 38

CS Hallmarks Stable Asymmetrical (most processing at decoder) acquisition/recovery process is numerically stable Asymmetrical (most processing at decoder) conventional: smart encoder, dumb decoder CS: dumb encoder, smart decoder Democratic each measurement carries the same amount of information robust to measurement loss and quantization “digital fountain” property Random measurements encrypted Universal same random projections / hardware can be used for any sparse signal class (generic) 39

Universality Random measurements can be used for signals sparse in any basis

Universality Random measurements can be used for signals sparse in any basis

Universality Random measurements can be used for signals sparse in any basis sparse coefficient vector nonzero entries

Compressive Sensing In Action

“Single-Pixel” CS Camera scene single photon detector image reconstruction or processing DMD DMD random pattern on DMD array DMD is used in projectors Multiply value of random pattern in mirror with value of signal (light intensity) in pixel lens is focused onto the photodiode w/ Kevin Kelly

“Single-Pixel” CS Camera scene single photon detector image reconstruction or processing DMD DMD random pattern on DMD array DMD is used in projectors Multiply value of random pattern in mirror with value of signal (light intensity) in pixel lens is focused onto the photodiode … Flip mirror array M times to acquire M measurements Sparsity-based (linear programming) recovery

First Image Acquisition target 65536 pixels 11000 measurements (16%) 1300 measurements (2%)

single photon detector Utility? Fairchild 100Mpixel CCD single photon detector DMD is used in projectors Multiply value of random pattern in mirror with value of signal (light intensity) in pixel lens is focused onto the photodiode DMD DMD

SWIR CS Camera Target (illuminated in SWIR only) Camera output InView “single-pixel” SWIR Camera (1024x768) Target (illuminated in SWIR only) Camera output M = 0.5 N

CS Hyperspectral Imager (Kevin Kelly Lab, Rice U) spectrometer hyperspectral data cube 450-850nm N=1M space x wavelength voxels M=200k random measurements

CS-MUVI for Video CS Pendulum speed: 2 sec/cycle Naïve, block-based L1 recovery of 64x64 video frames for 3 different values of W 1024 2048 4096

Recovered video (animated) CS-MUVI for Video CS Effective “compression ratio” = 60:1 Low-res preview (32x32) High-res video recovery (128x128) Recovered video (animated)

Analog-to-Digital Conversion Nyquist rate limits reach of today’s ADCs “Moore’s Law” for ADCs: technology Figure of Merit incorporating sampling rate and dynamic range doubles every 6-8 years Analog-to-Information (A2I) converter wideband signals have high Nyquist rate but are often sparse/compressible develop new ADC technologies to exploit new tradeoffs among Nyquist rate, sampling rate, dynamic range, … frequency hopper spectrogram frequency time

Random Demodulator

Sampling Rate Goal: Sample near signal’s (low) “information rate” rather than its (high) Nyquist rate A2I sampling rate number of tones / window Nyquist bandwidth

Example: Frequency Hopper 20x sub-Nyquist sampling Nyquist rate sampling spectrogram sparsogram

Example: Frequency Hopper 20x sub-Nyquist sampling Nyquist rate sampling spectrogram sparsogram conventional ADC CS-based AIC 20MHz sampling rate 1MHz sampling rate

More CS In Action CS makes sense when measurements are expensive Coded imagers x-ray, gamma-ray, IR, THz, … Camera networks sensing/compression/fusion Array processing exploit spatial sparsity of targets Ultrawideband A/D converters exploit sparsity in frequency domain Medical imaging MRI, CT, ultrasound …

Pros and Cons of Compressive Sensing

CS – Pro – Measurement Noise Stable recovery with additive measurement noise Noise is added to Stability: noise only mildly amplified in recovered signal

CS – Con – Signal Noise Often seek recovery with additive signal noise Noise is added to Noise folding: signal noise amplified in by 3dB for every doubling of Same effect seen in classical “bandpass subsampling”

CS – Con – Noise Folding slope = -3 CS recovered signal SNR

CS – Pro – Dynamic Range As amount of subsampling grows, can employ an ADC with a lower sampling rate and hence higher-resolution quantizer

Dynamic Range Corollary: CS can significantly boost the ENOB of an ADC system for sparse signals CS ADC w/ sparsity conventional ADC

CS – Pro – Dynamic Range As amount of subsampling grows, can employ an ADC with a lower sampling rate and hence higher-resolution quantizer Thus dynamic range of CS ADC can far exceed Nyquist ADC With current ADC trends, dynamic range gain is theoretically 7.9dB for each doubling in

CS – Pro – Dynamic Range slope = +5 (almost 7.9) dynamic range

CS – Pro vs. Con SNR: 3dB loss for each doubling of Dynamic Range: up to 7.9dB gain for each doubling of

Summary: CS Compressive sensing randomized dimensionality reduction exploits signal sparsity information integrates sensing, compression, processing Why it works: with high probability, random projections preserve information in signals with concise geometric structures Enables new sensing architectures cameras, imaging systems, ADCs, radios, arrays, … Important to understand noise-folding/dynamic range trade space

Open Research Issues Links with information theory new encoding matrix design via codes (LDPC, fountains) new decoding algorithms (BP, AMP, etc.) quantization and rate distortion theory Links with machine learning Johnson-Lindenstrauss, manifold embedding, RIP Processing/inference on random projections filtering, tracking, interference cancellation, … Multi-signal CS array processing, localization, sensor networks, … CS hardware ADCs, receivers, cameras, imagers, arrays, radars, sonars, … 1-bit CS and stable embeddings

dsp.rice.edu/cs