1. Compressive Signal
Processing
Richard Baraniuk
Rice University

2. Better, Stronger, Faster

3. Sense by Sampling sample
Example: Large Hadron Collider will produce 10 peta bytes of data per second; need to triage that down to 100/sec “potentially interesting events” in real time

4. Sense by Sampling too sample much data!
Example: Large Hadron Collider will produce 10 peta bytes of data per second; need to triage that down to 100/sec “potentially interesting events” in real time

5. Accelerating Data Deluge
1250 billion gigabytes generated in 2010
# digital bits > # stars in the universe
growing by a factor of 10 every 5 years
Total data generated > total storage
Increases in generation rate >> increases in comm rate
Available transmission bandwidth

7. Sense then Compress
sample
compress
JPEG
JPEG2000 …
decompress

8. Sparsity
large wavelet coefficients
(blue = 0)
pixels

9. Sparsity large wavelet coefficients pixels wideband signal samples
(blue = 0)
pixels
wideband signal samples
large Gabor (TF) coefficients
frequency
time

10. Concise Signal Structure
Sparse signal: only K out of N coordinates nonzero
sparse signal
nonzero entries
sorted index

11. Concise Signal Structure
Sparse signal: only K out of N coordinates nonzero
model: union of K-dimensional subspaces aligned w/ coordinate axes
sparse signal
nonzero entries
sorted index

12. Concise Signal Structure
Sparse signal: only K out of N coordinates nonzero
model: union of K-dimensional subspaces
Compressible signal: sorted coordinates decay rapidly with power-law approximately sparse
power-law decay
sorted index

13. Concise Signal Structure
Sparse signal: only K out of N coordinates nonzero
model: union of K-dimensional subspaces
Compressible signal: sorted coordinates decay rapidly with power-law
model: ball:
power-law decay
sorted index

14. What’s Wrong with this Picture?
Why go to all the work to acquire N samples only to discard all but K pieces of data?
sample
compress
decompress

15. What’s Wrong with this Picture?
linear processing
linear signal model
(bandlimited subspace)
nonlinear processing
nonlinear signal model
(union of subspaces)
sample
compress
decompress

16. Compressive Sensing
Directly acquire “compressed” data via dimensionality reduction
Replace samples by more general “measurements”
compressive sensing
recover

17. Sampling Signal is -sparse in basis/dictionary
WLOG assume sparse in space domain
sparse signal
nonzero entries

18. Sampling Signal is -sparse in basis/dictionary Sampling
WLOG assume sparse in space domain
Sampling
sparse signal
measurements
nonzero entries

19. Compressive Sampling
When data is sparse/compressible, can directly acquire a condensed representation with no/little information loss through linear dimensionality reduction
sparse signal
measurements
nonzero
entries

20. How Can It Work?
Projection not full rank… … and so loses information in general
Ex: Infinitely many ’s map to the same (null space)

21. How Can It Work?
Projection not full rank… … and so loses information in general
But we are only interested in sparse vectors
columns

22. How Can It Work?
Projection not full rank… … and so loses information in general
But we are only interested in sparse vectors
is effectively MxK
columns

23. How Can It Work?
Projection not full rank… … and so loses information in general
But we are only interested in sparse vectors
Design so that each of its MxK submatrices are full rank (ideally close to orthobasis)
Restricted Isometry Property (RIP)
see also phase transition approach of Donoho et al.
columns

24. RIP = Stable Embedding
An information preserving projection preserves the geometry of the set of sparse signals
RIP ensures that
K-dim subspaces 24

25. RIP = Stable Embedding
An information preserving projection preserves the geometry of the set of sparse signals
RIP ensures that 25

26. How Can It Work?
Projection not full rank… … and so loses information in general
Design so that each of its MxK submatrices are full rank (RIP)
Unfortunately, a combinatorial, NP-Hard design problem
columns

27. Insight from the 70’s [Kashin, Gluskin]
Draw at random
iid Gaussian
iid Bernoulli …
Then has the RIP with high probability provided
columns

28. Randomized Sensing
Measurements = random linear combinations of the entries of
No information loss for sparse vectors whp
sparse signal
measurements
nonzero
entries

29. CS Signal Recovery Goal: Recover signal from measurements
Problem: Random projection not full rank (ill-posed inverse problem)
Solution: Exploit the sparse/compressible geometry of acquired signal

30. CS Signal Recovery Random projection not full rank
Recovery problem: given find
Null space
Search in null space for the “best” according to some criterion
ex: least squares
(N-M)-dim hyperplane at random angle

31. Signal Recovery
Recovery: given (ill-posed inverse problem) find (sparse)
Optimization:
Closed-form solution:

32. Signal Recovery
Recovery: given (ill-posed inverse problem) find (sparse)
Optimization:
Closed-form solution:
Wrong answer! 32

33. Signal Recovery
Recovery: given (ill-posed inverse problem) find (sparse)
Optimization:
Closed-form solution:
Wrong answer! 33

34. Signal Recovery
Recovery: given (ill-posed inverse problem) find (sparse)
Optimization:
“find sparsest vector in translated nullspace” 34

35. Signal Recovery
Recovery: given (ill-posed inverse problem) find (sparse)
Optimization:
Correct!
“find sparsest vector in translated nullspace” 35

36. Signal Recovery
Recovery: given (ill-posed inverse problem) find (sparse)
Optimization:
Correct!
But NP-Complete alg
“find sparsest vector in translated nullspace” 36

37. Signal Recovery
Recovery: given (ill-posed inverse problem) find (sparse)
Optimization:
Convexify the optimization
Donoho
Candes Romberg Tao 37

38. Signal Recovery
Recovery: given (ill-posed inverse problem) find (sparse)
Optimization:
Convexify the optimization
Correct!
Polynomial time alg (linear programming)
Much recent alg progress
greedy, Bayesian approaches, … 38

39. CS Hallmarks Stable Asymmetrical (most processing at decoder)
acquisition/recovery process is numerically stable
Asymmetrical (most processing at decoder)
conventional: smart encoder, dumb decoder
CS: dumb encoder, smart decoder
Democratic
each measurement carries the same amount of information
robust to measurement loss and quantization
“digital fountain” property
Random measurements encrypted
Universal
same random projections / hardware can be used for any sparse signal class (generic) 39

40. Universality
Random measurements can be used for signals sparse in any basis

41. Universality
Random measurements can be used for signals sparse in any basis

42. Universality
Random measurements can be used for signals sparse in any basis
sparse coefficient
vector
nonzero
entries

43. Compressive Sensing In Action

44. “Single-Pixel” CS Camera
scene
single photon detector
image reconstruction or processing
DMD
DMD
random
pattern on
DMD array
DMD is used in projectors
Multiply value of random pattern in mirror with value of signal (light intensity) in pixel
lens is focused onto the photodiode
w/ Kevin Kelly

45. “Single-Pixel” CS Camera
scene
single photon detector
image reconstruction or processing
DMD
DMD
random
pattern on
DMD array
DMD is used in projectors
Multiply value of random pattern in mirror with value of signal (light intensity) in pixel
lens is focused onto the photodiode …
Flip mirror array M times to acquire M measurements
Sparsity-based (linear programming) recovery

46. First Image Acquisition
target pixels
11000 measurements (16%)
1300 measurements
(2%)

47. single photon detector
Utility?
Fairchild
100Mpixel
CCD
single photon detector
DMD is used in projectors
Multiply value of random pattern in mirror with value of signal (light intensity) in pixel
lens is focused onto the photodiode
DMD
DMD

48. SWIR CS Camera Target (illuminated in SWIR only) Camera output
InView “single-pixel”
SWIR Camera (1024x768)
Target (illuminated in SWIR only)
Camera output
M = 0.5 N

49. CS Hyperspectral Imager
(Kevin Kelly Lab, Rice U)
spectrometer
hyperspectral data cube nm
N=1M space x wavelength voxels
M=200k random measurements

50. CS-MUVI for Video CS Pendulum speed: 2 sec/cycle
Naïve, block-based L1 recovery of 64x64 video frames for 3 different values of W
1024
2048
4096

51. Recovered video (animated)
CS-MUVI for Video CS
Effective “compression ratio” = 60:1
Low-res preview
(32x32)
High-res video recovery (128x128)
Recovered video (animated)

52. Analog-to-Digital Conversion
Nyquist rate limits reach of today’s ADCs
“Moore’s Law” for ADCs:
technology Figure of Merit incorporating sampling rate and dynamic range doubles every 6-8 years
Analog-to-Information (A2I) converter
wideband signals have high Nyquist rate but are often sparse/compressible
develop new ADC technologies to exploit
new tradeoffs among Nyquist rate, sampling rate, dynamic range, …
frequency hopper spectrogram
frequency
time

53. Random Demodulator

54. Sampling Rate
Goal: Sample near signal’s (low) “information rate” rather than its (high) Nyquist rate
A2I sampling rate
number of tones / window
Nyquist bandwidth

55. Example: Frequency Hopper
20x sub-Nyquist sampling
Nyquist rate sampling
spectrogram
sparsogram

56. Example: Frequency Hopper
20x sub-Nyquist sampling
Nyquist rate sampling
spectrogram
sparsogram
conventional ADC
CS-based AIC
20MHz sampling rate
1MHz sampling rate

57. More CS In Action CS makes sense when measurements are expensive
Coded imagers
x-ray, gamma-ray, IR, THz, …
Camera networks
sensing/compression/fusion
Array processing
exploit spatial sparsity of targets
Ultrawideband A/D converters
exploit sparsity in frequency domain
Medical imaging
MRI, CT, ultrasound …

58. Pros and Cons of Compressive Sensing

59. CS – Pro – Measurement Noise
Stable recovery with additive measurement noise
Noise is added to
Stability: noise only mildly amplified in recovered signal

60. CS – Con – Signal Noise Often seek recovery with additive signal noise
Noise is added to
Noise folding: signal noise amplified in by dB for every doubling of
Same effect seen in classical “bandpass subsampling”

61. CS – Con – Noise Folding
slope = -3
CS recovered signal SNR

62. CS – Pro – Dynamic Range
As amount of subsampling grows, can employ an ADC with a lower sampling rate and hence higher-resolution quantizer

63. Dynamic Range
Corollary: CS can significantly boost the ENOB of an ADC system for sparse signals
CS ADC w/ sparsity
conventional ADC

64. CS – Pro – Dynamic Range
As amount of subsampling grows, can employ an ADC with a lower sampling rate and hence higher-resolution quantizer
Thus dynamic range of CS ADC can far exceed Nyquist ADC
With current ADC trends, dynamic range gain is theoretically 7.9dB for each doubling in

65. CS – Pro – Dynamic Range
slope = +5 (almost 7.9)
dynamic range

66. CS – Pro vs. Con SNR: 3dB loss for each doubling of
Dynamic Range: up to 7.9dB gain for each doubling of

67. Summary: CS Compressive sensing
randomized dimensionality reduction
exploits signal sparsity information
integrates sensing, compression, processing
Why it works: with high probability, random projections preserve information in signals with concise geometric structures
Enables new sensing architectures
cameras, imaging systems, ADCs, radios, arrays, …
Important to understand noise-folding/dynamic range trade space

68. Open Research Issues Links with information theory
new encoding matrix design via codes (LDPC, fountains)
new decoding algorithms (BP, AMP, etc.)
quantization and rate distortion theory
Links with machine learning
Johnson-Lindenstrauss, manifold embedding, RIP
Processing/inference on random projections
filtering, tracking, interference cancellation, …
Multi-signal CS
array processing, localization, sensor networks, …
CS hardware
ADCs, receivers, cameras, imagers, arrays, radars, sonars, …
1-bit CS and stable embeddings

69. dsp.rice.edu/cs