Compressed Sensing & Sparse Signal Recovery Trần Duy Trác ECE Department The Johns Hopkins University Baltimore, MD 21218

Outline Compressed Sensing: Quick Overview Motivations. Toy Examples Incoherent Bases and Restrictive Isometry Property Decoding Strategy L0 versus L1 versus L2 Basis Pursuit and Matching Pursuit Compressed Sensing in Video Processing 2D Separable Measurement Ensemble (SME) Distributed compressed video sensing (DISCOS) Layered compressed sensing for robust video transmission

Compressed Sensing History Emmanuel Candès and Terence Tao, ”Decoding by linear programming” IEEE Trans. on Information Theory, 51(12), pp. 4203 - 4215, December 2005 Emmanuel Candès, Justin Romberg, and Terence Tao, ”Robust uncertainty principles: Exact signal reconstruction from highly incomplete frequency information,” IEEE Trans. on Information Theory, 52(2) pp. 489 - 509, Feb. 2006. David Donoho, ”Compressed sensing,” IEEE Trans. on Information Theory, 52(4), pp. 1289 - 1306, Apr. 2006. Emmanuel Candès and Michael Wakin, ”An introduction to compressive sampling,” IEEE Signal Processing Magazine, 25(2), pp. 21 - 30, Mar. 2008.

Traditional Data Acquisition: Sampling Shannon Sampling Theorem In order for a band-limited signal x(t) to be reconstructed perfectly, it must be sampled at rate t x(t) ^ t x(t)

Traditional Compression Paradigm Receive Decompress Sample Compress Transmit/Store MP3, JPEG, JPEG200, MPEG… samples largest coefficients Sample first and then worry about compression later!

Sparse Signals Digital signals in practice are often sparse basis functions largest coefficients transform coefficients Digital signals in practice are often sparse Audio: MP3, AAC… ~10:1 compression Images: JPEG, JPEG2000… ~20:1 compression Video sequences: MPEG2, MPEG4… ~40:1 compression

Sparse Signals II basis functions N-pixel image transform coefficients nonzero entries : -sparse signal in non-sparse domain : -sparse signal

Definition & Notation N = length of signal x K = the sparsity level of x or x is called K-sparse M = the number of measurements (samples) taken at the encoder

Compressed Sensing Framework Encoding: obtain M measurements y from linear projection onto an incoherent basis Sensing matrix = has only K nonzero entries Compressed measurements Decoding: reconstruct x from measurements y via nonlinear optimization with sparsity prior

At Encoder: Signal Sensing y is not sparse, looks iid Random projection works well! Sensing & sparsifying matrix must be incoherent Each measurement y contains a little information of each sample of x

At Decoder: Signal Reconstruction Recover x from the set of measurements y Without the sparseness assumption, the problem is ill-posed With sparseness assumption, the L0-norm minimization problem is well-posed but computationally intractable With sparseness assumption, the L1-norm minimization can be solved via linear programming – Basis Pursuit!

L0- and L1-norm Reconstruction - norm reconstruction : take advantage of sparsity prior We find the sparsest solution Problem: Combinatorial searching Exhaustive computation - norm reconstruction : Compressed sensing framework This is a convex optimization problem Using linear programming to solve Also can find the sparsest which turns out to be the exact solution - 12 -

L1-Minimization Problem Let Standard LP Many available techniques Simplex, primal-dual interior-point, log-barrier…

CS Reconstruction: Matching Pursuit Problem Basis Pursuit Greedy Pursuit – Iterative Algorithms At each iteration, try to identify columns of A (atoms) that are associated with non-zero entries of x significant atoms y A x

Matching Pursuit MP: At each iteration, MP attempts to identify the most significant atom. After K iteration, MP will hopefully identify the signal! , set residual vector , selected index set Find index yielding the maximal correlation with the residue Augment selected index set: Update the residue: , and stop when t = K

Orthogonal Matching Pursuit OMP: guarantees that the residue is orthogonal to all previously chosen atoms  no atom will be selected twice! , set residual vector , index set Find index that yields the maximal correlation with residue Augment Find new signal estimate by solving Set the new residual : , and stop when t = K

Subspace Pursuit SP: pursues the entire subspace that the signal lives in at every iteration steps and adds a backtracking mechanism! Initialization Selected set Signal estimate Residue , go to Step 2; stop when residue energy does not decrease anymore

Incoherent Bases: Definition Suppose signal is sparse in a orthonomal transform domain Take K measurements from an orthonormal sensing matrix Definition : Coherence between and T With ,

Incoherent Bases: Properties Bound of coherence : When is small, we call 2 bases are incoherent Intuition: when 2 bases are incoherent, all entries of matrix are spread out each measurement will contains more information of signal we hope to have small Some pair of incoherent bases : DFT and identity matrix : Gaussian (Bernoulli) matrix and any other basis:

Universality of Incoherent Bases Random Gaussian white noise basis is incoherent with any fixed orthonormal basis with high probability If the signal is sparse in frequency, the sparsifying matrix is Product of is still Gaussian white noise!

Restricted Isometry Property non-zero entries Sufficient condition for exact recovery: All sub-matrices composed of columns are nearly orthogonal

L2-norm Reconstruction - norm reconstruction : classical approach We find with smallest energy Closed-form solution Unfortunately, this method almost never find the sparsest and correct answer

Why Is L1 Better Than L2? The line intersect Bad point The line intersect circle at a non-sparse point The line intersect diamond at the sparse point Unique and exact solution - 23 -

How Many Measurements Are Enough? Theorem (Candes, Romberg and Tao) : Suppose x has support on T , M rows of matrix F is selected uniformly at random from N rows of DFT matrix N x N, then if M obeying : Minimize L1 will recover x exactly with extremely high probability : In practice: is enough to perfectly recover

One-Pixel Compressed Sensing Camera Courtesy of Richard Baraniuk & Kevin Kelly @ Rice

CS Analog-to-Digital Converter

Modulated Wideband Converter

Imaging from a Random Lens Not only possible but also yields better quality when there are a large percentage of missing samples Can under-sample when the scene is incoherent

Compressed Sensing in Medical Imaging Goal so far: Achieve faster MR imaging while maintaining reconstruction quality Methods: under sample discrete Fourier space using some pseudo-random patterns then reconstruct using L1-minimization (Lustig) or homotopic L0-minimization (Trzasko)

Sparsity of MR Images Brain MR images, cardiology dynamic MR images Sparse in Wavelet, DCT domains Not sparse in spatial domain, or finite-difference domain Can be reconstructed with good quality using 5-10% of coefficients Angiogram images Space in finite-difference domain and spatial domain (edges of blood vessels occupy only 5% in space) allow very good Compressed Sensing performance

Sampling Methods Using smooth k-space trajectories Cartesian scanning Radial scanning Spiral scanning Fully sampling in each read-out Under sampling by: Cartesian grid: under sampling in phase encoding (uniform, non-uniform) Radial: angular under-sampling (using less angles) Spiral: using less spirals and randomly perturb spiral trajectories

Sampling Patterns: Spiral & Radial Spiral scanning: uniform density, varying density, and perturbed spiral trajectories New algorithm (FOCUSS) allows reconstruction without angular aliasing artifacts 32

Reconstruction Methods Lustig’s : L1-minimization with non-linear Conjugate Gradient method Trzasko’s: homotopic L0-minimization MIP: maximum intensity projection 33

Reconstruction Results (2DFT) Multi-slice 2DFT fast spin echo CS at 2.4 acceleration.

Results – 3DFT Contrast-enhanced 3D angiography reconstruction results as a function of acceleration. Left Column: Acceleration by LR. Note the diffused boundaries with acceleration. Middle Column: ZF-w/dc reconstruction. Note the increase of apparent noise with acceleration. Right Column: CS reconstruction with TV penalty from randomly under-sampled k-space

Results – Radial scan, FOCUSS reconstruction 1 3 2 4 1 Results – Radial scan, FOCUSS reconstruction Reconstruction results from full-scan with uniform angular sampling between 0◦–360◦. 1st row: Reference reconstruction from 190 views. 2nd row: Reconstruction results from 51 views using LINFBP. 3rd row: Reconstruction results from 51 views using CG-ALONE. 4th row: Reconstruction results from 51 views using PR-FOCUSS 36

Results – spiral (a) Sagittal T2-weighted image of the spine, (b) simulated k-space trajectory (multishot Cartesian spiral, 83% under-sampling), (c) minimum energy solution via zero-filling, (d) reconstruction by L1 minimization, (e) reconstruction by homotopic L0 minimization using (|∇u|,  ) = |∇u|/ (|∇u|+ ), (f) line profile across C6, (g-j) enlargements of (a,c-e), respectively.

Practical Sensing Matrix Design Compressed measurement Sensing matrix = Φ y x 38 38 38

Practical Sensing Matrix Design Sparsifying transform Sparse transform coefficients  Ψ Compressed measurement Sensing matrix = Φ y * = Argmin ||1 s.t. y = ΦΨ Sparse Signal Recovery x* = Ψ* 39 39 39

Structurally Random Matrix (SRM) Random down-sampler Local randomizer Fast transform FFT, WHT, DCT,… D F R Use a different d_{ii} for the randommizer and the downsampler 40 40 40 40

Structurally Random Matrix (SRM) Random downsampler Global randomizer Uniformly random permutation matrix Fast transform FFT, WHT, DCT,… D F R 41 41 41 41

Performance Analysis Theorem [Do, Lu, Nguyen & Tran] Assume a signal x is K-sparse, y=Ax where A=DFR is a SRM. If M ~O(KlogN), with high prob., x can be perfectly recovered from y via a l1 minimization reconstruction Proof Techniques Central Limit Theorem (CLT) Combinatorial CLT Concentration Inequalities Exact Recovery Principle[E. Candes] Do, Lu, Nguyen & Tran, Fast Compressive Sensing using Structurally Random Matrices, (Proceeding of ICASSP, pp. 3369-3372, April, 2008) Emmanuel Candès and Justin Romberg, Sparsity and incoherence in compressive sampling. (Inverse Problems, 23(3) pp. 969-985, 2007) 42 42 42

Structurally random matrices Completely random matrices (Gaussian) Feature Comparison Features Structurally random matrices Completely random matrices (Gaussian) No. of measurements for exact recovery O(KlogN) Sensing complexity NlogN O(KNlogN) Implementation in hardware & optics Very easy Difficult Fast computability Yes No Block processing 1 -1 43 43

Linear Regression noise or innovation new observation linear combination of past collected data

Sparse Model for Concealment • • • Dij ij + eij noise yij missing portion reference frame corrupted frame

Video De-noising noise Clean key frame xB • • • DB B + eB 46 46 46

Video Denoising: Sparse Model xB = DB B + eB = [DB| IB] B eB Noisy Block Identity matrix = WB B *B = Argmin |B|1 s.t. xB = WB B x*B = DB*B Denoised Block Sparsity constrained block prediction 47 47 47

Video Super-Resolution High resolution key frame Noisy low resolution non-key frames Unobservable high resolution non-key frame SB HB = + eB subsampling bluring yB xB noise Typical relationship between LR and HR patches 48 48 48

Video Super-Resolution: Sparse Model Dictionary of neighboring blocks xB= SBHByB + eB = SBHBDB B + eB = [SBHBDB| IB] B eB = WB B *B = Argmin |B|1 s.t. xB = WB B y*B = DB*B High resolution block approximation Sparsity constrained block prediction 49 49 49

Blocking-Effect Elimination Averaging from multiple approximations from shifted and overlapping grids 50 50 50

Joint Registration & Anomaly Detection reference image X test image Y M N …

Change Detection for SAR Images Reference Image Test Image L1-norm Map Change Detection

Conclusion Compressed sensing Sparse recovery A different paradigm for data acquisition Low sampling resolution in incoherent domain Sample less and compute more Simple encoding; most computation at decoder Exploit a priori signal sparsity at decoder Universality Robustness: CS measurements are democratic! Sparse recovery Still spot the common trend given incomplete & inaccurate data samples Detection, classification, recognition Denoising, concealment, enhancement

References http://www.dsp.ece.rice.edu/cs/ http://nuit-blanche.blogspot.com/search/label/compressed%20sensing