CS-MUVI Video compressive sensing for spatial multiplexing cameras Aswin Sankaranarayanan, Christoph Studer, Richard G. Baraniuk Rice University

Single pixel camera Digital micro-mirror device Photo-detector

Single pixel camera Each configuration of micro-mirrors yield ONE compressive measurement Non-visible wavelengths Sensor material costly in IR/UV bands Light throughput Half the light in the scene is directed to the photo-detector Much higher SNR as compared to traditional sensors Digital micro-mirror device Photo-detector

Single pixel camera Each configuration of micro-mirrors yield ONE compressive measurement static scene assumption Key question: Can we ignore motion in the scene ? Digital micro-mirror device Photo-detector

SPC on a time-varying scene Naïve approach: Collect W measurements together to compute an estimate of an image what happens ? t=1t=W measurements compressive recovery time varying scene

SPC on a time-varying scene Tradeoff Temporal resolution vs. spatial resolution t=1 Small W Less motion blur Lower spatial resolution Large W Higher spatial resolution More motion blur t=W (small) t=W (large)

SPC on a time-varying scene Lower spatial res. Higher temporal res. Higher spatial res. Lower temporal res. sweet spot

Dealing with Motion Motion information can help in obtaining better tradeoffs [Reddy et al. 2011] – State-of-the-art video compression

Dealing with Motion Motion information can help in obtaining better tradeoffs [Reddy et al. 2011] – State-of-the-art video compression naïve reconstruction motion estimates

Key points Motion blur and the failure of the sparsity assumption – Use least squares recovery ? Recover scene at lower spatial resolution – Lower dimensional problem requires lesser number of measurements – Tradeoff spatial resolution for temporal resolution Least squares and random matrices – Random matrices are ill-conditioned – Noise amplification Hadamard matrices – Orthogonal (no noise amplification) – Maximum light throughput – Optimal for least squares recovery [Harwit and Sloane, 1979]

Hadamard + least sq. recovery Hadamard Random

Hadamard + least sq. recovery

Designing measurement matrices Hadamard matrices – Higher temporal resolution – Poor spatial resolution Random matrices – Guarantees successful l 1 recovery – Full spatial resolution Can we simultaneously have both properties in the same measurement matrix ?

Dual-scale sensing (DSS) matrices 1. Start with a row of the Hadamard matrix 2. Upsample 3. Add high-freq. component Key Idea: Constructing high-resolution measurement matrices that have good properties when downsampled

CS-MUVI: Algorithm outline t=T t=1 t=t 0 t=t 0 +W t=W 1. obtain measurements with DSS matrices 1. obtain measurements with DSS matrices 2. low- resolution initial estimate 3. motion estimation 4. compressive recovery with motion constraints

Simulation result

CS-MUVI on SPC Single pixel camera setup Object InGaAs Photo-detector (Short-wave IR) SPC sampling rate: 10,000 sample/s Number of compressive measurements: M = 16,384 Recovered video: N = 128 x 128 x 61 = 61*M

CS-MUVI: IR spectrum Joint work with Xu and Kelly Recovered Video initial estimate Upsampled

CS-MUVI on SPC Naïve frame-to-frame recovery CS-MUVI Joint work with Xu and Kelly

CS-MUVI summary Key ingredients – Novel Measurement matrix design – Exploiting state-of-the-art motion model One of first practical video recovery algorithm for the SMC dsp.rice.edu

CS-MUVI summary Limitations – Need a priori knowledge of object speed – Motion at low-resolution – Robustness to errors in motion estimates Future work – Dual-scale to multi-scale matrix constructions – Multi-frame optical flow – Online recovery algorithms dsp.rice.edu