1. CS-MUVI Video compressive sensing for spatial multiplexing cameras Aswin Sankaranarayanan, Christoph Studer, Richard G. Baraniuk Rice University

2. Single pixel camera Digital micro-mirror device Photo-detector

3. Single pixel camera Each configuration of micro-mirrors yield ONE compressive measurement Non-visible wavelengths Sensor material costly in IR/UV bands Light throughput Half the light in the scene is directed to the photo-detector Much higher SNR as compared to traditional sensors Digital micro-mirror device Photo-detector

4. Single pixel camera Each configuration of micro-mirrors yield ONE compressive measurement static scene assumption Key question: Can we ignore motion in the scene ? Digital micro-mirror device Photo-detector

5. SPC on a time-varying scene Naïve approach: Collect W measurements together to compute an estimate of an image what happens ? t=1t=W measurements compressive recovery time varying scene

6. SPC on a time-varying scene Tradeoff Temporal resolution vs. spatial resolution t=1 Small W Less motion blur Lower spatial resolution Large W Higher spatial resolution More motion blur t=W (small) t=W (large)

7. SPC on a time-varying scene Lower spatial res. Higher temporal res. Higher spatial res. Lower temporal res. sweet spot

8. Dealing with Motion Motion information can help in obtaining better tradeoffs [Reddy et al. 2011] – State-of-the-art video compression

9. Dealing with Motion Motion information can help in obtaining better tradeoffs [Reddy et al. 2011] – State-of-the-art video compression naïve reconstruction motion estimates

10. Key points Motion blur and the failure of the sparsity assumption – Use least squares recovery ? Recover scene at lower spatial resolution – Lower dimensional problem requires lesser number of measurements – Tradeoff spatial resolution for temporal resolution Least squares and random matrices – Random matrices are ill-conditioned – Noise amplification Hadamard matrices – Orthogonal (no noise amplification) – Maximum light throughput – Optimal for least squares recovery [Harwit and Sloane, 1979]

11. Hadamard + least sq. recovery Hadamard Random

12. Hadamard + least sq. recovery

13. Designing measurement matrices Hadamard matrices – Higher temporal resolution – Poor spatial resolution Random matrices – Guarantees successful l 1 recovery – Full spatial resolution Can we simultaneously have both properties in the same measurement matrix ?

14. Dual-scale sensing (DSS) matrices 1. Start with a row of the Hadamard matrix 2. Upsample 3. Add high-freq. component Key Idea: Constructing high-resolution measurement matrices that have good properties when downsampled

15. CS-MUVI: Algorithm outline t=T t=1 t=t 0 t=t 0 +W t=W 1. obtain measurements with DSS matrices 1. obtain measurements with DSS matrices 2. low- resolution initial estimate 3. motion estimation 4. compressive recovery with motion constraints

16. Simulation result

17. CS-MUVI on SPC Single pixel camera setup Object InGaAs Photo-detector (Short-wave IR) SPC sampling rate: 10,000 sample/s Number of compressive measurements: M = 16,384 Recovered video: N = 128 x 128 x 61 = 61*M

18. CS-MUVI: IR spectrum Joint work with Xu and Kelly Recovered Video initial estimate Upsampled

19. CS-MUVI on SPC Naïve frame-to-frame recovery CS-MUVI Joint work with Xu and Kelly

20. CS-MUVI summary Key ingredients – Novel Measurement matrix design – Exploiting state-of-the-art motion model One of first practical video recovery algorithm for the SMC dsp.rice.edu

21. CS-MUVI summary Limitations – Need a priori knowledge of object speed – Motion at low-resolution – Robustness to errors in motion estimates Future work – Dual-scale to multi-scale matrix constructions – Multi-frame optical flow – Online recovery algorithms dsp.rice.edu