1. Rice/Duke | Compressive Optical Devices | August 2007 Richard Baraniuk Kevin Kelly Rice University Compressive Optical Imaging Systems – Theory, Devices, Implementation David Brady Rebecca Willett Duke University

2. Rice/Duke | Compressive Optical Devices | August 2007 Project Overview Richard Baraniuk

3. Digital Revolution

4. camera arrayshyperspectral cameras distributed camera networks

5. Sensing by Sampling Long-established paradigm for digital data acquisition –sample data at Nyquist rate (2x bandwidth) –compress data (signal-dependent, nonlinear) –brick wall to resolution/performance compress transmit/store receivedecompress sample sparse wavelet transform

6. Compressive Sensing (CS) Directly acquire “compressed” data Replace samples by more general “measurements” compressive sensing transmit/store receivereconstruct

7. Compressive Sensing When data is sparse/compressible, can directly acquire a condensed representation with no/little information loss through dimensionality reduction measurements sparse signal sparse in some basis

8. Compressive Sensing When data is sparse/compressible, can directly acquire a condensed representation with no/little information loss Random projection will work measurements sparse signal sparse in some basis [Candes-Romberg-Tao, Donoho, 2004] for signal reconstruction

9. Compressive Optical Imaging Systems – Theory, Devices, and Implementation $400k budget for roughly April 2006-2007 –administered by ONR –Rice portion expended; Duke portion in NCE Goals: –forge collaboration between Rice and Duke teams –demonstrate new Compressive Imaging technologies  hardware testbeds/demos at Rice and Duke  new theory/algorithms –quantify performance –articulate emerging directions Collaborations: –telecons, visits, joint projects, joint papers, artwork

10. Rice/Duke | Compressive Optical Devices | August 2007 Gerhard Richter 4096 Farben / 4096 Colours 1974 254 cm X 254 cm Laquer on Canvas Catalogue Raisonné: 359 Museum Collection: Staatliche Kunstsammlungen Dresden (on loan) Sales history: 11 May 2004 Christie's New York Post-War and Contemporary Art (Evening Sale), Lot 34 US$3,703,500

11. Rice/Duke | Compressive Optical Devices | August 2007 Gerhard Richter Dresden Cathedral Stained Glass

12. Agenda Rebecca Willett, Duke[theory/algorithms] Kevin Kelly, Rice[hardware] David Brady, Duke[hardware] Richard Baraniuk, Rice[theory/algorithms] Discussion and Conclusions

13. Rice/Duke | Compressive Optical Devices | August 2007 Compressive Image Processing Richard Baraniuk

14. Mike Wakin Marco Duarte Mark Davenport Shri Sarvotham Petros Boufounos Matthew Moravec Mona Sheikh Jason Laska

15. Rice/Duke | Compressive Optical Devices | August 2007 Image Classification/Segmentation using Duke Hyperspectral System (with Rebecca Willett)

16. Information Scalability If we can reconstruct a signal from compressive measurements, then we should be able to perform other kinds of statistical signal processing: –detection –classification –estimation … Hyperspectral image classification/segmentation

17. Classification Example spectrum 2 spectrum 1 spectrum 3

18. Nearest Projected Neighbor normalize measurements compute nearest neighbor

19. Naïve Results block size 32 16 8

20. Results naïve independent classification tree-based classification

21. Voting / Cycle Spinning block radius in pixels 162024 2832

22. Summary Direct hyperspectral classification/segmentation without reconstructing 3D data cube Future directions –replace nearest projected neighbor with more sophisticated methods  smashed filter  projected SVM  quad-tree based multiscale segmentation (HMTseg, …) Joint paper in the works

23. Rice/Duke | Compressive Optical Devices | August 2007 Performance Analysis of Multiplexed Cameras

24. Single-Pixel Camera Analysis random pattern on DMD array DMD photon detector image reconstruction or processing Analyze performance in terms of –dynamic range and #bits of A/D –MSE due to photon counting noise –number of measurements

25. Rice/Duke | Compressive Optical Devices | August 2007 Single Pixel Image Acquisiton For a N -pixel, K -sparse image under T -second exposure: Raster Scan: Acquire one pixel at a time, repeat N times Basis Scan:Acquire one coefficient of image in a fixed basis at a time, repeat N times CS Scan:Acquire one incoherent/random projection of the image at a time, repeat times

26. Rice/Duke | Compressive Optical Devices | August 2007 Worst-Case Performance N : Number of pixels P : Number of photons per pixel T : Total capture time M : Number of measurements C N : CS noise amplification constant Sensor array shown as baseline Table shows requirements to match worst-case performance CS beats Basis Scan if

27. Rice/Duke | Compressive Optical Devices | August 2007 Single Pixel Camera Experimental Performance N = 16384 M = 1640  = Daub-8

28. Multiplexed Camera Analysis random pattern on DMD array DMD S photon detectors image reconstruction or processing lens(es) Dude, you gotta multiplex!

29. Rice/Duke | Compressive Optical Devices | August 2007 S -Pixel Camera Performance N : Number of pixels P : Number of photons per pixel T : Total capture time M : Number of measurements C N : CS noise amplification constant Sensor array shown as baseline M measurements split across S sensors Single pixel camera: S = 1

30. Rice/Duke | Compressive Optical Devices | August 2007 S -Pixel Camera Performance N : Number of pixels P : Number of photons per pixel T : Total capture time M : Number of measurements C N : CS noise amplification constant Sensor array shown as baseline M measurements split across S sensors Single pixel camera: S = 1 CS beats Basis Scan if

31. Rice/Duke | Compressive Optical Devices | August 2007 Smashed Filter – Compressive Matched Filtering

32. Information Scalability If we can reconstruct a signal from compressive measurements, then we should be able to perform other kinds of statistical signal processing: –detection –classification –estimation … Smashed filter: compressive matched filter

33. Matched Filter Signal classification in additive white Gaussian noise –LRT: classify test signal as from Class i if it is closest to template signal i –GLRT: when test signal can be a transformed version of template, use matched filter When signal transformations are well-behaved, transformed templates form low-dimensional manifolds –GLRT = matched filter = nearest manifold classification M1M1 M2M2 M3M3

34. Compressive LRT Compressive observations By the Johnson-Lindenstrauss Lemma, random projection preserves pairwise distances with high probability

35. Smashed Filter Compressive observations of transformed signal Theorem: Structure of smooth manifolds is preserved by random projection w.h.p. provided distances, geodesic distances, angles, volume, dimensionality, topology, local neighborhoods, … [Wakin et al 2006; to appear in Foundations on Computational Mathematics] M1M1 M2M2 M3M3 M1M1 M2M2 M3M3

36. Stable Manifold Embedding Theorem: Let F ½ R N be a compact K -dimensional manifold with –condition number 1/ (curvature, self-avoiding) –volume V Let  be a random M x N orthoprojector with [Wakin et al 2006] Then with probability at least 1-, the following statement holds: For every pair x, y 2 F

37. Manifold Learning from Compressive Measurements ISOMAPHLLE Laplacian Eigenmaps R 4096 RMRM M =15 M =20

38. Smashed Filter – Experiments 3 image classes: tank, school bus, SUV N = 65536 pixels Imaged using single-pixel CS camera with –unknown shift –unknown rotation

39. Smashed Filter – Unknown Position Object shifted at random ( K =2 manifold) Noise added to measurements Goal: identify most likely position for each image class identify most likely class using nearest-neighbor test number of measurements M avg. shift estimate error classification rate (%) more noise

40. Smashed Filter – Unknown Rotation Object rotated each 2 degrees Goals: identify most likely rotation for each image class identify most likely class using nearest-neighbor test Perfect classification with as few as 6 measurements Good estimates of rotation with under 10 measurements number of measurements M avg. rot. est. error

41. How Low Can M Go? Empirical evidence that many fewer than measurements are needed for effective classification Late-breaking results (experimental+nascent theory)

42. Summary – Smashed Filter Compressive measurements are info scalable reconstruction > estimation > classification > detection Random projections preserve structure of smooth manifolds (analogous to sparse signals) Smashed filter: dimension-reduced GLRT for parametrically transformed signals –exploits compressive measurements and manifold structure –broadly applicable: targets do not have to have sparse representation in any basis –effective for detection/classification –number of measurements required appears to be independent of the ambient dimension

43. Rice/Duke | Compressive Optical Devices | August 2007 Compressive Phase Retrieval for Fourier Imagers

44. Coherent Diffraction Imaging Image by sampling in Fourier domain Challenge:we observe only the magnitude of the Fourier measurements

45. Phase Retrieval Given: Fourier magnitude + additional constraints (typically support) Goal:Estimate phase of Fourier transform Compressive Phase Retrieval (CPR) replace image support constraint with a sparsity/compressibility constraint nonconvex reconstruction

47. Rice/Duke | Compressive Optical Devices | August 2007 Conclusions and Future Directions

48. Rice/Duke | Compressive Optical Devices | August 2007 Project Outcomes Forged collaboration between Rice and Duke teams –several joint papers in progress Demonstrated new Compressive Imaging technologies –hardware testbeds/demos  hyperspectral, low-light, infrared DMD cameras  coded aperture spectral imagers –new theory/algorithms  spectral image reconstruction/classification methods  smashed filter Quantified performance –coded aperture tradeoffs –multiplexing tradeoff –number of measurements required for reconstruction/classification

49. Emerging Directions Nonimaging cameras –exploit information scalability –attentive/adaptive cameras –meta-analysis –separating “imaging process” from “display” Multiple cameras –image beamforming, 3D geometry imaging, … Deeper links between physics and signal processing –significance of coherence and spectral projections Links to analog-to-information program –nonidealities as challenges vs. opportunities Other modalities –THz, LWIR/MWIR, UV, soft x-rays, …

51. Rice/Duke | Compressive Optical Devices | August 2007 N - Pixel Camera Performance N : Number of pixels P : Number of photons per pixel T : Total capture time M : Number of measurements C N : CS noise amplification constant Sensor array shown as baseline 1 sensor per pixel - CS is unnecessary

52. Rice/Duke | Compressive Optical Devices | August 2007 Smashed Filter under Poisson noise Problem: vehicle image classification under variable parameter (shift, rotation, etc.) Image acquisition: M random projections under signal-dependent (Poisson) noise with single pixel camera Limited capture time T split among M projections Solution: use articulation manifold structure and generalized maximum likelihood classification (smashed filter)

53. Rice/Duke | Compressive Optical Devices | August 2007 Smashed Filter performance under Poisson noise Shift (2D manifold) Rotation (1D manifold) Small number of measurements M for good performance “Sweet spot” on M for shorter exposures T

54. CS Hallmarks CS changes the rules of the data acquisition game –exploits a priori signal sparsity information Universal –same random projections / hardware can be used for any compressible signal class (generic) Democratic –each measurement carries the same amount of information –simple encoding –robust to measurement loss and quantization Asymmetrical (most processing at decoder) Random projections weakly encrypted

55. Rice/Duke | Compressive Optical Devices | August 2007 Smashed Filter: How Low Can M Go?

56. Rice/Duke | Compressive Optical Devices | August 2007 Preservation of Manifold Structure Manifold Learning  Used for classification, visualization of high dimensional data, robust parameter estimation Network of single-pixel cameras ==== Randomly projected version of low-dimensional image manifold.  New result: stable manifold learning is possible without ever reconstructing the original images  Number of measurements sufficient for arbitrarily small learning error: linear in the information level K of the manifold

57. Rice/Duke | Compressive Optical Devices | August 2007 Translating disk manifold Learning algorithm: LTSA (Zhang, Zha. 2004.) ‏ 25 random projections 50 random projections 100 random projections N = 64 x 64 = 4096, K = 2 Learning with original data: (N = 4096) ‏

58. Rice/Duke | Compressive Optical Devices | August 2007 Manifold learning using random projections Demonstrates that random projections contain sufficient information about the manifold structure Two stages in manifold learning –Intrinsic dimension estimation –Construction of nonlinear map into low-dimensional Euclidean space –New result: estimation errors in both stages due to dimensionality-reducing projections can be controlled up to arbitrary accuracy with small number of measurements Ideal for distributed networks; sensors need to transmit very few pieces of information to the centralized learning algorithm

59. Rice/Duke | Compressive Optical Devices | August 2007 Intrinsic Dimension estimation GP algorithm used directly on random projections of hyperspheres Empirically compute the number of measurements required for estimate to be within 10% of the original. Observation: M linear in the intrinsic dimension K

60. Rice/Duke | Compressive Optical Devices | August 2007 Real data: Hand rotation database N = 64 x 60 = 3840, K = 2

61. Rice/Duke | Compressive Optical Devices | August 2007 New Bound for Classification? Smashed Filter – Nearest Neighbor classifier Indyk, Naor. 2007 : preservation of approximate nearest neighbors requires merely O(K) random projections Minimum number of measurements required for classification in noiseless case (where D is the minimum separation between signal classes ):

62. Rice/Duke | Compressive Optical Devices | August 2007 Experiment: Hyperspherical manifolds 1000 labeled training samples each from two unit 3-dimensional hyperspheres, separated by a distance D along an arbitrary direction in 2000- dimensional space Generate unlabeled samples, perform nearest neighbor classification in the compressed (“smashed”) domain Determine minimum number of measurements M required to obtain 99% classification rate. Bound: M decreases as square of the separation distance.

63. Rice/Duke | Compressive Optical Devices | August 2007 Hyperspherical manifolds: empirical verification of bound

64. Why Does CS Work (1)? Random projection not full rank, but stably embeds –sparse/compressible signal models (CS) –point clouds (JL) into lower dimensional space with high probability Stable embedding: preserves structure –distances between points, angles between vectors, …

65. Why Does CS Work (1)? Random projection not full rank, but stably embeds –sparse/compressible signal models (CS) –point clouds (JL) into lower dimensional space with high probability Stable embedding: preserves structure –distances between points, angles between vectors, … provided M is large enough: Compressive Sensing K -dim planes K -sparse model

66. CS Signal Recovery Recover sparse/compressible signal x from CS measurements y via linear programming K -dim planes K -sparse model recovery

67. Why Does CS Work (2)? Random projection not full rank, but stably embeds –sparse/compressible signal models (CS) –point clouds (JL) into lower dimensional space with high probability Stable embedding: preserves structure –distances between points, angles between vectors, … provided M is large enough: Johnson-Lindenstrauss Q points

69. Tree-based classification Refine classification of blocks having neighbors from a different class

70. Tree-based classification Refine classification of blocks having neighbors from a different class

71. Tree-based classification Refine classification of blocks having neighbors from a different class