2. High-resolution Hyperspectral Imaging via Matrix Factorization Rei Kawakami 1 John Wright 2 Yu-Wing Tai 3 Yasuyuki Matsushita 2 Moshe Ben-Ezra 2 Katsushi Ikeuchi 3 1 University of Tokyo, 2 Microsoft Research Asia (MSRA), 3 Korea Advanced Institute of Science and Technology (KAIST) CVPR 11

3. Giga-pixel Camera M. Ezra et al. Giga-pixel Camera @ Microsoft research Large-format lensCCD

4. Spectrum

5. RGB vs. Spectrum

6. Approach Low-res hyperspectral high-res RGB High-res hyperspectral image Combine

7. Problem formulation W (Image width) H (Image height) S Goal: Given:

8. Representation: Basis function W (Image width) H (Image height) S = … 0 1.0 0 … 0 = + x 0x 1.0x 0 ++

9. Two-step approach 1.Estimate basis functions from hyperspectral image 2.For each pixel in high-res RGB image, estimate coefficients for the basis functions

10. 1: Limited number of materials Sparse vector For all pixel (i,j) Sparse matrix W (Image width) H (Image height) S = … 0 0.4 0 … 0.6

11. 2: Sparsity in high-res image W H S Sparse coefficients Reconstruction

12. Simulation experiments

13. 460 nm550 nm620 nm 460 nm550 nm620 nm

14. 430 nm490 nm550 nm610 nm670 nm

15. Error images of Global PCA with back- projection Error images of local window with back-projection Error images of RGB clustering with back-projection

16. Estimated 430 nm

17. Ground truth

18. RGB image

19. Error image

22. HS camera Filter CMOSLens Aperture Focus Translational stage

23. Real data experiment Input RGBInput (550nm)Input (620nm)Estimated (550nm)Estimated (620nm)

24. Summary Method to reconstruct high-resolution hyperspectral image from ▫Low-res hyperspectral camera ▫High-res RGB camera Spatial sparsity of hyperspectral input ▫Search for a factorization of the input into  basis functions  set of maximally sparse coefficients