High-resolution Hyperspectral Imaging via Matrix Factorization Rei Kawakami 1 John Wright 2 Yu-Wing Tai 3 Yasuyuki Matsushita 2 Moshe Ben-Ezra 2 Katsushi Ikeuchi 3 1 University of Tokyo, 2 Microsoft Research Asia (MSRA), 3 Korea Advanced Institute of Science and Technology (KAIST) CVPR 11

Giga-pixel Camera M. Ezra et al. Giga-pixel Microsoft research Large-format lensCCD

Spectrum

RGB vs. Spectrum

Approach Low-res hyperspectral high-res RGB High-res hyperspectral image Combine

Problem formulation W (Image width) H (Image height) S Goal: Given:

Representation: Basis function W (Image width) H (Image height) S = … … 0 = + x 0x 1.0x 0 ++

Two-step approach 1.Estimate basis functions from hyperspectral image 2.For each pixel in high-res RGB image, estimate coefficients for the basis functions

1: Limited number of materials Sparse vector For all pixel (i,j) Sparse matrix W (Image width) H (Image height) S = … … 0.6

2: Sparsity in high-res image W H S Sparse coefficients Reconstruction

Simulation experiments

460 nm550 nm620 nm 460 nm550 nm620 nm

430 nm490 nm550 nm610 nm670 nm

Error images of Global PCA with back- projection Error images of local window with back-projection Error images of RGB clustering with back-projection

Estimated 430 nm

Ground truth

RGB image

Error image

HS camera Filter CMOSLens Aperture Focus Translational stage

Real data experiment Input RGBInput (550nm)Input (620nm)Estimated (550nm)Estimated (620nm)

Summary Method to reconstruct high-resolution hyperspectral image from ▫Low-res hyperspectral camera ▫High-res RGB camera Spatial sparsity of hyperspectral input ▫Search for a factorization of the input into  basis functions  set of maximally sparse coefficients