1. Generalized Mosaics Yoav Y. Schechner, Shree Nayar Department of Computer Science Columbia University

2. Mosaics Processing

3. Redundant Measurements Generalized Mosaicing: Yoav Schechner and Shree Nayar

4. Scanning with Less Redundancy Mantis Shrimp TakataC Different rows = Different optical characteristics

5. Generalized Mosaicing Field of View Brightness Dynamic Range Spectrum Depth of Field Polarization camera Spatially varying filter Schechner, Nayar, Generalized mosaics

6. Mosaic + High Dynamic Range 88 - 18,794

7. Fusion of Measurements Maximum-Likelihood Solution Schechner, Nayar, Generalized mosaics

8. 8 bit camera Dynamic range as 16 bits Generalized Mosaicing: Yoav Schechner and Shree Nayar 8 bits (almost) (pixels) -2 -4 -6 -8 x log M 2 Log of the mask = 110 -6 110 -2

9. M x average row Mask Self-Calibration Unknown filter (vignetting) consistency constraints Consistency, Smoothness A Schechner, Nayar, Generalized mosaics

10. Image Registration Schechner, Nayar, Generalized mosaics

11. Bias towards “no motion” Schechner, Nayar, Generalized mosaics Image Registration

12. Uncertainty Schechner, Nayar, Generalized mosaics Image Registration

13. Registration: Standard “Coarse to Fine” 5 pixels 10 pix 9 pix 18 pix 19 pix 4 pixels 2 pixels motion of However, we need: multiscale UNCERTAINTIES Schechner, Nayar, Generalized mosaics

14. Maximum-Likelihood Pyramid At each level L, for each pixel estimate 2 Schechner, Nayar, Generalized mosaics

15. Max-Likelihood Pyramid Gaussian PyramidML estimation

16. log I Intensity range Spatial range x Generalized Mosaic x log I Intensity range Spatial range frame Mosaic Schechner, Nayar, Generalized mosaics

17. Generalized Mosaicing: Yoav Schechner and Shree Nayar spectral y x Variable Spectral Filter

18. Multi Spectral Multi Spectral Mosaic 500700400600 500700400600 500700400600 Generalized Mosaicing: Yoav Schechner and Shree Nayar

19. Rendering : Any Illumination sunset fluorescent

20. Halogen y x Illumination at a Glance Fluorescent y x  Illuminant spectrum Extra information

21. Spatially Varying Polarizer Polarization Mosaicing: Yoav Schechner and Shree Nayar

22. 0 0.2 0.4 0.6 0.8 1 -30 60 90 30 0 -60 -90 o o o o o o polarizance polarizing angle o Polarization Mosaicing: Yoav Schechner and Shree Nayar transmittance Spatially Varying Polarizer

23. Raw images Polarization Mosaicing: Yoav Schechner and Shree Nayar

24. Polarization Mosaic Schechner Shamir Kiryati JOSA-A 2000 reflected structure transmitted painting

25. = y x Depth Insert a prism behind the lens Spatially Varying Focus Generalized Mosaicing: Yoav Schechner and Shree Nayar

27. All Focused All Focused Mosaic Generalized Mosaicing: Yoav Schechner and Shree Nayar … and a Depth Map

28. What else? Generalized Mosaicing: Yoav Schechner and Shree Nayar Dynamic Range & Spectrum & Polarization etc. Multi-dimensional Mosaics - Simultaneously

29.  Signal undergoing LPF  “Band limited” signal  Minimizing aliasing  Nyquist sampling rate 11111111 11111111? 10111001?? Sampling Criteria Generalized Mosaicing: Yoav Schechner and Shree Nayar Number of samples per scene point resolution  length L bandwidth B aperture D 1M 1011100101 1011100??? M # samples 1 1/2 2 1/4 3 9 101110????

30. MosaicGeneralized Mosaic Spatially varying filter Wide field of view Spectrum I Depth + Focus Dynamic range 88 – 18,794 Polarization