Introduction to Computational Photography

Computational Photography Digital Camera What is Computational Photography? Second breakthrough by IT First : electronic image sensor (digital camera) Digital representation of “image formed by lens” Second : Re-definition of whole camera (optics, usage) Image is reconstructed by computation Image processing Image Image sensor Optics Whole part of camera is affected by computational photography Film camera Digital Camera

Light field (Light space) What is camera? Camera is a machine to record the distribution of the light in a scene How to represent the distribution of the light in the scene? 3-D coordinate of the point where the light passing through : X, Y, Z Direction of the light : θ，Φ Wavelength of the light (color of the light) : λ Time : t The 7 parameters function P which represent the distribution of the light is called “Plenoptic function” Light source Object

Light field (light ray)Optics (lens)Image sensor (pixel) Integration of camera Camera integrates the light for all 7 parameters Position（range of X, Y, Z : aperture size should not be zero) Direction（range of θ，Φ : pixel size is not zero) Wavelength（ range of λ : No single wavelength filter) Exposure time ( range of t : shutter speed should not be too fast) Multiple samples - θ,Φ:number of pixel, λ：RGB，t : burst shot So, what is multiple sampling for X, Y, Z?

Camera array Measuring the distribution of the light at multiple position ProFUSION25 (ViewPlus, Inc.) The Stanford Multi-Camera Array (Marc University)

Use of camera array Free-viewpoint image Defocus generation by synthetic aperture 3-D video (Matsuyama lab, Kyoto Univ.)Synthetic aperture

Defocus control by Uncalibrated Synthetic Aperture Natsumi Kusumoto, Shinsaku Hiura and Kosuke Sato, Uncalibrated Synthetic Aperture for Defocus Control, CVPR2009 (Jun. 2009)

Reviewing “integration” Some part of information is lost by integration Sine wave which period is just as same as the integration duration is lost Blur of object within an exposure time Defocus by misfocus × = 0

Coded Exposure Coded exposure : exposure is coded in time axis Flutter Shutter Camera

This Traditional Coded Exposure Image of Static Object Deblurred Image Slide by R. Raskar

Coded Exposure Temporal 1-D broadband code: Motion Deblurring Coded Aperture Spatial 2-D broadband mask: Focus Deblurring Slide by R. Raskar

Captured Blurred Photo Slide by R. Raskar

Refocused on Person Slide by R. Raskar

Coded Aperture Depth estimation by single image (manual operation is necessary)

Coded Aperture

Coded Aperture

Multi-focus camera with Coded Aperture Stabilizing the depth estimation and deblur by coded aperture Simultaneous capture of 3 images with different focused distance Hiura et al, CVPR(1998), SSII(1999)

Multi-Focus Range Sensor using Coded Aperture

Invariant integration Defocus : changed according to the distance Blur : changed according to the speed of the object Reconstruction is not easy because the estimation of the speed or distance is necessary  Is it possible to make defocus or blur invariant to the distance or speed?

Invariant integration Defocus Special optics : Wavefront Coding Motion of the image sensor while exposure Blur Reciprocal motion of the camera CDM Optics, Inc.

Motion of the image sensor for invariant defocus H. Nagahara, S. Kuthirummal, C. Zhou, and S.K. Nayar, Flexible Depth of Field Photography, ECCV2008 H. Nagahara, S. Kuthirummal, C. Zhou, and S.K. Nayar, Flexible Depth of Field Photography, ECCV2008

H. Nagahara, S. Kuthirummal, C. Zhou, and S.K. Nayar, Flexible Depth of Field Photography, ECCV2008 Motion of the image sensor for invariant defocus

Deblur by reciprocal motion of the camera A. Levin, P. Sand, T. S. Cho, F. Durand, W. T. Freeman. Motion- Invariant Photography. SIGGRAPH2008. A. Levin, P. Sand, T. S. Cho, F. Durand, W. T. Freeman. Motion- Invariant Photography. SIGGRAPH2008. Input imageDeblurred image

A. Levin, P. Sand, T. S. Cho, F. Durand, W. T. Freeman. Motion- Invariant Photography. SIGGRAPH2008. EquipmentConceptual figure for Light sources with different speed Deblur by reciprocal motion of the camera

More.. Resources on www Wikipedia : computational photographycomputational photography Conferences International Conference on Computational Photography SIGGRAPH, CVPR,.. Session about computational photography