Depth from Diffusion Supported by ONR Changyin ZhouShree NayarOliver Cossairt Columbia University

Optical Diffuser

Micrograph of a Holographic Diffuser (RPC Photonics) ~ 10 micron [Gray, 1978] [Chang et al., 2006] [Garcia-Guerrero et al. 2007]

Diffusers as Accessories Diffuser to preview the image (B&H) Diffusers to soften the image Diffusers for illumination (B&H)

Camera Diffuser Object Camera Diffuser Object The amount of diffusion varies with depth. Diffusion Encodes Depth

Geometry of Diffusion: A Pinhole Camera Sensor Pinhole Object P Q Miss

Object P Geometry of Diffusion: A Pinhole Camera Sensor θ Pinhole Diffuser

Geometry of Diffusion: A Pinhole Camera Pinhole θ θ Sensor Diffuser Object P A B

Geometry of Diffusion: A Pinhole Camera O V U Z θ θ Pinhole Diffusion Law: Sensor A B Object 2r P Object Diffuser

Geometry of Diffusion: A Pinhole Camera O V U Z Pinhole Sensor A B Object 2r P Diffusion Size and Depth: Object Diffuser θ θ

Geometry of Diffusion: A Pinhole Camera O V U Pinhole Sensor Object 2r Diffuser as a proxy object Diffusion Size and Depth: P Z Diffuser

Diffusion as Convolution: A Pinhole Camera Latent clear image Captured Image Diffusion Size Diffusion PSF Assume field angle and depth are constant for small image patches, we have:

Geometry of Diffusion: A Lens Camera

O V U Pinhole Sensor Object 2r Diffuser as a proxy object P Z Diffuser

Geometry of Diffusion: A Lens Camera V U Sensor Object 2r Diffuser as a proxy object P Z Diffuser Lens The captured image can be further blurred due to defocus.

Diffusion as Convolution: A Lens Camera The Final PSF Defocus PSF Diffusion PSF For a lens camera with a diffuser, we have: is the diffusion PSF if a pinhole were used. is the defocus PSF if the diffuser were removed.

Depth from Diffusion (DFDiff) Algorithm Same form as in DFD 2. Estimate Blur Size r 3. Compute Depth Z 1. Capture Two Images With a diffuser Without a diffuser

Depth from Diffusion vs. Depth from Defocus [Pentland, 1987] [Subbarao, 1988] [Watanabe & Nayar, 1996] [Chaudhuri & Rajagopalan, 1999] [Favaro & Soatto, 2005] [Schechner & Kiryati, 2000] Depth from Defocus Aperture pattern P Lens Focal Plane Sensor Z r Depth from Diffusion Diffusion pattern Sensor Pinhole θ θ Diffuser Z P r

Depth from Diffusion vs. Depth from Defocus Depth from Diffusion P Any lens is fine! Suppose 22.5x15mm Sensor, 10 um pixel, 100 mm EFL Object distance = 1000 mm Object A Diffuser of 21.8 o Depth precision is about 0.1 mm. Field of View

Depth from Diffusion vs. Depth from Defocus Depth from Defocus P Lens Aperture diameter ? Suppose 22.5x15mm Sensor, 10 um pixel, 100 mm EFL Depth precision is about 0.1 mm. Object distance = 1000 mm Object Field of View

Depth from Diffusion vs. Depth from Defocus Depth from Defocus Object distance = 1000 mm Suppose 22.5x15mm Sensor, 10 um pixel, 100 mm EFL Depth precision is about 0.1 mm. Aperture diameter 800 mm Object P

Depth from Diffusion vs. Depth from Defocus Depth from Diffusion Suppose 22.5x15mm Sensor, 10 um pixel, 100 mm EFL P Depth precision is about 1.0 mm. Object distance = 5000 mm Object Any lens is fine! A Diffuser of 11.2 o

Depth from Diffusion vs. Depth from Defocus Depth from Defocus P Lens Aperture diameter ? Suppose 22.5x15mm Sensor, 10 um pixel, 100 mm EFL Depth precision is about 1.0 mm. Object distance = 5000 mm Object

Depth from Diffusion vs. Depth from Defocus Depth from Defocus Suppose 22.5x15mm Sensor, 10 um pixel, 100 mm EFL Depth precision is about 1.0 mm. Aperture diameter 2000 mm P Object distance = 5000 mm Object

PSF Measurement: A Pinhole Camera F/22, Field Angle = 0 o Z = 2 mm Z = 5 mm CapturedModeled - Canon EOS T1i; EF 50mm F/1.8 Lens; - Luminit Holographic Diffuser (10 o Gaussian ); - Diffuser distance: U = 1m

Z = 2 mm Z = 5 mm PSF Measurement: A Pinhole Camera CapturedModeled F/22, Field Angle = 10 o - Canon EOS T1i; EF 50mm F/1.8 Lens; - Luminit Holographic Diffuser (10 o Gaussian ); - Diffuser distance: U = 1m

PSF Measurement: A Lens Camera F/1.8, Field Angle = 10 o CapturedModeled - Canon EOS T1i; EF 50mm F/1.8 Lens; - Luminit Holographic Diffuser (10 o Gaussian ); - Diffuser distance: U = 1m

Experiments Five playing cards, 0.29mm thick each Canon 20D + 50mm Lens Luminit Diffuser (20 o )

Experiments Captured WITHOUT a DiffuserCaptured WITH a Diffuser

Experiments Computed Depth Map (~ 0.1 mm precision) (mm) Five playing cards, 0.29mm thick each

Experiments A small sculpture of about 4mm thickness Canon G5 Compact Camera Luminit Diffuser (5 o )

Experiments Captured WITHOUT a DiffuserCaptured WITH a Diffuser

Experiments Computed Depth Map A 3D View of Depth Map A small sculpture of about 4mm thickness

Experiments Canon 20D; Gaussian Diffuser (10 o ) 450 mm 650 mm

Experiments Stitched Depth Map (precision) (mm)

Summary Formulated the image formation with optical diffusers Formulated the image formation with optical diffusers Proposed Depth from Diffusion Proposed Depth from Diffusion - Require a diffuser on the object side + High-precision depth estimation + Distant objects + Less sensitive to lens aberrations Demonstrated high-precision depth estimation Demonstrated high-precision depth estimation Camera Diffuser Object

Depth from Diffusion Supported by ONR Changyin ZhouShree NayarOliver Cossairt Columbia University