Multi-Aperture Photography Paul Green – MIT CSAIL Wenyang Sun – MERL Wojciech Matusik – MERL Frédo Durand – MIT CSAIL

Motivation Portrait Landscape Small Aperture Large Aperture Depth of Field Control Shallow Depth of Field Large Depth of Field

plane of focus Depth and Defocus Blur sensor lens defocus blur depends on distance from plane of focus subject rays from point in focus converge to single pixel circle of confusion

Defocus Blur & Aperture lens plane of focus defocus blur depends on aperture size aperture sensor subject circle of confusion

Goals Aperture size is a critical parameter for photographers ■post-exposure depth of field control ■extrapolate shallow depth of field beyond physical aperture

Outline Multi-Aperture Camera ■ New camera design ■ Capture multiple aperture settings simultaneously Applications ■ Depth of field control ■ Depth of field extrapolation ■ (Limited) refocusing

Related Work Computational Cameras ■ Plenoptic Cameras ■ Adelson and Wang ‘92 ■ Ng et al ‘05 ■ Georgiev et al ‘06 ■ Split-Aperture Camera ■ Aggarwal and Ahuja ‘04 ■ Optical Splitting Trees ■ McGuire et al ‘07 ■ Coded Aperture ■ Levin et al ’07 ■ Veeraraghavan et al ’07 ■ Wavefront Coding ■ Dowski and Cathey ‘95 Depth from Defocus ■ Pentland ‘87 Georgiev et al‘06 Aggarwal and Ahuja ‘04 McGuire et al ‘07 Adelson and Wang ‘92 Levin et al ’07 Veeraraghavan et al ’07

Plenoptic Cameras Capture 4D LightField ■ 2D Spatial (x,y) ■ 2D Angular (u,v Aperture) Trade resolution for flexibility after capture ■ Refocusing ■ Depth of field control ■ Improved Noise Characteristics Lens Aperture u v Sensor (x,y) Lenslet Array Subject Lens (u,v)

1D vs 2D Aperture Sampling u v Aperture 2D Grid Sampling

4 Samples u v Aperture 2D Grid Sampling 1D vs. 2D Aperture Sampling Aperture 1D “Ring” Sampling 45 Samples

Optical Splitting Trees General framework for sampling imaging parameters ■ Beamsplitters ■ Multiple cameras Large Aperture Camera Small Aperture Camera McGuire et al ‘07 Beamsplitter Incoming light

Goals ■post-exposure depth of field control ■extrapolate shallow depth of field ■(limited) refocusing ■1d sampling ■no beamsplitters ■single sensor ■removable

Outline Multi-Aperture Camera ■ New camera design ■ Capture multiple aperture settings simultaneously Applications ■ Depth of field control ■ Depth of field extrapolation ■ Refocusing

Optical Design Principles Aperture 3D sampling ■ 2D spatial ■ 1D aperture size ■ 1 image for each “ring” Sensor

Goal: Split aperture into 4 separate optical paths ■ concentric tilted mirrors ■ at aperture plane Aperture Splitting Tilted Mirrors

Aperture Splitting Incoming light Sensor Mirrors Focusing lenses Tilted Mirrors

Aperture Splitting Photographic Lens Aperture Plane Relay system Aperture splitting optics New Aperture Plane X Ideally at aperture plane, but not physically possible! Solution: Relay Optics to create virtual aperture plane

Optical Prototype Mirror Close-up main lens relay optics mirrors tilted mirrors lenses SLR Camera

Sample Data Raw data from our camera

Ideally would be rings Gaps are from occlusion Point Spread Function Occlusion combined inner ring 1 ring 2 outer

Outline Multi-Aperture Camera ■ New camera design ■ Capture multiple aperture settings simultaneously Applications ■ Depth of field control ■ Depth of field extrapolation ■ Refocusing

DOF Navigation

Approximate defocus blur as convolution DOF Extrapolation? ? - Circular aperture blurring kernel Depends on depth and aperture size What is at each pixel in ?

Blur size Aperture Diameter Largest physical aperture DOF Extrapolation Roadmap capture estimate blur fit model extrapolate blur I I E E I 1 I 2 I 0 I 3

Blur size Aperture Diameter D I 1 I 2 I E I 0 σ I 3 Largest physical aperture Defocus Gradient Defocus blur G is slope of this line Defocus Gradient Map Defocus Gradient focal length aperture diameter sensor distance object distance Blur proportional to aperture diameter

Optimization solve for discrete defocus gradient values G at each pixel Data term Graph Cuts with spatial regularization term Defocus Gradient Map Smallest Aperture Image

Depth of Field Extrapolation

Synthetic Refocusing Modify gradient labels and re-synthesize image gradient map “refocused” map extrapolated f/1.8 “refocused” synthetic f/1.8

Synthetic Refocusing Video

Depth Guided Deconvolution Deconvolve (deblur) with kernel given by defocus gradient map Before After depth-guided deconvolution Defocus gradient map Smallest aperture image

Discussion ■Occlusion ■Could help depth discrimination (coded aperture) ■Difficult alignment process ■Mostly because prototype ■Refocusing limited by Depth of Field ■helped by depth-guided deconvolution ■Texture required for accurate defocus gradient map ■Not critical for depth of field and refocus

Summary ■ Multi-aperture camera ■ 1D sampling of aperture ■ Removable ■ Post-Exposure depth of field control ■ Depth of field extrapolation ■ Limited refocusing ■ Depth-guided deconvolution

Thanks People ■ John Barnwell ■ Jonathan Westhues ■ SeBaek Oh ■ Daniel Vlasic ■ Eugene Hsu ■ Tom Mertens ■ Britton Bradley ■ Jane Malcolm ■ MIT Graphics Group Funding ■ NSF CAREER award ■ Ford Foundation predoctoral Fellowship ■ Microsoft Research New Faculty Fellowship ■ Sloan Fellowship