1. Multi-Aperture Photography Paul Green – MIT CSAIL Wenyang Sun – MERL Wojciech Matusik – MERL Frédo Durand – MIT CSAIL

2. Motivation http://photographertips.net Portrait Landscape Small Aperture Large Aperture Depth of Field Control Shallow Depth of Field Large Depth of Field

3. plane of focus Depth and Defocus Blur sensor lens defocus blur depends on distance from plane of focus subject rays from point in focus converge to single pixel circle of confusion

4. Defocus Blur & Aperture lens plane of focus defocus blur depends on aperture size aperture http://photographertips.net sensor subject circle of confusion

5. Goals Aperture size is a critical parameter for photographers ■post-exposure depth of field control ■extrapolate shallow depth of field beyond physical aperture

6. Outline Multi-Aperture Camera ■ New camera design ■ Capture multiple aperture settings simultaneously Applications ■ Depth of field control ■ Depth of field extrapolation ■ (Limited) refocusing

7. Related Work Computational Cameras ■ Plenoptic Cameras ■ Adelson and Wang ‘92 ■ Ng et al ‘05 ■ Georgiev et al ‘06 ■ Split-Aperture Camera ■ Aggarwal and Ahuja ‘04 ■ Optical Splitting Trees ■ McGuire et al ‘07 ■ Coded Aperture ■ Levin et al ’07 ■ Veeraraghavan et al ’07 ■ Wavefront Coding ■ Dowski and Cathey ‘95 Depth from Defocus ■ Pentland ‘87 Georgiev et al‘06 Aggarwal and Ahuja ‘04 McGuire et al ‘07 Adelson and Wang ‘92 Levin et al ’07 Veeraraghavan et al ’07

8. Plenoptic Cameras Capture 4D LightField ■ 2D Spatial (x,y) ■ 2D Angular (u,v Aperture) Trade resolution for flexibility after capture ■ Refocusing ■ Depth of field control ■ Improved Noise Characteristics Lens Aperture u v Sensor (x,y) Lenslet Array Subject Lens (u,v)

9. 1D vs 2D Aperture Sampling u v Aperture 2D Grid Sampling http://photographertips.net

10. 4 Samples u v Aperture 2D Grid Sampling 1D vs. 2D Aperture Sampling Aperture 1D “Ring” Sampling 45 Samples http://photographertips.net

11. Optical Splitting Trees General framework for sampling imaging parameters ■ Beamsplitters ■ Multiple cameras Large Aperture Camera Small Aperture Camera McGuire et al ‘07 Beamsplitter Incoming light

12. Goals ■post-exposure depth of field control ■extrapolate shallow depth of field ■(limited) refocusing ■1d sampling ■no beamsplitters ■single sensor ■removable

13. Outline Multi-Aperture Camera ■ New camera design ■ Capture multiple aperture settings simultaneously Applications ■ Depth of field control ■ Depth of field extrapolation ■ Refocusing

14. Optical Design Principles Aperture 3D sampling ■ 2D spatial ■ 1D aperture size ■ 1 image for each “ring” Sensor http://photographertips.net

15. Goal: Split aperture into 4 separate optical paths ■ concentric tilted mirrors ■ at aperture plane Aperture Splitting Tilted Mirrors

16. Aperture Splitting Incoming light Sensor Mirrors Focusing lenses Tilted Mirrors

17. Aperture Splitting Photographic Lens Aperture Plane Relay system Aperture splitting optics New Aperture Plane X Ideally at aperture plane, but not physically possible! Solution: Relay Optics to create virtual aperture plane

18. Optical Prototype Mirror Close-up main lens relay optics mirrors tilted mirrors lenses SLR Camera

19. Sample Data Raw data from our camera

20. Ideally would be rings Gaps are from occlusion Point Spread Function Occlusion combined inner ring 1 ring 2 outer

21. Outline Multi-Aperture Camera ■ New camera design ■ Capture multiple aperture settings simultaneously Applications ■ Depth of field control ■ Depth of field extrapolation ■ Refocusing

22. DOF Navigation

23. Approximate defocus blur as convolution DOF Extrapolation? ? - Circular aperture blurring kernel Depends on depth and aperture size What is at each pixel in ?

24. Blur size Aperture Diameter Largest physical aperture DOF Extrapolation Roadmap capture estimate blur fit model extrapolate blur I I E E I 1 I 2 I 0 I 3

25. Blur size Aperture Diameter D I 1 I 2 I E I 0 σ I 3 Largest physical aperture Defocus Gradient Defocus blur G is slope of this line Defocus Gradient Map Defocus Gradient focal length aperture diameter sensor distance object distance Blur proportional to aperture diameter

26. Optimization solve for discrete defocus gradient values G at each pixel Data term Graph Cuts with spatial regularization term Defocus Gradient Map Smallest Aperture Image

27. Depth of Field Extrapolation

28. Synthetic Refocusing Modify gradient labels and re-synthesize image gradient map “refocused” map extrapolated f/1.8 “refocused” synthetic f/1.8

29. Synthetic Refocusing Video

30. Depth Guided Deconvolution Deconvolve (deblur) with kernel given by defocus gradient map Before After depth-guided deconvolution Defocus gradient map Smallest aperture image

31. Discussion ■Occlusion ■Could help depth discrimination (coded aperture) ■Difficult alignment process ■Mostly because prototype ■Refocusing limited by Depth of Field ■helped by depth-guided deconvolution ■Texture required for accurate defocus gradient map ■Not critical for depth of field and refocus

32. Summary ■ Multi-aperture camera ■ 1D sampling of aperture ■ Removable ■ Post-Exposure depth of field control ■ Depth of field extrapolation ■ Limited refocusing ■ Depth-guided deconvolution

33. Thanks People ■ John Barnwell ■ Jonathan Westhues ■ SeBaek Oh ■ Daniel Vlasic ■ Eugene Hsu ■ Tom Mertens ■ Britton Bradley ■ Jane Malcolm ■ MIT Graphics Group Funding ■ NSF CAREER award 0447561 ■ Ford Foundation predoctoral Fellowship ■ Microsoft Research New Faculty Fellowship ■ Sloan Fellowship