1. MIT Media Lab Camera Culture Image Destabilization: Programmable Defocus using Lens and Sensor Motion Ankit Mohan, Douglas Lanman, Shinsaku Hiura, Ramesh Raskar MIT Media Lab MIT Media Lab Camera Culture

2. Lots of glass; Heavy; Bulky; Expensive Defocus Blur

3. MIT Media Lab Camera Culture Image Destabilization Lens Sensor Camera Static Scene

4. MIT Media Lab Camera Culture Image Destabilization Static Scene Lens Motion Sensor Motion Camera

5. MIT Media Lab Camera Culture Related Work [Bae and Durand 2007] [Hasinoff and Kutulakos 2007] f8 f/4 f/2 f/1 extrapolated aperture [Hiura et al. 2009] [Vaish et al. 2004]

6. MIT Media Lab Camera Culture Laminography X-Ray Source X-Ray Sensor Plane of focus Related technique: Time Delay and Integration (TDI) Motion direction

7. MIT Media Lab Camera Culture A B A B Lens based Focusing LensSensor

8. MIT Media Lab Camera Culture A B A B Lens based Focusing LensSensor

9. MIT Media Lab Camera Culture A B B A Lens Smaller aperture Smaller defocus blur Sensor

10. MIT Media Lab Camera Culture A B A B Pinhole Pinhole: All In-Focus Sensor

11. MIT Media Lab Camera Culture A B A B Pinhole Shifting Pinhole Sensor vpvp

12. MIT Media Lab Camera Culture A B Pinhole A B Shifting Pinhole Sensor vpvp

13. MIT Media Lab Camera Culture A B Pinhole A B Shifting Pinhole Sensor vpvp

14. MIT Media Lab Camera Culture A B Pinhole A B Shifting Pinhole Sensor vpvp

15. MIT Media Lab Camera Culture B PinholeSensor A B Shifting Pinhole A dada dbdb dsds tptp vpvp

16. MIT Media Lab Camera Culture A B A B Pinhole Shifting Pinhole and Sensor vpvp Sensor vsvs dada dbdb dsds Focus Here

17. MIT Media Lab Camera Culture A B Pinhole A B vpvp Sensor vsvs dada dbdb dsds Focus Here Shifting Pinhole and Sensor

18. MIT Media Lab Camera Culture A B Pinhole A B vpvp Sensor vsvs dada dbdb dsds Focus Here Shifting Pinhole and Sensor

19. MIT Media Lab Camera Culture A B Pinhole A B vpvp Sensor vsvs dada dbdb dsds Focus Here Shifting Pinhole and Sensor

20. MIT Media Lab Camera Culture A Lens in Time! Analogous to shift and sum based Light field re-focusing. Lens Equation: Virtual Focal Length: Virtual F-Number:

21. MIT Media Lab Camera Culture Our Prototype

22. MIT Media Lab Camera Culture Adjusting the Focus Plane all-in-focus pinhole image

23. MIT Media Lab Camera Culture Adjusting the Focus Plane focused in the front using destabilization (10 second exposure)

24. MIT Media Lab Camera Culture Adjusting the Focus Plane focused in the middle using destabilization (5 second exposure)

25. MIT Media Lab Camera Culture Adjusting the Focus Plane focused in the back using destabilization (10 second exposure)

26. MIT Media Lab Camera Culture Adjusting the Virtual Aperture focused in the middle using destabilization (5mm pinhole translation)

27. MIT Media Lab Camera Culture Adjusting the Virtual Aperture focused in the middle using destabilization (30mm pinhole translation)

28. MIT Media Lab Camera Culture Shifting Lens and Sensor Defocus Physical vs. synthetic focusSimilar physical and synthetic focus aperture sensor real focus virtual focus Defocus Exaggeration

29. MIT Media Lab Camera Culture Defocus Exaggeration static lens with an f/2.8 aperture

30. MIT Media Lab Camera Culture Defocus Exaggeration destabilization simulates a reduced f-number

31. MIT Media Lab Camera Culture Defocus Invariance aperture sensor real focus virtual focus Differing physical and synthetic focus

32. MIT Media Lab Camera Culture * = real PSF virtual PSF overall PSF Defocus Invariance Related work –[Nagahara et al. 2008] –[Cathey and Dowski 1995] –[Levin et al. 2008] PSF not depth invariant –only size is depth invariant Gaussian special case –depth invariant PSF –inversion is ill-conditioned * = real PSF virtual PSF overall PSF [Nagahara et al. 2008]

33. MIT Media Lab Camera Culture Defocus Invariance depth-invariant blur size (horizontal slit + destabilization)

34. MIT Media Lab Camera Culture Defocus Invariance Richardson-Lucy deconvolution result

35. MIT Media Lab Camera Culture Tilted Sensor C D C D focus plane aperture planesensor plane Scheimpflug intersection dCdC dDdD dCdC dDdD

36. MIT Media Lab Camera Culture Tilted Sensor C D vs?vs? focus plane aperture planesensor plane vpvp dCdC dDdD

37. MIT Media Lab Camera Culture Tilted Sensor C D vpvp C D vsvs focus plane aperture planesensor plane dCdC dDdD dCdC dDdD

38. MIT Media Lab Camera Culture Tuning the PSF aperture sensor real focus pinhole image (static f/22 aperture)

39. MIT Media Lab Camera Culture Tuning the PSF real focus aperture sensor large aperture image (static f/2.8 aperture)

40. MIT Media Lab Camera Culture Tuning the PSF virtual focus aperture sensor destabilized image using a pinhole (translated f/22 aperture)

41. MIT Media Lab Camera Culture Tuning the PSF aperture sensor real focus virtual focus destabilized image using a large aperture (translated f/2.8 aperture)

42. MIT Media Lab Camera Culture Tuning the PSF real focus virtual focus aperture sensor simulated aspheric lens using a vertical slit aperture and destabilization

43. MIT Media Lab Camera Culture Extension to 2D Displacements elliptical linearcircular figure 8hypocycloidaltrispiral

44. MIT Media Lab Camera Culture Large apertures with tiny lenses? Benefits No time or light inefficiency wrt cheap cameras Exploits unused area around the lens Compact design With near-pinhole apertures (mobile phones) many possibilities Limitations Coordinated mechanical movement required Diffraction (due to small aperture) cannot be eliminated [Zhang and Levoy, tomorrow] [Our group: augmented LF for wave analysis] Scene motion during exposure

45. MIT Media Lab Camera Culture Acknowledgements Grace Woo MIT CSAIL Quinn Smithwick MIT Media Lab Gabriel Taubin Brown University Jaewon Kim MIT Media Lab MIT Media Lab: Camera Culture

46. MIT Media Lab Camera Culture SLRs with tiny lenses? Analysis of space of relative lens/sensor displacement Destabilization as virtual focusing mechanism Shallower depth of field than physical aperture Depth-independent defocus blur size