1. MERL, MIT Media Lab Reinterpretable Imager Agrawal, Veeraraghavan & Raskar Amit Agrawal, Ashok Veeraraghavan and Ramesh Raskar Mitsubishi Electric Research Labs (MERL) MIT Media Lab Cambridge, MA, USA Reinterpretable Imager: Towards Variable Post-Capture Space, Angle & Time Resolution in Photography

2. Captured Photo

3. Video from Single-Shot (Temporal Frames)

4. Captured Photo

5. Rotating Doll in Focus

15. Captured 2D Photo

16. Captured 2D Photo In-Focus High Resolution 2D Image Static Scene Parts

17. Captured 2D Photo In-FocusOut of Focus High Resolution 2D Image 4D Light Field Static Scene Parts

18. Captured 2D Photo In-FocusOut of Focus In-Focus High Resolution 2D Image 4D Light Field Video Static Scene PartsDynamic Scene Parts

19. Captured 2D Photo In-FocusOut of FocusIn-Focus High Resolution 2D Image 4D Light Field Video Static Scene PartsDynamic Scene Parts 1D Parallax + Motion Out of Focus

20. MERL, MIT Media Lab Reinterpretable Imager Agrawal, Veeraraghavan & Raskar Key Idea Resolution tradeoff for Conventional Imaging –Fixed before capture video camera, lightfield camera –Scene independent Resolution tradeoff for Reinterpretable Imager –Variable in post-capture –Scene dependent –Different for different parts of the scene/captured photo

21. MERL, MIT Media Lab Reinterpretable Imager Agrawal, Veeraraghavan & Raskar Reinterpretable Imager Capture Time-Varying Light Field Spatial dimensions = 2 Angular dimensions = 2 Temporal dimensions = 1 We capture 4D subsets in single-shot Single Camera Design can act as –Still camera, Video camera, lightfield camera

22. Optical Design

23. Dynamic Aperture Mask Sensor Static Mask Sensor Static Aperture Mask Sensor Coded ApertureOptical HeterodyningReinterpretable Imager Veeraraghavan et al. SIGGRAPH 2007 This Paper Static Mask Digital Refocusing

24. Dynamic Aperture Mask Sensor Static Mask Sensor Static Aperture Mask Sensor Coded ApertureOptical HeterodyningReinterpretable Imager Veeraraghavan et al. SIGGRAPH 2007 This Paper Static Mask Digital Refocusing Light Field Capture

25. Static Mask Sensor Static Aperture Mask Sensor Coded ApertureOptical Heterodyning Veeraraghavan et al. SIGGRAPH 2007 This Paper Dynamic Aperture Mask Sensor Reinterpretable Imager Static Mask Digital Refocusing Light Field Capture Post-Capture Resolution Control

26. MERL, MIT Media Lab Reinterpretable Imager Agrawal, Veeraraghavan & Raskar Reinterpretable Imager Dynamic Aperture Mask –Pinhole moved across the aperture during exposure time Single shot video, lightfield, high res image –Vertical slit moved across the aperture 1D parallax + motion Near-Sensor Mask –Similar to Veeraraghvan et al. SIGGRAPH 2007

27. MERL, MIT Media Lab Reinterpretable Imager Agrawal, Veeraraghavan & Raskar Implementation Camera Motor Wheel Aperture Mask on Wheel Shutter Near-Sensor Mask

28. MERL, MIT Media Lab Reinterpretable Imager Agrawal, Veeraraghavan & Raskar Related Work Hand-held Light Field Camera Single shot Micro-lens Ng et al. 2005 Prims+lens Georgiev et al. 2006 Mask at sensor Veeraraghavan et al. 2007 Light Field camera + Aperture Modulation –Horstmeyer ICCP 09 –Polarization, Spectral Multiplexing techniques –Assorted Pixels –Illumination multiplexing, Schechner et al. ICCV 2003

29. MERL, MIT Media Lab Reinterpretable Imager Agrawal, Veeraraghavan & Raskar Recovering Full Resolution 2D Image Sensor p Mask No Mask: pixel value = I(p) With Mask: pixel value = β(p)I(p) In-focus static scene Mask

30. For Static In-Focus Scene Captured 2D Photo Divide by Calibration Photo High Resolution Image

31. MERL, MIT Media Lab Reinterpretable Imager Agrawal, Veeraraghavan & Raskar Recovering Full Resolution 2D Image For static in-focus scene –Inserting Masks == Spatially Varying Image Attenuation –Compensate using normalization photo β(p) In Focus Out of Focus In Focus Out of Focus Captured PhotoNormalization PhotoFull Resolution Image

32. MERL, MIT Media Lab Reinterpretable Imager Agrawal, Veeraraghavan & Raskar Recovering Light Fields for Static Scene Capture Light Fields: –Map angular variations to spatial dimension –Angle To Space Mapping For static scenes –Mask close to sensor == captures light field Heterodyning, Veeraraghavan et al. SIGGRAPH 2007 –Mask in aperture == no impact, only lose light

33. For Static Scenes Captured 2D Photo Compute 4D Light Field Digital Refocusing Sub-Aperture Views == Angular Samples

34. Captured Photo (Static Scene)

35. Reconstructed Sub-Aperture Views (3 by 3 Light Field) Angle

40. MERL, MIT Media Lab Reinterpretable Imager Agrawal, Veeraraghavan & Raskar Recovering Video from Single Shot In-focus dynamic scene –Mask in aperture maps temporal variations to angular variations –Mask close to sensor captures angular variations as a light field Mapping Time to Space Via – Time to Angle + Angle to Space

41. Static Mask Moving Pinhole K K Scene patch (t 1 ) Capturing In-focus Dynamic Scenes

42. Static Mask K K Scene patch (t 2 ) Capturing In-focus Dynamic Scenes

43. Sensor Static Mask d K K spot Scene patch (t 3 )

44. For Dynamic In-focus Scene Captured 2D Photo Compute 4D Light Field Sub-Aperture Views == Temporal Frames

45. Captured Photo

46. Reconstructed Sub-Aperture Views (3 by 3 Light Field) Time

47. Rotating Doll

48. Reconstructed Sub-Aperture Views (3 by 3 Light Field)

49. Time For Rotating Doll

50. Angle For Static Scene Parts

51. MERL, MIT Media Lab Reinterpretable Imager Agrawal, Veeraraghavan & Raskar Recovering 1D Parallax + Motion Vertical slit moved across the aperture –Map angular variations to vertical dimension –Maps temporal variations to horizontal dimension Capture dynamic out-of-focus scene –However, only 1D out-of-focus blur (bokeh)

52. For Dynamic Out-of-focus Scene Captured 2D Photo Compute 4D Light Field Sub-Aperture Views == Temporal Frames (Horizontally) Sub-Aperture Views == Angular Samples (Vertically) Refocus Using Vertical Views

53. Captured Photo

54. Reconstructed Sub-Aperture Views (3 by 3 Light Field) Time Angle

55. Digital Refocusing on Moving Rubik’s Cube

61. Keeping Playing Card in Focus

67. Captured Photo

68. Static Object (in-focus)

69. Static Objects (Out-of-focus)

70. Moving Object (in depth)

71. Rotating Object (in focus)

72. Reconstructed Sub-Aperture Views (3 by 3 Light Field)

73. All Static and Dynamic Objects are sharp (No focus blur, no motion blur)

74. Angle For Static Objects

75. Time Angle For Moving Toy in Middle

76. Time For Rotating Toy on Right

77. Refocused on Static Toy High Resolution Image

78. Digital Refocusing on Static Objects

84. Digital Refocusing on Toy Moving in Depth

90. Video frames of Rotating Toy Video for Rotating Toy in-focus

91. MERL, MIT Media Lab Reinterpretable Imager Agrawal, Veeraraghavan & Raskar Limitations Light Loss –To get extra information –Both at aperture and sensor –Micro-lens at sensor (Ng et al.) for lightfield capture Temporal Frames –No. of frames = Max angular resolution –Not independent as in a video camera –Large motions cause motion blur –Viewpoint shift –Ghosting artifacts across sub-aperture views Does not capture full 5D information –Video light field camera

92. MERL, MIT Media Lab Reinterpretable Imager Agrawal, Veeraraghavan & Raskar Future Work LCD’s for modulation –Benefit: Faster modulation –Issues: Contrast, Diffraction Using Computer Vision –No high/mid-level processing at present Adaptive (Active) Sampling

93. MERL, MIT Media Lab Reinterpretable Imager Agrawal, Veeraraghavan & Raskar Acknowledgements Anonymous Reviewers MERL –Jay Thornton, Kojima Keisuke, Joseph Katz, John Barnwell, Brandon Taylor, Clifton Forlines and Yuichi Taguchi Mitsubishi Electric, Japan –Haruhisa Okuda & Kazuhiko Sumi

94. MERL, MIT Media Lab Reinterpretable Imager Agrawal, Veeraraghavan & Raskar Google: ‘Reinterpretable Imager’ Captured 2D Photo In-FocusOut of FocusIn-FocusOut of Focus High Resolution 2D Image 4D Light Field Video 1D Parallax + Motion Static Scene PartsDynamic Scene Parts Dynamic Aperture Mask Sensor Reinterpretable Imager Static Mask