MERL, MIT Media Lab Reinterpretable Imager Agrawal, Veeraraghavan & Raskar Amit Agrawal, Ashok Veeraraghavan and Ramesh Raskar Mitsubishi Electric Research Labs (MERL) MIT Media Lab Cambridge, MA, USA Reinterpretable Imager: Towards Variable Post-Capture Space, Angle & Time Resolution in Photography

Captured Photo

Video from Single-Shot (Temporal Frames)

Captured Photo

Rotating Doll in Focus

Captured 2D Photo

Captured 2D Photo In-Focus High Resolution 2D Image Static Scene Parts

Captured 2D Photo In-FocusOut of Focus High Resolution 2D Image 4D Light Field Static Scene Parts

Captured 2D Photo In-FocusOut of Focus In-Focus High Resolution 2D Image 4D Light Field Video Static Scene PartsDynamic Scene Parts

Captured 2D Photo In-FocusOut of FocusIn-Focus High Resolution 2D Image 4D Light Field Video Static Scene PartsDynamic Scene Parts 1D Parallax + Motion Out of Focus

MERL, MIT Media Lab Reinterpretable Imager Agrawal, Veeraraghavan & Raskar Key Idea Resolution tradeoff for Conventional Imaging –Fixed before capture video camera, lightfield camera –Scene independent Resolution tradeoff for Reinterpretable Imager –Variable in post-capture –Scene dependent –Different for different parts of the scene/captured photo

MERL, MIT Media Lab Reinterpretable Imager Agrawal, Veeraraghavan & Raskar Reinterpretable Imager Capture Time-Varying Light Field Spatial dimensions = 2 Angular dimensions = 2 Temporal dimensions = 1 We capture 4D subsets in single-shot Single Camera Design can act as –Still camera, Video camera, lightfield camera

Optical Design

Dynamic Aperture Mask Sensor Static Mask Sensor Static Aperture Mask Sensor Coded ApertureOptical HeterodyningReinterpretable Imager Veeraraghavan et al. SIGGRAPH 2007 This Paper Static Mask Digital Refocusing

Dynamic Aperture Mask Sensor Static Mask Sensor Static Aperture Mask Sensor Coded ApertureOptical HeterodyningReinterpretable Imager Veeraraghavan et al. SIGGRAPH 2007 This Paper Static Mask Digital Refocusing Light Field Capture

Static Mask Sensor Static Aperture Mask Sensor Coded ApertureOptical Heterodyning Veeraraghavan et al. SIGGRAPH 2007 This Paper Dynamic Aperture Mask Sensor Reinterpretable Imager Static Mask Digital Refocusing Light Field Capture Post-Capture Resolution Control

MERL, MIT Media Lab Reinterpretable Imager Agrawal, Veeraraghavan & Raskar Reinterpretable Imager Dynamic Aperture Mask –Pinhole moved across the aperture during exposure time Single shot video, lightfield, high res image –Vertical slit moved across the aperture 1D parallax + motion Near-Sensor Mask –Similar to Veeraraghvan et al. SIGGRAPH 2007

MERL, MIT Media Lab Reinterpretable Imager Agrawal, Veeraraghavan & Raskar Implementation Camera Motor Wheel Aperture Mask on Wheel Shutter Near-Sensor Mask

MERL, MIT Media Lab Reinterpretable Imager Agrawal, Veeraraghavan & Raskar Related Work Hand-held Light Field Camera Single shot Micro-lens Ng et al Prims+lens Georgiev et al Mask at sensor Veeraraghavan et al Light Field camera + Aperture Modulation –Horstmeyer ICCP 09 –Polarization, Spectral Multiplexing techniques –Assorted Pixels –Illumination multiplexing, Schechner et al. ICCV 2003

MERL, MIT Media Lab Reinterpretable Imager Agrawal, Veeraraghavan & Raskar Recovering Full Resolution 2D Image Sensor p Mask No Mask: pixel value = I(p) With Mask: pixel value = β(p)I(p) In-focus static scene Mask

For Static In-Focus Scene Captured 2D Photo Divide by Calibration Photo High Resolution Image

MERL, MIT Media Lab Reinterpretable Imager Agrawal, Veeraraghavan & Raskar Recovering Full Resolution 2D Image For static in-focus scene –Inserting Masks == Spatially Varying Image Attenuation –Compensate using normalization photo β(p) In Focus Out of Focus In Focus Out of Focus Captured PhotoNormalization PhotoFull Resolution Image

MERL, MIT Media Lab Reinterpretable Imager Agrawal, Veeraraghavan & Raskar Recovering Light Fields for Static Scene Capture Light Fields: –Map angular variations to spatial dimension –Angle To Space Mapping For static scenes –Mask close to sensor == captures light field Heterodyning, Veeraraghavan et al. SIGGRAPH 2007 –Mask in aperture == no impact, only lose light

For Static Scenes Captured 2D Photo Compute 4D Light Field Digital Refocusing Sub-Aperture Views == Angular Samples

Captured Photo (Static Scene)

Reconstructed Sub-Aperture Views (3 by 3 Light Field) Angle

MERL, MIT Media Lab Reinterpretable Imager Agrawal, Veeraraghavan & Raskar Recovering Video from Single Shot In-focus dynamic scene –Mask in aperture maps temporal variations to angular variations –Mask close to sensor captures angular variations as a light field Mapping Time to Space Via – Time to Angle + Angle to Space

Static Mask Moving Pinhole K K Scene patch (t 1 ) Capturing In-focus Dynamic Scenes

Static Mask K K Scene patch (t 2 ) Capturing In-focus Dynamic Scenes

Sensor Static Mask d K K spot Scene patch (t 3 )

For Dynamic In-focus Scene Captured 2D Photo Compute 4D Light Field Sub-Aperture Views == Temporal Frames

Captured Photo

Reconstructed Sub-Aperture Views (3 by 3 Light Field) Time

Rotating Doll

Reconstructed Sub-Aperture Views (3 by 3 Light Field)

Time For Rotating Doll

Angle For Static Scene Parts

MERL, MIT Media Lab Reinterpretable Imager Agrawal, Veeraraghavan & Raskar Recovering 1D Parallax + Motion Vertical slit moved across the aperture –Map angular variations to vertical dimension –Maps temporal variations to horizontal dimension Capture dynamic out-of-focus scene –However, only 1D out-of-focus blur (bokeh)

For Dynamic Out-of-focus Scene Captured 2D Photo Compute 4D Light Field Sub-Aperture Views == Temporal Frames (Horizontally) Sub-Aperture Views == Angular Samples (Vertically) Refocus Using Vertical Views

Captured Photo

Reconstructed Sub-Aperture Views (3 by 3 Light Field) Time Angle

Digital Refocusing on Moving Rubik’s Cube

Keeping Playing Card in Focus

Captured Photo

Static Object (in-focus)

Static Objects (Out-of-focus)

Moving Object (in depth)

Rotating Object (in focus)

Reconstructed Sub-Aperture Views (3 by 3 Light Field)

All Static and Dynamic Objects are sharp (No focus blur, no motion blur)

Angle For Static Objects

Time Angle For Moving Toy in Middle

Time For Rotating Toy on Right

Refocused on Static Toy High Resolution Image

Digital Refocusing on Static Objects

Digital Refocusing on Toy Moving in Depth

Video frames of Rotating Toy Video for Rotating Toy in-focus

MERL, MIT Media Lab Reinterpretable Imager Agrawal, Veeraraghavan & Raskar Limitations Light Loss –To get extra information –Both at aperture and sensor –Micro-lens at sensor (Ng et al.) for lightfield capture Temporal Frames –No. of frames = Max angular resolution –Not independent as in a video camera –Large motions cause motion blur –Viewpoint shift –Ghosting artifacts across sub-aperture views Does not capture full 5D information –Video light field camera

MERL, MIT Media Lab Reinterpretable Imager Agrawal, Veeraraghavan & Raskar Future Work LCD’s for modulation –Benefit: Faster modulation –Issues: Contrast, Diffraction Using Computer Vision –No high/mid-level processing at present Adaptive (Active) Sampling

MERL, MIT Media Lab Reinterpretable Imager Agrawal, Veeraraghavan & Raskar Acknowledgements Anonymous Reviewers MERL –Jay Thornton, Kojima Keisuke, Joseph Katz, John Barnwell, Brandon Taylor, Clifton Forlines and Yuichi Taguchi Mitsubishi Electric, Japan –Haruhisa Okuda & Kazuhiko Sumi

MERL, MIT Media Lab Reinterpretable Imager Agrawal, Veeraraghavan & Raskar Google: ‘Reinterpretable Imager’ Captured 2D Photo In-FocusOut of FocusIn-FocusOut of Focus High Resolution 2D Image 4D Light Field Video 1D Parallax + Motion Static Scene PartsDynamic Scene Parts Dynamic Aperture Mask Sensor Reinterpretable Imager Static Mask