Motion Deblurring Using Hybrid Imaging Moshe Ben-Ezra and Shree K. Nayar Columbia University IEEE CVPR Conference June 2003, Madison, USA

Image Recording Requires Time Niépce hours exposure Daguerre /2 hour exposure

Motion Blur is Everywhere Object Motion Camera Motion

Stabilized Lenses 1/250 second (< -1 stop) Stabilization drifts with time Rotation only 1/15 second (< -5 stops) Canon Stabilized lens 400mm

Blind Image Deconvolution Accurate Point Spread Function (PSF) Needed.

Motion Point Spread Function (PSF) Motion PSF is a Function of: 1. Motion path 2. Motion speed X Y Energy ~ 1/ speed Spatial spread H

PSF Detector? Camera PSF Detector Can the PSF detector be a small and simple imaging device ?

Electron wells Fundamental Limits of Imaging Detector’s noise level Photon flux Detector Pixel’s Signal Noise

Fundamental Resolution Tradeoff Spatial resolution (pixels) Temporal resolution (fps) K 720x480 Conventional video camera M 2048x1536 Hi-resolution camera 75K 320x240 Low-resolution camera Hybrid imaging system A Hybrid camera enjoys both worlds

Overview of Approach PSF Estimation Low-Res. camera Hi-Res. camera Same time period Deconvolution Motion Analysis x y

Global Motion From Low Resolution Detector TranslationRotation Objective function (Optical flow constraint) Lucas Kanade

Simulations: Motion Accuracy from Low- Res. Images Noise Resolution  = 3  = 9  = 27  = x640 (1:1) x320 (1:4) x160 (1:16) x80 (1:64) Average Motion Error in Pixels

Constraints on Continuous PSF Energy conservation constraint: Path is continuous and twice differentiable Constant flux assumption: Smoothness constraint:

PSF Estimation from Computed Motion x f1f1 f2f2 f3f3 f4f4 f5f5 f6f6 y Frame 2 … Frame 5 y f1f1 f2f2 f3f3 f4f4 f5f5 f6f6 x y h h2h2 h3h3 h4h4 h5h5 Frame 2 … Frame 5 y h h2h2 h3h3 h4h4 h5h5 Frame 2 … Frame 5 x

Deconvolution of High Resolution Image Standard iterative ratio-based algorithm* Guaranties non-negative pixel result * Richardson [72] Lucy [74] ErrorPSFImage estimate

Designs for Hybrid Imaging A rig of two cameras Using a special chip Using a beam splitter

Our Prototype: Rig of Two Cameras Primary detector (2048x1536) Secondary detector (360x240) Resolution ratio of 1 : 36

Example 1 - Blurred Hi-Res Image f = 633mm, Exp. Time 1 Sec (> -9 stops)

PSF Estimation from Motion Low resolution sequence. X (Pixels) Y (Pixels) Estimated PSF f = 633mm, Exp. Time 1 Sec

Deblurred Image f = 633mm, Exp. Time 1 Sec

Example 1 - Comparison Deblurred image Blurred image f = 633mm, Exp. Time 1 Sec Tripod image (Ground Truth)

Example 2 - Blurred Night Image f = 884mm, Exp. Time 4 Sec (> -11 stops)

PSF Estimation from Motion X (Pixels) Y (Pixels) f = 884mm, Exp. Time 4 Sec Low resolution sequence.

Deblurred Night Image f = 884mm, Exp. Time 4 Sec

Example 3 - Comparison Deblurred image Blurred image Tripod image (Ground Truth) f = 884mm, Exp. Time 4 Sec

Object Deblurring Problem Moving objects blend into the background

Hybrid Imaging Solution (simulated) Requires clear high-resolution background image

Quantifying The Affect of Motion Blur Empirical tests: RMS error. Volume of Solutions (Linear Model): High-Resolution Image Uncertainty (Quantization) Input Images Volume of Solutions 1/det(A ) Blur Decimation

Example 2 - Blurred Indoor Image f = 604mm, Exp. Time 0.5 Sec

PSF Estimation from Motion X (Pixels) Y (Pixels) Estimated PSF f = 604mm, Exp. Time 0.5 Sec Low resolution sequence.

Deblurred Indoor Image f = 604mm, Exp. Time 0.5 Sec

Example 2 - Comparison Deblurred image Blurred image Tripod image (Ground Truth) f = 604mm, Exp. Time 0.5 Sec

Example 2 – Details Tripod Blurred f = 604mm, Exp. Time 0.5 Sec Deblurred

Example 3 – Details f = 884mm, Exp. Time 4 Sec Deblurred Tripod Blurred