1. Motion Deblurring Using Hybrid Imaging Moshe Ben-Ezra and Shree K. Nayar Columbia University IEEE CVPR Conference June 2003, Madison, USA

2. Image Recording Requires Time Niépce 1827 8 hours exposure Daguerre 1829 1/2 hour exposure

3. Motion Blur is Everywhere Object Motion Camera Motion

4. Stabilized Lenses 1/250 second (< -1 stop) Stabilization drifts with time Rotation only 1/15 second (< -5 stops) Canon Stabilized lens 400mm

5. Blind Image Deconvolution Accurate Point Spread Function (PSF) Needed.

6. Motion Point Spread Function (PSF) Motion PSF is a Function of: 1. Motion path 2. Motion speed X Y Energy ~ 1/ speed Spatial spread H

7. PSF Detector? Camera PSF Detector Can the PSF detector be a small and simple imaging device ?

8. Electron wells Fundamental Limits of Imaging Detector’s noise level Photon flux Detector Pixel’s Signal Noise

9. Fundamental Resolution Tradeoff Spatial resolution (pixels) Temporal resolution (fps) 30 330K 720x480 Conventional video camera 130 3 3M 2048x1536 Hi-resolution camera 75K 320x240 Low-resolution camera Hybrid imaging system A Hybrid camera enjoys both worlds

10. Overview of Approach PSF Estimation Low-Res. camera Hi-Res. camera Same time period Deconvolution Motion Analysis x y

11. Global Motion From Low Resolution Detector TranslationRotation Objective function (Optical flow constraint) Lucas Kanade

12. Simulations: Motion Accuracy from Low- Res. Images Noise Resolution  = 3  = 9  = 27  = 81 640x640 (1:1)0.01 0.020.04 320x320 (1:4)0.030.040.050.1 160x160 (1:16)0.030.040.070.4 80x80 (1:64)0.130.210.392.6 Average Motion Error in Pixels

13. Constraints on Continuous PSF Energy conservation constraint: Path is continuous and twice differentiable Constant flux assumption: Smoothness constraint:

14. PSF Estimation from Computed Motion x f1f1 f2f2 f3f3 f4f4 f5f5 f6f6 y Frame 2 … Frame 5 y f1f1 f2f2 f3f3 f4f4 f5f5 f6f6 x y h h2h2 h3h3 h4h4 h5h5 Frame 2 … Frame 5 y h h2h2 h3h3 h4h4 h5h5 Frame 2 … Frame 5 x

15. Deconvolution of High Resolution Image Standard iterative ratio-based algorithm* Guaranties non-negative pixel result * Richardson [72] Lucy [74] ErrorPSFImage estimate

16. Designs for Hybrid Imaging A rig of two cameras Using a special chip Using a beam splitter

17. Our Prototype: Rig of Two Cameras Primary detector (2048x1536) Secondary detector (360x240) Resolution ratio of 1 : 36

18. Example 1 - Blurred Hi-Res Image f = 633mm, Exp. Time 1 Sec (> -9 stops)

19. PSF Estimation from Motion Low resolution sequence. X (Pixels) 10130 10 90 Y (Pixels) 0.001 0.06 Estimated PSF f = 633mm, Exp. Time 1 Sec

20. Deblurred Image f = 633mm, Exp. Time 1 Sec

21. Example 1 - Comparison Deblurred image Blurred image f = 633mm, Exp. Time 1 Sec Tripod image (Ground Truth)

22. Example 2 - Blurred Night Image f = 884mm, Exp. Time 4 Sec (> -11 stops)

23. PSF Estimation from Motion X (Pixels) 1060 10 30 Y (Pixels) 0.001 0.003 f = 884mm, Exp. Time 4 Sec Low resolution sequence.

24. Deblurred Night Image f = 884mm, Exp. Time 4 Sec

25. Example 3 - Comparison Deblurred image Blurred image Tripod image (Ground Truth) f = 884mm, Exp. Time 4 Sec

26. Object Deblurring Problem Moving objects blend into the background

27. Hybrid Imaging Solution (simulated) Requires clear high-resolution background image

29. Quantifying The Affect of Motion Blur Empirical tests: RMS error. Volume of Solutions (Linear Model): High-Resolution Image Uncertainty (Quantization) Input Images Volume of Solutions 1/det(A ) Blur Decimation

30. Example 2 - Blurred Indoor Image f = 604mm, Exp. Time 0.5 Sec

31. PSF Estimation from Motion X (Pixels) 1090 10 30 Y (Pixels) 0.002 0.022 Estimated PSF f = 604mm, Exp. Time 0.5 Sec Low resolution sequence.

32. Deblurred Indoor Image f = 604mm, Exp. Time 0.5 Sec

33. Example 2 - Comparison Deblurred image Blurred image Tripod image (Ground Truth) f = 604mm, Exp. Time 0.5 Sec

34. Example 2 – Details Tripod Blurred f = 604mm, Exp. Time 0.5 Sec Deblurred

35. Example 3 – Details f = 884mm, Exp. Time 4 Sec Deblurred Tripod Blurred