1. Mitsubishi Electric Research Labs (MERL) Super-Res from Single Motion Blur PhotoAgrawal & Raskar Amit Agrawal and Ramesh Raskar Mitsubishi Electric Research Labs (MERL) Cambridge, MA, USA Resolving Objects at Higher Resolution from a Single Motion-Blurred Image

2. Mitsubishi Electric Research Labs (MERL) Super-Res from Single Motion Blur PhotoAgrawal & Raskar Can we recover more detail by letting objects blur? Low Res Camera: Static Object Object Moves Text Low Res Camera: Blurred Image Blurred Text: 30 pixels wide Our approach: Resolution enhanced deblurring Resolution enhancement factor = 3 Traditional Motion Deblurring: Recovers 10 pixels 10 pixels 30 pixels Text: 10 pixels wide

3. Mitsubishi Electric Research Labs (MERL) Super-Res from Single Motion Blur PhotoAgrawal & RaskarApplications Time = 0Time = T Aerial Imaging Goal: Automatic license plate number recognition Electronic Toll Booths

4. Mitsubishi Electric Research Labs (MERL) Super-Res from Single Motion Blur PhotoAgrawal & RaskarOverview Exploiting motion blur for super-res –Linear system –No hallucination, image priors or training data Motion deblurring is ill-posed –Practical Solution: Flutter Shutter Camera Resolution enhanced deblurring –Using Flutter Shutter Camera

5. Mitsubishi Electric Research Labs (MERL) Super-Res from Single Motion Blur PhotoAgrawal & Raskar Sensor point spread function (PSF) Ground Truth Static Scene (High Resolution Camera) 20 pixel object … 0 0 0 1 1 0 0 0 … Decimation Matrix D Static Scene (Low Resolution Camera) 10 Decimation Matrix: High Resolution -> Low Resolution

6. Mitsubishi Electric Research Labs (MERL) Super-Res from Single Motion Blur PhotoAgrawal & Raskar Exploiting motion blur for super-res Ground Truth Static Scene (High Resolution Camera) 20 pixel object Static Scene (Low Resolution Camera) 10 pixels Object Moves Traditional Motion Deblurring Blurred Image (High Resolution Camera) 40 Blurred Image (Low Resolution Camera) 20 pixels

7. Mitsubishi Electric Research Labs (MERL) Super-Res from Single Motion Blur PhotoAgrawal & Raskar Blurring Matrix A Sharp Image x Blurred Image b Traditional Motion Deblurring 20 10 Blurred Image (Low Resolution Camera) 20 ? Static Scene (Low Resolution Camera) 10

8. Mitsubishi Electric Research Labs (MERL) Super-Res from Single Motion Blur PhotoAgrawal & Raskar Blurring Matrix A Sharp Image x Blurred Image Low Resolution Traditional Motion Deblurring 10 Blurring Matrix A High Resolution … 0 0 0 1 1 0 0 0 … Decimation Matrix D Sharp Image x ? Resolution Enhanced Deblurring Blurred Image Low Resolution 20

9. Mitsubishi Electric Research Labs (MERL) Super-Res from Single Motion Blur PhotoAgrawal & Raskar Deblurring is ill-posed K = D * A: Combined deblurring + resolution enhancement matrix Typical Case: Condition number of K ~ 10 18 Blurred Image + Noise K Estimated Ground Truth

10. Mitsubishi Electric Research Labs (MERL) Super-Res from Single Motion Blur PhotoAgrawal & RaskarOverview Exploiting motion blur for super-res –Linear system –No hallucination, image priors or training data Motion deblurring is ill-posed –Practical Solution: Flutter Shutter Camera [Raskar, Agrawal, Tumblin, SIGGRAPH 2006] Resolution enhancement deblurring –Using Flutter Shutter Camera

11. Mitsubishi Electric Research Labs (MERL) Super-Res from Single Motion Blur PhotoAgrawal & Raskar Blurring == Convolution Traditional Camera: Shutter is OPEN: Box Filter Frequency Spectrum Magnitude Sinc Function ω

12. Mitsubishi Electric Research Labs (MERL) Super-Res from Single Motion Blur PhotoAgrawal & Raskar Traditional Coded Exposure Image of Static Object Matlab Lucy Deblurred Image

13. Mitsubishi Electric Research Labs (MERL) Super-Res from Single Motion Blur PhotoAgrawal & Raskar Implementation LCD Shutter in front of lens becomes opaque and transparent

14. Mitsubishi Electric Research Labs (MERL) Super-Res from Single Motion Blur PhotoAgrawal & Raskar License Plate Retrieval

15. Mitsubishi Electric Research Labs (MERL) Super-Res from Single Motion Blur PhotoAgrawal & Raskar Related Work Super-Resolution & Motion Deblurring –Sondhi, Proc. IEEE 1972 –Irani & Peleg, CVGIP’91, CVPR’92 –Baker & Kanade, PAMI’02 –Ben-Ezra et al. PAMI 2005 –Bascle et al. ECCV’96 –Tappen et al. SCTV’03 –Shechtman et al. ECCV’02 Coding & Modulation in Cameras/Projectors –HDR, Nayar & Mitsunaga, CVPR’00 –Multiplexed Illumination, Schechner et al. ICCV’03 –Flutter shutter camera, Raskar et al. SIGGRAPH’06 –Coded aperture, Hiura & Matsuyama, CVPR’98 –Coded aperture with lens, light field modulation, SIGGRAPH’07 With Jack Tumblin, Ankit Mohan and Ashok Veeraraghavan (UMD)

16. Mitsubishi Electric Research Labs (MERL) Super-Res from Single Motion Blur PhotoAgrawal & RaskarOverview Exploiting motion blur for super-res –Linear system –No hallucination, image priors or training data Motion deblurring is ill-posed –Practical Solution: Flutter Shutter Camera –Raskar et al. SIGGRAPH 2006 Resolution enhancement deblurring –Using Flutter Shutter Camera

17. Mitsubishi Electric Research Labs (MERL) Super-Res from Single Motion Blur PhotoAgrawal & Raskar M. Ezra, A. Zomet & S.K. Nayar, ‘Video Super-Resolution Using Controlled Subpixel Detector Shifts’, PAMI 2005 Super-resolution with motion blur Jitter Camera Z. Lin & H.Y Shum, ‘Fundamental Limits of Reconstruction-Based Super- resolution Algorithms under Local Translation’, PAMI 2004

18. Mitsubishi Electric Research Labs (MERL) Super-Res from Single Motion Blur PhotoAgrawal & Raskar Experimental Setup Resolution Chart is moved horizontally along the tracks

19. Mitsubishi Electric Research Labs (MERL) Super-Res from Single Motion Blur PhotoAgrawal & Raskar Flutter Shutter: Shutter is OPEN and CLOSED Preserves High Spatial Frequencies

20. Mitsubishi Electric Research Labs (MERL) Super-Res from Single Motion Blur PhotoAgrawal & Raskar Input Blurred Photo (cropped) Deblurred Image Super-Resolution Deblurring Result Deblurred Image Upsampled Unblurred Width Blurred Width Recovered at normal resolution Recovered at triple resolution

21. Mitsubishi Electric Research Labs (MERL) Super-Res from Single Motion Blur PhotoAgrawal & Raskar Input Image Patch Deblurred Patch Our approach

22. Mitsubishi Electric Research Labs (MERL) Super-Res from Single Motion Blur PhotoAgrawal & Raskar Future Work Exploiting Motion Blur –2D Motion Blur: Camera shake –Super-resolution with RGB Bayer mosaic –Coded exposure video with Pointgrey Dragonfly2 camera Focus blur –Coded Aperture –Handling motion and focus blur simultaneously Fluttering Codes –Binary vs Continuous –Suitable for other applications

23. Mitsubishi Electric Research Labs (MERL) Super-Res from Single Motion Blur PhotoAgrawal & RaskarAcknowledgments Jack Tumblin, Northwestern University Hideaki Nii, University of Tokyo MERL –Fatih Porikli, Jay Thornton –Joseph Katz, Keisuke Kojima Haruhisa Okuda, MELCO

24. Mitsubishi Electric Research Labs (MERL) Super-Res from Single Motion Blur PhotoAgrawal & Raskar Super-res from single motion blurred photo Exploiting single photo motion blur –Letting objects blur in 1D –Solving simple linear system –More blur = more resolution but with increased noise Practical implementation –Coded Exposure (Flutter Shutter) Camera –Makes deblurring well-posed Results on 1D resolution enhancement –Using Flutter Shutter Camera

25. Mitsubishi Electric Research Labs (MERL) Super-Res from Single Motion Blur PhotoAgrawal & Raskar Google: ‘Flutter Shutter’ Motion Deblurring Motion Deblurring + Resolution Enhancement Coded Exposure (Flutter Shutter) Camera

26. Mitsubishi Electric Research Labs (MERL) Super-Res from Single Motion Blur PhotoAgrawal & Raskar Extra Slides

27. Mitsubishi Electric Research Labs (MERL) Super-Res from Single Motion Blur PhotoAgrawal & Raskar Captured Blurred Photo Blurred Image Patch Traditional Deblurring Resolution Enhancement Deblurring Upsampled by three times

28. Mitsubishi Electric Research Labs (MERL) Super-Res from Single Motion Blur PhotoAgrawal & Raskar Are all codes “good”? Alternate All ones Random Our Code

29. Mitsubishi Electric Research Labs (MERL) Super-Res from Single Motion Blur PhotoAgrawal & Raskar Need to consider zero padded codes!!!

30. Mitsubishi Electric Research Labs (MERL) Super-Res from Single Motion Blur PhotoAgrawal & Raskar

31. Mitsubishi Electric Research Labs (MERL) Super-Res from Single Motion Blur PhotoAgrawal & Raskar

32. Mitsubishi Electric Research Labs (MERL) Super-Res from Single Motion Blur PhotoAgrawal & Raskar Input Image

33. Mitsubishi Electric Research Labs (MERL) Super-Res from Single Motion Blur PhotoAgrawal & Raskar Rectified Crop Deblurred Result

34. Mitsubishi Electric Research Labs (MERL) Super-Res from Single Motion Blur PhotoAgrawal & Raskar Blurred image: Traditional camera open closed Exposure Time Shutter Continuous blur Blurred image: Coded exposure camera Exposure Time open closed Shutter Coded blur code 110 010100010011000 11 1111111111111111111111 Code: 1101101010001001100011

35. Mitsubishi Electric Research Labs (MERL) Super-Res from Single Motion Blur PhotoAgrawal & Raskar Comparison

36. Mitsubishi Electric Research Labs (MERL) Super-Res from Single Motion Blur PhotoAgrawal & Raskar Inverse Filter Unstable Inverse Filter stable

37. Mitsubishi Electric Research Labs (MERL) Super-Res from Single Motion Blur PhotoAgrawal & Raskar Traditional Motion Deblurring Captured Blurred ImageDeblurred Image Deblurring is at the same resolution

38. Mitsubishi Electric Research Labs (MERL) Super-Res from Single Motion Blur PhotoAgrawal & Raskar Traditional Camera Shutter is OPEN

39. Mitsubishi Electric Research Labs (MERL) Super-Res from Single Motion Blur PhotoAgrawal & Raskar Our Camera Flutter Shutter

40. Mitsubishi Electric Research Labs (MERL) Super-Res from Single Motion Blur PhotoAgrawal & Raskar Shutter is OPEN and CLOSED