Download presentation
Presentation is loading. Please wait.
1
Single Image Rolling Shutter Distortion Correction
Ph.D. seminar talk – 2 Vijay Rengarajan EE11D035 Guides: Prof. A.N.Rajagopalan and Prof. R.Aravind March 22, 2017 Image Processing and Computer Vision lab Department of Electrical Engineering, IIT Madras
2
Camera Motion in Mobile Phones
3
Motion Causes Geometric Distortions
4
Sequential Exposure of Rolling Shutter
Global shutter CCD image sensor Exposure time te Top row time All pixels expose at the same time Bottom row Exposure open Exposure close te Top row time Each row starts exposing sequentially Bottom row Td Total line delay Rolling shutter CMOS image sensor
5
Rolling Shutter Causes Local Distortions
Different rows see the scene at different poses of the moving camera Even short exposure causes distortions Scene Scene rz x z y x z y tx time time Captured image Captured image
6
Prior works use multi-source information
2 3 1 Known undistorted reference frame Local blur kernels within the blurred image Multi-image feature correspondences Motion deblurring Video rectification Super-resolution Forssén and Ringaby (CVPR 2010) Grundmann et al. (ICCP 2012) Punnappurath et al. ICCV 2015 Su and Heidrich CVPR 2015 Super-resolution Change detection Rengarajan et al. ICIP 2016 Rengarajan et al. ECCV 2014, TPAMI 2016
7
Problem Definition - Single Image Correction
Goal Correct the rolling shutter image as if it is captured by a global shutter camera Challenges Only one distorted image No motion blur Geometric distortions
8
Geometry-based Correction
Corrected Image Feature Extraction Motion Estimation Distortion Correction How to correct the distorted image? Which camera motion will “correct” these features? What image features get distorted by camera motion?
9
Camera motion is embedded in curvatures
rz A C D B E A C D B E time Forward mapping Inverse mapping Camera matrix Rotations Translations
10
Curves as distorted features
Use Hough transform to detect small line segments Join line segments based on spatial and angular proximity to detect curves Group curves as vertical, horizontal, and slanted
11
Straightness to be perceptually desirable
1 Estimate camera motion that would make curves straight 3 6M unknowns! 2
12
Polynomial Camera Trajectory
Short exposure warrants simple motion model Model polynomial trajectory w.r.t. row index 6(n+1) unknowns
13
Straightness to be perceptually desirable
Estimate camera motion that would make curves straight Multiple solutions exist along the row dimension that preserve straightness Global inplane rotation Vertical shrinking Horizontal shearing
14
More constraints to zero-in
1 Make curves as lines Nonlinear least squares optimization 1 2 3 L2 distance Curves in vertical/horizontal group as vertical/horizontal lines Preserve vertical length of curves after correction 2 3
15
Results Camera motion estimation based on desirability costs
Distorted image Curve detection Corrected image Correction over iterations
16
Camera motion estimation based on desirability costs
Results Camera motion estimation based on desirability costs Distorted image Curve detection Corrected image Distorted image Corrected image Distorted image Corrected image Vijay Rengarajan, A.N. Rajagopalan, and R. Aravind, “From Bows to Arrows: Rolling Shutter Rectification of Urban Scenes,” IEEE International Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, Nevada, June 2016.
17
On par with video methods
Forssen and Ringaby CVPR 2010 Grundmann et al. ICCP 2012 Captured image Video-based methods Our correction
18
Should all curves be straight?
Distorted image Corrected image
19
Let machines extract desired features
RS Distorted Image Corrected Image Feature Extraction Motion Estimation Distortion Correction Motion Fitting Distortion Correction Corrected Image RS Distorted Image CNN CNN – Convolutional Neural Network Non-linearly maps distorted image to camera motion Estimates motion values at fixed number of rows Fits a polynomial trajectory
20
Learn image to motion as a nonlinear mapping
Motion Mean Squared Error tx and rz at 15 rows Vanilla Convolutional Neural Network ReLU Tanh HardTanh 21 19 23 34 1 5 2 4 45 31 11 7 9 MaxPool /2 MaxPooling layer Convolution layer Conv Fully connected layer FC
21
Training the CNN Stochastic Gradient Descent Iteration Image Motion
- Pass an input forward to obtain its output - Update for this input Calculate output gradient, Backpropagate the gradient through all layers to obtain Use a batch of images instead of a single input CNN Image Motion Training Data Generation Generate random camera motion Apply on undistorted image Datasets Chessboard Urban scenes Faces 7k k k Sun Oxford Zurich LFW
22
Correction Results of VanillaCNN
1 2 Translations only Translations and rotations Distorted image Corrected by VanillaCNN Distorted image Corrected by VanillaCNN
23
VanillaCNN filters cover smaller field of view
11x11 7x7 5x5 3x3 Without maxpooling 11x11 18x18 23x23 26x26 1x1 11x11 14x14 20x20 24x24 With maxpooling 25x25 45x45 69x69 11x11
24
Modify filter shape to suit rolling shutter
Long filters for convolutions Square filters for convolutions Captures information slowly over many layers Captures information along row dimension early Captures information along time dimension early
25
RowColCNN architecture
VanillaCNN Initial feature extraction Feature combination
26
Use long filters for RowColCNN
Motion Mean Squared Error
27
Correction Results of RowColCNN
Vijay Rengarajan, Yogesh Balaji, and A.N. Rajagopalan, “Unrolling the Shutter: CNN to Correct Motion Distortions,” IEEE International Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, Hawaii, July 2017.
28
Learning wins in challenging conditions
Distorted input Geometric-based Learning-based Rengarajan et al. 2016 Heflin et al. 2010
29
Machines as good as training data
Geometry-based correction Human tailored distorted-feature selection Motion mapping based on straightness measure Learning-based correction Machines learn feature extraction and motion mapping Dependent on training data
30
Machines (only) as good as training data
Geometry-based correction Human tailored distorted-feature selection Motion mapping based on straightness measure Learning-based correction Machines learn feature extraction and motion mapping Dependent on training data Trained on building datasets Vijay Rengarajan, A.N. Rajagopalan, and R. Aravind, “From Bows to Arrows: Rolling Shutter Rectification of Urban Scenes,” IEEE International Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, Nevada, June 2016. Vijay Rengarajan, Yogesh Balaji, and A.N. Rajagopalan, “Unrolling the Shutter: CNN to Correct Motion Distortions,” IEEE International Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, Hawaii, July 2017. apvijay.github.io/rs_demo
31
Backup
32
Rolling Shutter Causes Local Distortions
Different rows see different poses of the camera under motion Even short exposure causes distortions x z y x z y tx rz Captured image Scene Captured image Scene time time
33
Rolling Shutter Causes Local Distortions
Different rows see different poses of the camera under motion Even short exposure causes distortions x z y x z y tx rz Captured image Scene Captured image Scene time time
34
What to learn? Motion or Image?
Similar presentations
