1. EE 392J – Digital Video Processing
Super-Resolution
Deepesh Jain
EE 392J – Digital Video Processing
Stanford University
Winter

2. Motivation
Create High Resolution Video from a low-resolution one
Create High Resolution Image(s) from a video or collection of low-res images. Applications:
Action Packed Sports Images (Basketball dunk, Gymnastics, etc)
Astronomy
Medical Imaging
This project – Create a high-res image from bunch of low-res ones (constraints: global motion – shift & rotation)

3. Approach Image Registration – Motion Estimation
Projection onto High-Res grid
Nonuniform Interpolation
Frequency Domain
Iterative Back Projection (IBP)
POCS (Projection onto convex sets)
Registration
Projection
Low-res Images
Registration
(sub-pixel grid)
High Res Grid

4. 1.1 Registration (angle)
Rotation Calculation
Correlate 1st LR image with all LR images at all angles OR
Calculate energy at all angles for all LR images. Correlate energy vector to find the rotation angle
Anglei = max index(correlation(I1(θ), Ii (θ)))
Energy at angle Ii(θ)
LR image 1
Energy at angle I2(θ)
LR image 2
i = 2,3,..,N (number of LR images)

5. Δs = angle( Fi (uT) / F1(uT) )
1.2 Registration (shift)
Shift Calculated using Frequency Domain Method
Fi (uT) = ej2πuΔsF1(uT)
Δs = angle( Fi (uT) / F1(uT) )
2πu
Δs  [Δx Δy]T
u  [fx fy]
Used only 6% lower u (high freq could be aliased)
Used least square to calculate Δs

6. 2.1 Frequency Domain Input  Down-sampled aliased images
Goal I Correct the low-freq aliased data
Goal II  Predict the lost high freq values π -π
Original High-Res π -π
Down-sampled
π/2
-π/2 π
Aliased (fix it)
Lost (find it)
Up-sampled π -π
Desired High-Res

7. 2.2 Projection onto High-res grid
Papoulis-Gerchberg Algorithm (special case of POCS)
Correct the low-freq values. Assumes high-freq part to be zero.
Projection onto 2 convex sets
Known pixel values
Known Cut-off freq in the HR image
Algorithm:
I (known pixel positions) = Known Values
I_fft = fft2(I)
I_fft(higher Freq) = 0
I= ifft2 (I_fft)

8. Papoulis – Gerchberg Algorithm
Initial Setup
Taj Mahal – Low-res image I
FFT(Reconstructed image)
Reconstructed image from known pixels

9. Papoulis – Gerchberg Algorithm
Known Pixel
Values
Image at iteration 0
Image after 1st iteration
I(high freq) =0
FFT

10. Papoulis – Gerchberg Algorithm
Known Pixel
Values
Image at iteration 1
Image after 10 iterations
I(high freq) =0
FFT

11. Papoulis – Gerchberg Algorithm
After 50 iterations
Taj Mahal – Low-res image 1
Bilinear Interpolation
Bicubic Interpolation
SR Reconstructed image

12. Results (Synthetic Images)
Results (Real images)
Took 4 snaps using a high-res digital camera
Cropped the same part of each image
Applied SR algorithm & compared it with bicubic interpolation
Results (Synthetic Images)
Constructed 4 low-res images by shifting and down-sampling 1 high-res image.
Applied SR algorithm & compared it with bicubic interpolation

13. Results (Real Images - I)
Original Low-res images
(Courtesy: Patrick Vandewalle)

14. Results (Real Images - I)
Bicubic Interpolation

15. Results (Real Images - I)
Super-resolution

16. Results (Real Images - II)
Low-Res Image I
Low-Res Image II
Didn’t WORK !!!
Motion was not restricted to shifts & rotation
Images had affine mapping.
Rule I – Need Correct Registration

17. Results (Synthetic Image - I)
Original High-Res
Down-sampled

18. Results (Synthetic Image - I)
Bicubic Interpolation

19. Results (Synthetic Image - I)
Super-Resolution

20. Results (Synthetic Image - II)
Original
Bicubic SR
Why didn’t SR work???
Low-res images were created by forcing shifts at critical velocities
Rule II  If low-res images are at critical velocities, can’t create good HR image

21. Results (Synthetic Image - III)
Original
Bicubic SR
Why did SR work so well???
Low-res images were created by forcing shifts at non-critical velocities
Rule III  If low-res images have all the info about high-res then HR image can be perfectly constructed

22. Future Work
Superresolution with multiple motions between frames  create high res video
Predict the high-res frequency components using wavelet methods
Predict
Predict
Predict

23. Acknowledgements Prof John Apostolopoulos Prof Susie Wee
Patrick Vandewalle
Q & A ???
Comments !!!!