1. Authors: Wagner, Waagen and Cassabaum Presented By: Mukul Apte
Super-Resolution
Authors: Wagner, Waagen and Cassabaum
Presented By: Mukul Apte

2. Introduction
Definition: Create High Resolution visual output from Low Resolution visual input.
Mathematical assistance to features viz. motion detection, face recognition, person detection.
Action Packed Sports Images
Astronomy
Medical Imaging
Surveillance
Range of resources: low (single camera, polaroid lenses) to high (super-resolution chips, androids-ASIMO)

3. Basic idea
Given: A set of degraded (warped, blurred, decimated, noised) images:
Required: Fusion of the measurements into a higher resolution image:

4. Types
Static Super-Resolution (SSR) - The creation of a single improved image, from the finite measured sequence of images
Dynamic Super-Resolution (SSR) - Low Quality Movie In - High Quality Movie Out

5. Simple Example
Concepts I: Raster, CCD, Image in mathematical domain, Matlab
Concepts II: Bresenham’s line algorithm, non-uniform interpolation, frequency domain

6. Simple Example D
For a given band-limited image, the Nyquist sampling theorem states that if a uniform sampling is fine enough (pixel density = twice highest static frequency), perfect reconstruction is possible.

7. Simple Example 2D
Due to our limited camera resolution, we sample using an insufficient 2D grid

8. Simple Example 2D
However, we are allowed to take a second picture and so, shifting the camera ‘slightly to the right’ we obtain

9. Simple Example
Similarly, by shifting down we get a third image 2D 2D

10. Simple Example
And finally, by shifting down and to the right we get the fourth image 2D 2D

11. Simple Example - Conclusion
It is trivial to see that interlacing the four images, we get that the desired resolution is obtained, and thus perfect reconstruction is guaranteed.
This is Super-Resolution in its simplest form

12. Uncontrolled Displacements
In the previous example we counted on exact movement of the camera by D in each direction.
What if the camera displacement is uncontrolled?

13. Uncontrolled Displacements
It turns out that there is a sampling theorem due to Yen (1956) and Papoulis (1977) covering this case, guaranteeing perfect reconstruction for periodic uniform sampling if the sampling density is high enough (1 sample per each D-by-D square).

14. Uncontrolled Rotation/Scale/Disp.
In the previous examples we restricted the camera to move horizontally/vertically parallel to the photograph object.
What if the camera rotates? Gets closer to the object (zoom)?

15. Uncontrolled Rotation/Scale/Disp.
There is no sampling theorem covering this case

16. Further Complications
Sampling is not a point operation – there is a blur
Motion may include perspective warp, local motion, etc.
Samples may be noisy – any reconstruction process must take that into account.

17. Static Super-Resolution
Low Resolution Measurements
Static
Super-Resolution
Algorithm
High Resolution Reconstructed Image

18. Dynamic Super-Resolution
Low Resolution Measurements t
Dynamic
Super-Resolution
Algorithm
High Resolution Reconstructed Images t

19. Approach Image Registration Motion Estimation
Projection onto High-Resolution Grid
Non-Uniform Interpolation
Frequency domain – alias correction
Registration
Projection
Low-res Images
Registration
(sub-pixel grid)
High Res Grid

20. 1.1 Registration (angle)
Rotation Calculation
Correlate 1st LR image with all LR images at all angles OR
Calculate energy at all angles for all LR images. Correlate energy vector to find the rotation angle
Anglei = max index(correlation(I1(θ), Ii (θ)))
LR image 1
LR image 2
i = 2,3,..,N (number of LR images)
Energy at angle Ii(θ)
Energy at angle I2(θ)

21. Δs = angle( Fi (uT) / F1(uT) )
1.1 Registration (shift)
Shift Calculated using Frequency Domain Method
Fi (uT) = ej2πuΔsF1(uT)
Δs = angle( Fi (uT) / F1(uT) )
2πu
Δs  [Δx Δy]T
u  [fx fy]
Used only 6% lower u (high freq could be aliased)
Used least square to calculate Δs

22. 2.1 Frequency Domain Input: Down-sampled aliased images
Goal I: Correct the low-freq aliased data
Goal II: Predict the lost high freq values -π π -π π
Original High-Res
Down-sampled
Aliased (fix it)
Lost (find it)
-π/2
π/2 π -π π
Up-sampled
Desired High-Res

23. 2.2 Projection onto High-res grid
Papoulis-Gerchberg Algorithm
Projection onto convex sets
Known pixel values
Known Cut-off freq in the HR image
Algorithm:
I (known pixel positions) = Known Values
I_fft = fft2(I)
I_fft(higher Freq) = 0
I= ifft2 (I_fft)

24. SSR – The Model Geometric Warp X F =I F H Blur D Decimation Y Low-1 F N
Geometric Warp H
Blur 1 N D 1 N
Decimation Y 1 N
Low-
Resolution
Images X
High-
Resolution
Image V 1 N
Additive Noise

25. Per every point in X find a matching point in Z
Warp – Linear Operation X Z
Per every point in X find a matching point in Z
F[j,i]=1

26. Model Assumptions
We assume that the images Yk and the operators Hk, Dk, Fk,& Wk are known to us, and we use them for the recovery of X.
Yk – The measured images (noisy, blurry, down-sampled ..)
Hk – The blur can be extracted from the camera characteristics
Dk – The decimation is dictated by the required resolution ratio
Fk – The warp can be estimated using motion estimation
Wk – The noise covariance can be extracted from the camera
characteristics

27. Special Condition - Noiseless
Clearly, this linear system of equations should have more equations than unknowns in order to make it possible to have a unique solution.

28. SSR – Handling Problems
Single image de-noising
Single image restoration
Single image scaling
Motion compensation average
Using

29. (Typically 15-20 iterations are performed.)
SSR Standard Equation
Simulated error
Weighted edges
Back projection
All the above operations can be interpreted as iterative operations performed on images.
(Typically iterations are performed.)

30. Thumb Rule on Desired Resolution
Assume that we have N images of M-by-M pixels, and we would like to produce an image X of size L-by-L. Then –

31. Papoulis – Gerchberg Algorithm
Initial Setup
FFT (Initial image)
Taj Mahal – Low-res image I

32. Papoulis – Gerchberg Algorithm
Known Pixel
Values
Image at iteration 0
Image after 1st iteration
FFT
I(high freq) =0

33. Papoulis – Gerchberg Algorithm
Known Pixel
Values
Image at iteration 1
Image after 10 iterations
FFT
I(high freq) =0

34. Papoulis – Gerchberg Algorithm
After 50 iterations
Bilinear Interpolation
Bicubic Interpolation
SR Reconstructed image

35. Results (Images - I) Input: 4 snaps using a high-res digital camera
Cropped the same part of each image
SR algorithm compared with bicubic interpolation
Original Low-res images
(Courtesy: Patrick Vandewalle)

36. Results (Images - I)
Bicubic Interpolation

37. Results (Images - I)
Super-resolution

38. Results (Images - II) SR Algorithm didn’t work as expected !!! Reason:
Low-Res Image I
Low-Res Image II
SR Algorithm didn’t work as expected !!!
Reason:
Motion was not restricted to shifts & rotation
Images had affine mapping
Rule I - Need Correct Registration

39. Results (Frames - I) Why didn’t SR work???
Original
Bicubic SR
Why didn’t SR work???
Low-res images were created by forcing shifts at critical velocities
Rule II - If low-res frames are at critical velocities, can’t create good high-res frame

40. Results (Frames - II) Why did SR work so well???
Original
Bicubic SR
Why did SR work so well???
Low-res images were created by forcing shifts at non-critical velocities
Rule III  If low-res images have all the info about high-res then HR image can be perfectly constructed

41. Future Work
Superresolution with multiple motions between frames - create high res video
Predict the high-res frequency components using wavelet methods
Predict
Predict
Predict

42. Acknowledgements Prof John Apostolopoulos Prof Susie Wee
Patrick Vandewalle

43. THANK YOU!!!
QUESTIONS?
COMMENTS?