1. Dr. Yossi Rubner yossi@rubner.co.il
Super-Resolution
Dr. Yossi Rubner
Many slides from Miki Elad - Technion

2. Example - Video
53 images, ratio 1:4

3. Example – Surveillance
40 images ratio 1:4

4. Example – Enhance Mosaics

6. Super-Resolution - Agenda
The basic idea
Image formation process
Formulation and solution
Special cases and related problems
Limitations of Super-Resolution
SR in time

7. Intuition D
For a given band-limited image, the Nyquist sampling theorem states that if a uniform sampling is fine enough (D), perfect reconstruction is possible.

8. Intuition 2D
Due to our limited camera resolution, we sample using an insufficient 2D grid

9. Intuition 2D
However, if we take a second picture, shifting the camera ‘slightly to the right’ we obtain:

10. Intuition
Similarly, by shifting down we get a third image: 2D 2D

11. Intuition
And finally, by shifting down and to the right we get the fourth image: 2D 2D

12. Intuition
It is trivial to see that interlacing the four images, we get that the desired resolution is obtained, and thus perfect reconstruction is guaranteed.

13. Rotation/Scale/Disp.
What if the camera displacement is Arbitrary ? What if the camera rotates? Gets closer to the object (zoom)?

14. Rotation/Scale/Disp.
There is no sampling theorem covering this case

15. A Small Example
3:1 scale-up in each axis using 9 images, with pure global translation between them

16. Further Complications
Complicated motion
perspective, local motion, …
Blur
sampling is not a point operation
Spatially variant blur
Temporally variant blur
Noise
Changes in the scene

17. Super-Resolution - Agenda
The basic idea
Image formation process
Formulation and solution
Special cases and related problems
Limitations of Super-Resolution
SR in time

18. Image Formation LR HR Scene Geometric transformation Optical Blur
Sampling
Noise HR LR
Can we write these steps as linear operators?

19. Geometric Transformation
Any appropriate motion model
Every frame has different transformation
Usually found by a separate registration algorithm
Geometric
transformation
Scene

20. Geometric Transformation
Can be modeled as a linear operation =

21. Optical Blur Due to the lens PSF and pixel integration Usually
Geometric
transformation
Optical
Blur

22. H
PSF *
PIXEL = H

23. Optical Blur
Can be modeled as a linear operation =

24. Sampling
Pixel operation consists of area integration followed by decimation
D is the decimation only
Usually
Optical Blur
Sampling

25. Decimation
Can be modeled as a linear operation = o 1

26. Super-Resolution - Agenda
The basic idea
Image formation process
Formulation and solution
Special cases and related problems
Limitations of Super-Resolution
SR in time

27. Super-Resolution - Model
F =I 1 F N
Geometric warp H
Blur 1 N D 1 N
Decimation Y 1 N
Low-
Resolution
Exposures X
High-
Resolution
Image V 1 N
Additive Noise

28. Simplified Model X F =I F Geometric warp H Blur D Decimation Y Low-1 F N
Geometric warp H
Blur D
Decimation Y 1 N
Low-
Resolution
Exposures X
High-
Resolution
Image V 1 N
Additive Noise

29. The Super-Resolution Problem
Given
Yk – The measured images (noisy, blurry, down-sampled ..)
H – The blur can be extracted from the camera characteristics
D – The decimation is dictated by the required resolution ratio
Fk – The warp can be estimated using motion estimation
n – The noise can be extracted from the camera / image
Recover
X – HR image

30. The Model as One Equation
=[10M×16M]
=[16M×1]
r = resolution factor = 4
MXM = size of the frames = 1000X1000
N = number of frames = 10
r = resolution factor
MXM = size of the frames
N = number of frames
Linear algebra notation is intended only to develop algorithm

31. SR - Solutions Maximum Likelihood (ML):
Often ill posed problem!
Maximum Aposteriori Probability (MAP)
Smoothness constraint
regularization

32. ML Reconstruction (LS)
Minimize:
Thus, require:

33. LS - Iterative Solution
Steepest descent
Simulated error
Back projection
All the above operations can be interpreted as operations performed on images.
There is no actual need to use the Matrix-Vector notations as shown here.

34. LS - Iterative Solution
Steepest descent
For k=1..N -
inverse
geometry
wrap
geometry
wrap
convolve
with H
down
sample up
sample
convolve
with HT -

35. Example LR + noise X4 Least squares HR image
Simulated example from Farisu at al.
IEEE trans. On Image Processing, 04

36. Robust Reconstruction
Cases of measurements outlier:
Some of the images are irrelevant
Error in motion estimation
Error in the blur function
General model mismatch

37. Robust Reconstruction
Minimize:

38. Robust Reconstruction
Steepest descent
For k=1..N -
inverse
geometry
wrap
geometry
wrap
convolve
with H
down
sample up
sample
convolve
with HT
sign -

39. Robust Reconstruction
Example - Outliers
Least squares
LR + noise X4
Robust Reconstruction
HR image
Simulated example from Farisu at al.
IEEE trans. On Image Processing, 04

40. Example – Registration Error
L2 norm based
L1 norm based
20 images, ratio 1:4

41. MAP Reconstruction Regularization term: Tikhonov cost function
Total variation
Bilateral filter

42. Robust Estimation + Regularization
Minimize:

43. Robust Estimation + Regularization
For k=1..N -
inverse
geometry
wrap
geometry
wrap
convolve
with H
down
sample up
sample
convolve
with HT
sign -
For l,m=-P..P
horizontal
shift l
vertical
shift m -
horizontal
shift -l
vertical
shift -m -
sign
From Farisu at al. IEEE trans. On Image Processing, 04

44. Example 8 frames Resolution factor of 4
From Farisu at al. IEEE trans. On Image Processing, 04

45. Example
Images from Vigilant Ltd.

46. Handling Color in SR
Handling color: the classic approach is to convert the measurements to YCbCr, apply the SR on the Y and use trivial interpolation on the Cb and Cr.
Better treatment can be obtained if the statistical dependencies between the color layers are taken into account (i.e. forming a prior for color images).
In case of mosaiced measurements, demosaicing followed by SR is sub-optimal. An algorithm that directly fuse the mosaic information to the SR is better.

47. SR for Full Color
20 images, ratio 1:4

48. SR+Demosaicing 20 images, ratio 1:4 Mosaiced input
Mosaicing and then SR Combined treatment

49. Super-Resolution - Agenda
The basic idea
Image formation process
Formulation and solution
Special cases and related problems
Limitations of Super-Resolution
SR in time

50. Special Case – Translational Motion
In this case H and F commute:
SR is decomposed into 2 steps
Find blur HR image from LR images  non-iterative
Deconvolve the result using H  iterative

51. Intuition Z=PIXEL*PSF*X X PSF*X
Using the samples can, at most, reconstruct Z
To recover X, need to deconvolve Z

52. Step I – Find Blurred HR
Minimize:
L2  For all frames, copy registered pixels to HR grid and average [Elad & Hel-Or, 01]
L1  For all frames, copy registered pixels to HR grid and use median [Farisu, 04]

53. Solution for L2 Minimize: Thus, require: Diagonal, number
Of occurrences
per HR grid
Sum of HR grid

54. Step II - Deblur
Minimize:

55. Example 256X256 Before deblur 256X256 After deblur 64X64 LR
From Pham at al. Proc. Of SPIE-IS&T, 05. Simulated.

56. Related Problems Denoising (multiple frames) Denoising (single frame)
Deblurring
Interpolation – “single-image super-resolution”

57. Super-Resolution - Agenda
The basic idea
Image formation process
Formulation and solution
Special cases and related problems
Limitations of Super-Resolution
SR in time

58. Limiting Factors Main factors SNR PSF (optical+pixel) Number of inputs
Pixel size (sampling rate)
Fill factor
Image content

59. Limiting Factors … SNR PSF # of inputs LR1 Registration Fusion
Deblurring HR …
Pixel size
Fill factor
Image content
LRN

60. SR Limits Analysis Noise SR factor Registration noise Fusion noise
Point-Spread-Function (PSF)
Optical Transfer Function (OTF)
Sensor pixel size
Sensor Transfer Function (STF)

61. Registration Noise Using Cramer Rao Lower Bound (CRLB)
Lower bounds for shift estimation:
Better registration accuracy by:
Less noise
Higher derivatives in image
Bigger registration area
Narrow PSF

62. Fusion Noise
r = super-resolution factor
N = number of images

63. Registration + Fusion Noise
If N∞ then
Fusion error vanishes
Registration error is equivalent to Gaussian blur

64. Super-Resolution - Agenda
The basic idea
Image formation process
Formulation and solution
Special cases and related problems
Limitations of Super-Resolution
SR in time
Work and slides by Michal Irani & Yaron Caspi (ECCV’02)

65. “Classical” Image Super-Resolution
Scene:
Low-resolution images:
High-resolution image:

66. Space-Time Super-Resolution
Low-resolution images
video sequences:
time
time
High space-time resolution sequence:
time
Super-resolution in space and in time.

67. What is Super-Resolution in Time?
Observing events “faster” than frame-rate.
Handles:
(1) Motion aliasing
(2) Motion blur
Application areas:
- sports scenes
- scientific imaging
- etc...

68. The “Wagon wheel” effect:
(1) Motion Aliasing
The “Wagon wheel” effect:
Slow-motion:
time
Continuous signal
time
Sub-sampled in time
time
“Slow motion”

69. (2) Motion Blur

70. (2) Motion Blur Camera 1: long exposure-time
Camera 2: short exposure-time
Motion-blur – A Spatial artifact caused by Temporal blurring.
exposure time
time
Operation of camera:
1/frame-rate

71. Space-Time Super-Resolution
Sh(xh,yh,th) t x y y x t
Blur kernel:
PSF
Exposure time

72. Example: Motion-Aliasing
Input 1
Input 2
Input 3
Input 4
25 [frames/sec]

73. Example: Motion-Aliasing
Input sequence in slow-motion (x3):
75 [frames/sec]
Super-resolution in time (x3):
75 [frames/sec]

74. Without estimating motion of the ball!
Deblurring:
Input:
Output:
Output sequence:
(x15 frame-rate)
Output trajectory:
Without estimating motion of the ball!
3 out of 18 low-resolution input sequences (frame overlays; trajectories):

75. Example: Motion-Blur (real)
4 input sequences:
Frames at collision:
Video 1
Video 2
Output frame at collision:
Video 3
Video 4