1. CPSC 641: Image Registration
Jinxiang Chai

2. Review
Image warping
Image morphing

3. Image Warping Warping function - similarity, affine, projective etc
- forward warping and two-pass 1D warping
- backward warping
Resampling filter
- point sampling
- bilinear filter
- anisotropic filter x u
Inverse y v
forward
S(x,y)
T(u,v)

4. Image Morphing
Point based image morphing
Vector based image morphing

5. Image Registration Image warping: given h and f, compute g
g(x) = f(h(x)) g? f h
Image registration: given h and g, compute f g f h?

6. Why Image Registration?
Lots of uses
Correct for camera jitter (stabilization)
Align images (mosaics)
View morphing
Image based modeling/rendering
Special effects
Etc.

7. Image Registration How do we align two images automatically?
Two broad approaches:
Feature-based alignment
Find a few matching features in both images
compute alignment
Direct (pixel-based) alignment
Search for alignment where most pixels agree

8. Outline Image registration - feature-based approach
- pixel-based approach

9. Readings
Bergen et al. Hierarchical model-based motion estimation. ECCV’92, pp. 237–252.
Shi, J. and Tomasi, C. (1994). Good features to track. In CVPR’94, pp. 593–600.
Baker, S. and Matthews, I. (2004). Lucas-kanade 20 years on: A unifying framework. IJCV, 56(3), 221–255.

10. Outline Image registration - feature-based approach
- pixel-based approach

11. Feature-based Alignment
Find a few important features (aka Interest Points)
Match them across two images
Compute image transformation

12. Feature-based Alignment
Find a few important features (aka Interest Points)
Match them across two images
Compute image transformation
How to choose features
Choose only the points (“features”) that are salient, i.e. likely to be there in the other image

13. Feature-based Alignment
Find a few important features (aka Interest Points)
Match them across two images
Compute image transformation
How to choose features
Choose only the points (“features”) that are salient, i.e. likely to be there in the other image
How to find these features?
windows where has two large eigenvalues
Harris Corner detector

14. Feature Detection
Two images taken at the same place with different angles
Projective transformation H3X3

15. Feature Matching ?
Two images taken at the same place with different angles
Projective transformation H3X3

16. Feature Matching Intensity/Color similarity Distance constraint
The intensity of pixels around the corresponding features should have similar intensity
SSD, Cross-correlation
Distance constraint
The displacement of features should be smaller than a user-defined threshold

17. Feature-space Outlier Rejection
Can we now compute H3X3 from the blue points?
No! Still too many outliers…
What can we do?

18. Robust Estimation
What if set of matches contains gross outliers?

19. RANSAC
Objective
Robust fit of model to data set S which contains outliers
Algorithm
Randomly select a sample of s data points from S and instantiate the model from this subset.
Determine the set of data points Si which are within a distance threshold t of the model. The set Si is the consensus set of samples and defines the inliers of S.
If the subset of Si is greater than some threshold T, re-estimate the model using all the points in Si and terminate
If the size of Si is less than T, select a new subset and repeat the above.
After N trials the largest consensus set Si is selected, and the model is re-estimated using all the points in the subset Si

20. RANSAC
Repeat M times:
Sample minimal number of matches to estimate two view relation (affine, perspective, etc).
Calculate number of inliers or posterior likelihood for relation.
Choose relation to maximize number of inliers.

21. RANSAC Line Fitting Example
Task:
Estimate best line

22. RANSAC Line Fitting Example
Sample two points

23. RANSAC Line Fitting Example
Fit Line

24. RANSAC Line Fitting Example
Total number of points within a threshold of line.

25. RANSAC Line Fitting Example
Repeat, until get a good result

26. RANSAC Line Fitting Example
Repeat, until get a good result

27. RANSAC Line Fitting Example
Repeat, until get a good result

28. proportion of outliers e
How Many Samples?
Choose N so that, with probability p, at least one random sample is free from outliers. e.g. p=0.99
proportion of outliers e s 5%
10%
20%
25%
30%
40%
50% 2 3 5 6 7 11 17 4 9 19 35 13 34 72 12 26 57
146 16 24 37 97
293 8 20 33 54
163
588 44 78
272
1177

29. proportion of outliers e
How Many Samples?
Choose N so that, with probability p, at least one random sample is free from outliers. e.g. p=0.99
proportion of outliers e s 5%
10%
20%
25%
30%
40%
50% 2 3 5 6 7 11 17 4 9 19 35 13 34 72 12 26 57
146 16 24 37 97
293 8 20 33 54
163
588 44 78
272
1177
Affine transform

30. proportion of outliers e
How Many Samples?
Choose N so that, with probability p, at least one random sample is free from outliers. e.g. p=0.99
proportion of outliers e s 5%
10%
20%
25%
30%
40%
50% 2 3 5 6 7 11 17 4 9 19 35 13 34 72 12 26 57
146 16 24 37 97
293 8 20 33 54
163
588 44 78
272
1177
Projective transform

31. RANSAC for Estimating Projective Transformation
RANSAC loop:
Select four feature pairs (at random)
Compute the transformation matrix H (exact)
Compute inliers where SSD(pi’, H pi) < ε
Keep largest set of inliers
Re-compute least-squares H estimate on all of the inliers

32. RANSAC

33. Outline Image registration - feature-based approach
- pixel-based approach

34. Direct (pixel-based) Alignment : Optical flow
Will start by estimating motion of each pixel separately
Then will consider motion of entire image

35. Problem Definition: Optical Flow
How to estimate pixel motion from image H to image I?

36. Problem Definition: Optical Flow
How to estimate pixel motion from image H to image I?
Solve pixel correspondence problem
given a pixel in H, look for nearby pixels of the same color in I

37. Problem Definition: Optical Flow
How to estimate pixel motion from image H to image I?
Solve pixel correspondence problem
given a pixel in H, look for nearby pixels of the same color in I
Key assumptions
color constancy: a point in H looks the same in I
For grayscale images, this is brightness constancy
small motion: points do not move very far
This is called the optical flow problem

38. Optical Flow Constraints
Let’s look at these constraints more closely
brightness constancy: Q: what’s the equation?
A: H(x,y) = I(x+u, y+v)

39. Optical Flow Constraints
Let’s look at these constraints more closely
brightness constancy: Q: what’s the equation?
A: H(x,y) = I(x+u, y+v)
H(x,y) - I(x+u,v+y) = 0

40. Optical Flow Constraints
Let’s look at these constraints more closely
brightness constancy: Q: what’s the equation?
A: H(x,y) = I(x+u, y+v)
H(x,y) - I(x+u,v+y) = 0
small motion: (u and v are less than 1 pixel)
suppose we take the Taylor series expansion of I:

41. Optical Flow Constraints
Let’s look at these constraints more closely
brightness constancy: Q: what’s the equation?
A: H(x,y) = I(x+u, y+v)
H(x,y) - I(x+u,v+y) = 0
small motion: (u and v are less than 1 pixel)
suppose we take the Taylor series expansion of I:

42. Optical Flow Equation
Combining these two equations

43. Optical Flow Equation
Combining these two equations

44. Optical Flow Equation
Combining these two equations

45. Optical Flow Equation
Combining these two equations

46. Optical Flow Equation Combining these two equations
In the limit as u and v go to zero, this becomes exact

47. Optical Flow Equation How many unknowns and equations per pixel?
A: u and v are unknown, 1 equation

48. Optical Flow Equation How many unknowns and equations per pixel?
Intuitively, what does this constraint mean?
A: u and v are unknown, 1 equation

49. Optical Flow Equation How many unknowns and equations per pixel?
Intuitively, what does this constraint mean?
The component of the flow in the gradient direction is determined
The component of the flow parallel to an edge is unknown
A: u and v are unknown, 1 equation

50. Optical Flow Equation How many unknowns and equations per pixel?
Intuitively, what does this constraint mean?
The component of the flow in the gradient direction is determined
The component of the flow parallel to an edge is unknown
A: u and v are unknown, 1 equation
This explains the Barber Pole illusion

51. Optical Flow Equation How many unknowns and equations per pixel?
Intuitively, what does this constraint mean?
The component of the flow in the gradient direction is determined
The component of the flow parallel to an edge is unknown
A: u and v are unknown, 1 equation
This explains the Barber Pole illusion

52. Aperture Problem

53. Aperture Problem Stripes moved upwards 6 pixels
Stripes moved left 5 pixels

54. Solving the Aperture Problem
How to get more equations for a pixel?
Basic idea: impose additional constraints
most common is to assume that the flow field is smooth locally
one method: pretend the pixel’s neighbors have the same (u,v)
If we use a 5x5 window, that gives us 25 equations per pixel!

55. RGB Version How to get more equations for a pixel?
Basic idea: impose additional constraints
most common is to assume that the flow field is smooth locally
one method: pretend the pixel’s neighbors have the same (u,v)
If we use a 5x5 window, that gives us 25 equations per pixel!

56. Lukas-Kanade Flow
Prob: we have more equations than unknowns

57. Lukas-Kanade Flow Prob: we have more equations than unknowns
Solution: solve least squares problem

58. Lukas-Kanade Flow Prob: we have more equations than unknowns
Solution: solve least squares problem
minimum least squares solution given by solution (in d) of:

59. Lukas-Kanade Flow
The summations are over all pixels in the K x K window
This technique was first proposed by Lukas & Kanade (1981)

60. Lukas-Kanade Flow When is this Solvable? ATA should be invertible
ATA should not be too small due to noise
eigenvalues l1 and l2 of ATA should not be too small
ATA should be well-conditioned
l1/ l2 should not be too large (l1 = larger eigenvalue)

61. Edge
Bad for motion estimation
- large l1, small l2

62. Low Texture Region Bad for motion estimation:
- gradients have small magnitude
- small l1, small l2

63. High Textured Region Good for motion estimation:
- gradients are different, large magnitudes
- large l1, large l2

64. Observation This is a two image problem BUT
Can measure sensitivity by just looking at one of the images!
This tells us which pixels are easy to track, which are hard
very useful later on when we do feature tracking...

65. Errors in Lucas-Kanade
What are the potential causes of errors in this procedure?
Suppose ATA is easily invertible
Suppose there is not much noise in the image

66. Errors in Lucas-Kanade
What are the potential causes of errors in this procedure?
Suppose ATA is easily invertible
Suppose there is not much noise in the image
When our assumptions are violated
Brightness constancy is not satisfied
The motion is not small
A point does not move like its neighbors
window size is too large
what is the ideal window size?

67. Iterative Refinement Iterative Lukas-Kanade Algorithm
Estimate velocity at each pixel by solving Lucas-Kanade equations
Warp H towards I using the estimated flow field
- use image warping techniques
Repeat until convergence

68. Revisiting the Small Motion Assumption
Is this motion small enough?
Probably not—it’s much larger than one pixel (2nd order terms dominate)
How might we solve this problem?

69. Reduce the Resolution!

70. Coarse-to-fine Optical Flow Estimation
Gaussian pyramid of image H
Gaussian pyramid of image I
image I
image H
u=10 pixels
u=5 pixels
u=2.5 pixels
u=1.25 pixels
image H
image I

71. Coarse-to-fine Optical Flow Estimation
Gaussian pyramid of image H
Gaussian pyramid of image I
image I
image H
run iterative L-K
warp & upsample
run iterative L-K .
image J
image I

72. Beyond Translation So far, our patch can only translate in (u,v)
What about other motion models?
rotation, affine, perspective

73. Warping Function
w(x;p) describes the geometric relationship between two images: x
Input Image I(x)
Template Image H(x)

74. Warping Functions
Translation:
Affine:
Perspective:

75. Image Registration
Find the warping parameter p that minimizes the intensity difference between template image and the warped input image

76. Image Registration
Find the warping parameter p that minimizes the intensity difference between template image and the warped input image
Mathematically, we can formulate this as an optimization problem:

77. Image Registration
Find the warping parameter p that minimizes the intensity difference between template image and the warped input image
Mathematically, we can formulate this as an optimization problem:
The above problem can be solved by many optimization algorithm:
- Steepest descent
- Gauss-newton
- Levenberg-marquardt, etc. 

78. Image Registration
Find the warping parameter p that minimizes the intensity difference between template image and the warped input image
Mathematically, we can formulate this as an optimization problem:
The above problem can be solved by many optimization algorithm:
- Steepest descent
- Gauss-newton
- Levenberg-marquardt, etc

79. Image Registration
Mathematically, we can formulate this as an optimization problem:

80. Image Registration
Mathematically, we can formulate this as an optimization problem:
Similar to optical flow:
Taylor series expansion

81. Image Registration
Mathematically, we can formulate this as an optimization problem:
Similar to optical flow:
Taylor series expansion

82. Image Registration
Mathematically, we can formulate this as an optimization problem:
Similar to optical flow:
Taylor series expansion
Image gradient

83. Image Registration
Mathematically, we can formulate this as an optimization problem:
Similar to optical flow:
Taylor series expansion
translation
affine
Image gradient
Jacobian matrix ……

84. Gauss-newton Optimization
- This is a quadratic function of Δp
- Reaching the minimum when partial derivative reaches zero

85. Gauss-newton Optimization
- This is a quadratic function of Δp
- Reaching the minimum when partial derivative reaches zero

86. Gauss-newton Optimization
- This is a quadratic function of Δp
- Reaching the minimum when partial derivative reaches zero
Rearrange

87. Gauss-newton Optimization
- This is a quadratic function of Δp
- Reaching the minimum when partial derivative reaches zero
Rearrange A Δp = b
Linear equation

88. Gauss-newton Optimization
- This is a quadratic function of Δp
- Reaching the minimum when partial derivative reaches zero
Rearrange
A-1 b

89. Lucas-Kanade Registration
Initialize p=p0:
Iterate:
1. Warp I with w(x;p) to compute I(w(x;p))

90. Lucas-Kanade Registration
Initialize p=p0:
Iterate:
1. Warp I with w(x;p) to compute I(w(x;p))
2. Compute the error image

91. Lucas-Kanade Registration
Initialize p=p0:
Iterate:
1. Warp I with w(x;p) to compute I(w(x;p))
2. Compute the error image
3. Warp the gradient with w(x;p)

92. Lucas-Kanade Registration
Initialize p=p0:
Iterate:
1. Warp I with w(x;p) to compute I(w(x;p))
2. Compute the error image
3. Warp the gradient with w(x;p)
4. Evaluate the Jacobian at (x;p)

93. Lucas-Kanade Registration
Initialize p=p0:
Iterate:
1. Warp I with w(x;p) to compute I(w(x;p))
2. Compute the error image
3. Warp the gradient with w(x;p)
4. Evaluate the Jacobian at (x;p)
5. Compute the

94. Lucas-Kanade Registration
Initialize p=p0:
Iterate:
1. Warp I with w(x;p) to compute I(w(x;p))
2. Compute the error image
3. Warp the gradient with w(x;p)
4. Evaluate the Jacobian at (x;p)
5. Compute the
6. Update the parameters

95. Lucas-Kanade Algorithm
Iteration 1:
H(x)
I(w(x;p))
H(x)-I(w(x;p))

96. Lucas-Kanade Algorithm
Iteration 2:
H(x)
I(w(x;p))
H(x)-I(w(x;p))

97. Lucas-Kanade Algorithm
Iteration 3:
H(x)
I(w(x;p))
H(x)-I(w(x;p))

98. Lucas-Kanade Algorithm
Iteration 4:
H(x)
I(w(x;p))
H(x)-I(w(x;p))

99. Lucas-Kanade Algorithm
Iteration 5:
H(x)
I(w(x;p))
H(x)-I(w(x;p))

100. Lucas-Kanade Algorithm
Iteration 6:
H(x)
I(w(x;p))
H(x)-I(w(x;p))

101. Lucas-Kanade Algorithm
Iteration 7:
H(x)
I(w(x;p))
H(x)-I(w(x;p))

102. Lucas-Kanade Algorithm
Iteration 8:
H(x)
I(w(x;p))
H(x)-I(w(x;p))

103. Lucas-Kanade Algorithm
Iteration 9:
H(x)
I(w(x;p))
H(x)-I(w(x;p))

104. Lucas-Kanade Algorithm
Final result:
H(x)
I(w(x;p))
H(x)-I(w(x;p))

105. Image Alignment

106. Lucas-Kanade for Image Alignment
Pros:
All pixels get used in matching
Can get sub-pixel accuracy (important for good mosaicing!)
Relatively fast and simple
Cons:
Prone to local minima
Relative small movement

107. Beyond 2D Registration
So far, we focus on registration between 2D images
The same idea can be used in registration between 3D and 2D

108. 3D-to-2D Registration
The transformation between 3D object and 2D images

109. Perspective projection
3D-to-2D Registration
From world coordinate to image coordinate
Viewport projection
Perspective projection
View transformation u sx u0 v
-sy v0 1 1

110. 3D-to-2D Registration
Similarly, we can formulate this as an optimization problem: