1. Algorithms & Applications in Computer Vision
Lihi Zelnik-Manor
Lecture 9: Optical Flow

2. Last Time: Alignment Homographies Rotational Panoramas RANSAC
Global alignment
Warping
Blending

3. Today: Motion and Flow Motion estimation
Patch-based motion (optic flow)
Regularization and line processes
Parametric (global) motion
Layered motion models

4. Optical Flow
Where did each pixel in image 1 go to in image 2

5. Optical Flow
Pierre Kornprobst's Demo

6. Introduction
Given a video sequence with camera/objects moving we can better understand the scene if we find the motions of the camera/objects.

7. Scene Interpretation How is the camera moving?
How many moving objects are there?
Which directions are they moving in?
How fast are they moving?
Can we recognize their type of motion (e.g. walking, running, etc.)?

8. Applications Recover camera ego-motion.
Result by MobilEye (

9. Applications Motion segmentation
Result by: L.Zelnik-Manor, M.Machline, M.Irani
“Multi-body Segmentation: Revisiting Motion Consistency” To appear, IJCV

10. Applications Structure from Motion Input Reconstructed shape
Result by: L. Zhang, B. Curless, A. Hertzmann, S.M. Seitz
“Shape and motion under varying illumination: Unifying structure from motion, photometric stereo, and multi-view stereo” ICCV’03

11. Examples of Motion fields
Forward motion
Rotation
Horizontal translation
Closer objects appear to move faster!!

12. Motion Field & Optical Flow Field
Motion Field = Real world 3D motion
Optical Flow Field = Projection of the motion field onto the 2d image
3D motion vector
2D optical flow vector
CCD

13. When does it break? The screen is stationary yet displays motion
Homogeneous objects generate zero optical flow.
Fixed sphere. Changing light source.
Non-rigid texture motion

14. The Optical Flow Field Still, in many cases it does work….
Goal: Find for each pixel a velocity vector which says:
How quickly is the pixel moving across the image
In which direction it is moving

15. How do we actually do that?

16. Estimating Optical Flow
Assume the image intensity is constant
Time = t
Time = t+dt

17. Brightness Constancy Equation
First order Taylor Expansion
Simplify notations:
Gradient and flow are orthogonal
Divide by dt and denote:
Problem I: One equation, two unknowns

18. Problem II: “The Aperture Problem”
For points on a line of fixed intensity we can only recover the normal flow
Time t
Time t+dt
Time t+dt ?
Where did the blue point move to?
We need additional constraints

19. Use Local Information
Sometimes enlarging the aperture can help

20. Local smoothness Lucas Kanade (1984)
Assume constant (u,v) in small neighborhood

21. Lucas Kanade (1984)
Goal: Minimize
Method: Least-Squares
2x2
2x1

22. How does Lucas-Kanade behave?
We want this matrix to be invertible.
i.e., no zero eigenvalues

23. How does Lucas-Kanade behave?
Edge  becomes singular

24. How does Lucas-Kanade behave?
Homogeneous   0 eigenvalues

25. How does Lucas-Kanade behave?
Textured regions  two high eigenvalues

26. How does Lucas-Kanade behave?
Edge  becomes singular
Homogeneous regions  low gradients
High texture 

27. Iterative Refinement
Estimate velocity at each pixel using one iteration of Lucas and Kanade estimation
Warp one image toward the other using the estimated flow field
(easier said than done)
Refine estimate by repeating the process

28. Optical Flow: Iterative Estimation
estimate update
Initial guess:
Estimate: x0 x

29. Optical Flow: Iterative Estimationx x0
estimate update
Initial guess:
Estimate:

30. Optical Flow: Iterative Estimationx x0
estimate update
Initial guess:
Estimate:
Initial guess:
Estimate:

31. Optical Flow: Iterative Estimationx0 x

32. Optical Flow: Iterative Estimation
Some Implementation Issues:
Warping is not easy (ensure that errors in warping are smaller than the estimate refinement)
Warp one image, take derivatives of the other so you don’t need to re-compute the gradient after each iteration.
Often useful to low-pass filter the images before motion estimation (for better derivative estimation, and linear approximations to image intensity)

33. Other break-downs Correlation based methods
Brightness constancy is not satisfied
A point does not move like its neighbors
what is the ideal window size?
The motion is not small (Taylor expansion doesn’t hold)
Aliasing
Correlation based methods
Regularization based methods
Use multi-scale estimation

34. Optical Flow: Aliasing
Temporal aliasing causes ambiguities in optical flow because images can have many pixels with the same intensity.
I.e., how do we know which ‘correspondence’ is correct?
actual shift
estimated shift
nearest match is correct (no aliasing)
nearest match is incorrect (aliasing)
To overcome aliasing: coarse-to-fine estimation.

35. Multi-Scale Flow Estimation
u=10 pixels
u=5 pixels
u=2.5 pixels
u=1.25 pixels
image It-1
image I
image It
image It+1
Gaussian pyramid of image It
Gaussian pyramid of image It+1

36. Multi-Scale Flow Estimation
run Lucas-Kanade
warp & upsample
run Lucas-Kanade .
image It-1
image I
image It
image It+1
Gaussian pyramid of image It
Gaussian pyramid of image It+1

37. Other break-downs Brightness constancy is not satisfied
A point does not move like its neighbors
what is the ideal window size?
The motion is not small (Taylor expansion doesn’t hold)
Aliasing
Correlation based methods
Regularization based methods
Use multi-scale estimation

38. Spatial Coherence Assumption
* Neighboring points in the scene typically belong to the same
surface and hence typically have similar motions.
* Since they also project to nearby points in the image, we expect
spatial coherence in image flow.
Black

39. Formalize this Idea
Noisy 1D signal: u x
Noisy measurements p(x)
Black

40. Regularization Find the “best fitting” smoothed function v(x) u q(x) x
Noisy measurements p(x)
Black

41. Membrane model Find the “best fitting” smoothed function v(x) u q(x) x
Black

42. Membrane model Find the “best fitting” smoothed function q(x) u q(x) x
Black

43. Membrane model Find the “best fitting” smoothed function q(x) u q(x)
Black

44. Regularization u q(x) x Spatial smoothness Minimize: assumption
Faithful to the data
Black

45. Bayesian Interpretation
Black

46. Discontinuities u q(x) x What about this discontinuity?
What is happening here?
What can we do?
Black

47. Robust Estimation
Noise distributions are often non-Gaussian, having much heavier tails. Noise samples from the tails are called outliers.
Sources of outliers (multiple motions):
specularities / highlights
jpeg artifacts / interlacing / motion blur
multiple motions (occlusion boundaries, transparency)
velocity space u1 u2 +
Black

48. Occlusion occlusion disocclusion shear
Multiple motions within a finite region.
Black

49. Coherent Motion
Possibly Gaussian.
Black

50. Multiple Motions
Definitely not Gaussian.
Black

51. Weak membrane model u
v(x) x
Black

52. Analog line process
Penalty function
Family of quadratics
Black

53. Analog line process Infimum defines a robust error function.
Minima are the same:
Black

54. Least Squares Estimation
Standard Least Squares Estimation allows too much influence for outlying points

55. Robust Regularization
v(x) x
Treat large spatial derivatives as outliers.
Minimize:
Black

56. Robust Estimation
Problem: Least-squares estimators penalize deviations between data & model with quadratic error fn (extremely sensitive to outliers)
error penalty function
influence function
Redescending error functions (e.g., Geman-McClure) help to reduce the influence of outlying measurements.
error penalty function
influence function
Black

57. Regularization Horn and Schunk (1981)
Add global smoothness term
Smoothness error:
Error in brightness constancy equation
Minimize:
Solve by calculus of variations

58. Robust Estimation Black & Anandan (1993)
Regularization can over-smooth across edges
Use “smarter” regularization

59. Robust Estimation Black & Anandan (1993)
Regularization can over-smooth across edges
Use “smarter” regularization
Minimize:
Brightness constancy
Smoothness

60. Optimization Gradient descent Coarse-to-fine (pyramid)
Deterministic annealing
Black

61. Example

62. Multi-resolution registration

63. Optical Flow Results

64. Optical Flow Results

65. Parametric motion estimation

66. Global (parametric) motion models
2D Models:
Affine
Quadratic
Planar projective transform (Homography)
3D Models:
Instantaneous camera motion models
Homography+epipole
Plane+Parallax
Szeliski

67. Motion models Translation 2 unknowns Affine 6 unknowns Perspective
3D rotation
3 unknowns
Szeliski

68. Example: Affine Motion
For panning camera or planar surfaces:
Only 6 parameters to solve for Better results
Least Square Minimization (over all pixels): [ ] 2 å = a
Err ) ( t y x I + p r

69. Last lecture: Alignment / motion warping
“Alignment”: Assuming we know the correspondences, how do we get the transformation?
Expressed in terms of absolute coordinates of corresponding points…
Generally presumed features separately detected in each frame
e.g., affine model in abs. coords…

70. Today: “flow”, “parametric motion”
Two views presumed in temporal sequence…track or analyze spatio-temporal gradient
Sparse or dense in first frame
Search in second frame
Motion models expressed in terms of position change

71. Today: “flow”, “parametric motion”
Two views presumed in temporal sequence…track or analyze spatio-temporal gradient
Sparse or dense in first frame
Search in second frame
Motion models expressed in terms of position change

72. Today: “flow”, “parametric motion”
Two views presumed in temporal sequence…track or analyze spatio-temporal gradient
Sparse or dense in first frame
Search in second frame
Motion models expressed in terms of position change

73. Today: “flow”, “parametric motion”
Two views presumed in temporal sequence…track or analyze spatio-temporal gradient
(ui,vi)
Sparse or dense in first frame
Search in second frame
Motion models expressed in terms of position change

74. Today: “flow”, “parametric motion”
Two views presumed in temporal sequence…track or analyze spatio-temporal gradient
(ui,vi)
Sparse or dense in first frame
Search in second frame
Motion models expressed in terms of position change

75. Today: “flow”, “parametric motion”
Two views presumed in temporal sequence…track or analyze spatio-temporal gradient
Previous Alignment model:
(ui,vi)
Now, Displacement model:
Sparse or dense in first frame
Search in second frame
Motion models expressed in terms of position change

76. Other 2D Motion Models
Quadratic – instantaneous approximation to planar motion
Projective – exact planar motion
Szeliski

77. 3D Motion Models Instantaneous camera motion: Homography+Epipole
Local Parameter:
Instantaneous camera motion:
Global parameters:
Global parameters:
Homography+Epipole
Local Parameter:
Residual Planar Parallax Motion
Global parameters:
Local Parameter:

78. Segmentation of Affine Motion= +
Input
Segmentation result
Result by: L.Zelnik-Manor, M.Irani
“Multi-frame estimation of planar motion”, PAMI 2000

79. Panoramas
Input
Motion estimation by Andrew Zisserman’s group

80. Stabilization Result by: L.Zelnik-Manor, M.Irani
“Multi-frame estimation of planar motion”, PAMI 2000

81. Discrete Search vs. Gradient Based
Consider image I translated by
The discrete search method simply searches for the best estimate.
The gradient method linearizes the intensity function and solves for the estimate
Szeliski

82. Correlation and SSD For larger displacements, do template matching
Define a small area around a pixel as the template
Match the template against each pixel within a search area in next image.
Use a match measure such as correlation, normalized correlation, or sum-of-squares difference
Choose the maximum (or minimum) as the match
Sub-pixel estimate (Lucas-Kanade)
Szeliski

83. Shi-Tomasi feature tracker
Find good features (min eigenvalue of 22 Hessian)
Use Lucas-Kanade to track with pure translation
Use affine registration with first feature patch
Terminate tracks whose dissimilarity gets too large
Start new tracks when needed
Szeliski

84. Tracking result

85. How well do these techniques work?

86. A Database and Evaluation Methodology for Optical Flow
Simon Baker, Daniel Scharstein, J.P Lewis, Stefan Roth, Michael Black, and Richard Szeliski
ICCV 2007

87. Limitations of Yosemite
Only sequence used for quantitative evaluation
Limitations:
Very simple and synthetic
Small, rigid motion
Minimal motion discontinuities/occlusions
Image 7
Image 8
Yosemite
Ground-Truth Flow
Flow Color
Coding
Szeliski

88. Limitations of Yosemite
Only sequence used for quantitative evaluation
Current challenges:
Non-rigid motion
Real sensor noise
Complex natural scenes
Motion discontinuities
Need more challenging and more realistic benchmarks
Image 7
Image 8
Yosemite
Ground-Truth Flow
Flow Color
Coding
Szeliski

89. Realistic synthetic imagery
Randomly generate scenes with “trees” and “rocks”
Significant occlusions, motion, texture, and blur
Rendered using Mental Ray and “lens shader” plugin
Rock
Grove
Motion estimation
Szeliski

90. Modified stereo imagery
Recrop and resample ground-truth stereo datasets to have appropriate motion for OF
Venus
Moebius
Szeliski

91. Dense flow with hidden texture
Paint scene with textured fluorescent paint
Take 2 images: One in visible light, one in UV light
Move scene in very small steps using robot
Generate ground-truth by tracking the UV images
Setup
Visible UV
Lights
Image
Cropped
Szeliski

92. Experimental results http://vision.middlebury.edu/flow/
Motion estimation
Szeliski

93. Conclusions
Difficulty: Data substantially more challenging than Yosemite
Diversity: Substantial variation in difficulty across the various datasets
Motion GT vs Interpolation: Best algorithms for one are not the best for the other
Comparison with Stereo: Performance of existing flow algorithms appears weak
Szeliski

94. Layered Scene Representations

95. Motion representations
How can we describe this scene?
Szeliski

96. Block-based motion prediction
Break image up into square blocks
Estimate translation for each block
Use this to predict next frame, code difference (MPEG-2)
Szeliski

97. Layered motion Break image sequence up into “layers”:  =
 =
Describe each layer’s motion
Szeliski

98. Layered Representation
For scenes with multiple affine motions
Estimate dominant motion parameters
Reject pixels which do not fit
Convergence
Restart on remaining pixels

99. Layered motion Advantages: can represent occlusions / disocclusions
each layer’s motion can be smooth
video segmentation for semantic processing
Difficulties:
how do we determine the correct number?
how do we assign pixels?
how do we model the motion?
Szeliski

100. How do we form them?
Szeliski

101. How do we estimate the layers?
compute coarse-to-fine flow
estimate affine motion in blocks (regression)
cluster with k-means
assign pixels to best fitting affine region
re-estimate affine motions in each region…
Szeliski

102. Layer synthesis For each layer:
stabilize the sequence with the affine motion
compute median value at each pixel
Determine occlusion relationships
Szeliski

103. Results
Szeliski

104. Recent results: SIFT Flow

105. Recent results: SIFT Flow

106. Recent GPU Implementation
Real time flow exploiting robust norm + regularized mapping

107. Today: Motion and Flow Motion estimation
Patch-based motion (optic flow)
Regularization and line processes
Parametric (global) motion
Layered motion models

108. Slide Credits
Trevor Darrell
Rick Szeliski
Michael Black