1. Motion Computing in Image Analysis
- Mani V Thomas
CISC 489/689

2. Roadmap Optic Flow Constraint Optic Flow Computation
Gradient Based Approach
Feature Based Approach
Estimation Criterion
Block Matching algorithms
Conclusion
Some slides and illustrations are from M. Pollefeys and M. Shah

3. Importance of Visual Motion
Apparent motion of objects on the image plane is a strong cue to understand structure and 3D motion
Biological visual systems infer properties of the 3D world via motion
Two sub-problems of motion
Problem of correspondence estimation
Which elements of a frame correspond to which elements of the next frame
Problem of reconstruction
Given the correspondence and the camera’s intrinsic parameters can we infer 3D motion and/or structure
Courtesy: E. Trucco and A. Verri, “Introductory techniques for 3D Computer Vision”

4. Apparent Motion Apparent motion of objects on the image plane
Caution required!!
Consider a perfectly uniform sphere that is rotating but no change in the light direction
Optic flow is zero
Perfectly uniform sphere that is stationary but the light is changing
Optic flow exists
Hope – apparent motion is very close to the actual motion
Courtesy: E. Trucco and A. Verri, “Introductory techniques for 3D Computer Vision”

5. Optic Flow Computation
Two strategies for computing motion
Differential Methods
Spatio temporal derivatives for estimation of flow at every position
Multi-scale analysis required if motion not constrained within a small range
Dense flow measurements
Matching Methods
Feature extraction(Image edges, corners)
Feature/Block Matching and error minimization
Sparse flow measurements
Courtesy: E. Trucco and A. Verri, “Introductory techniques for 3D Computer Vision”

6. Optic Flow Computation
Image Brightness Constancy assumption
Let E be the image intensity as captured by the camera
Using Taylor series to expand E
Apparent brightness of moving objects remains constant

7. Optic Flow Computation
Image Brightness Constancy assumption
Apparent brightness of moving objects remains constant
The are the image gradient while the are the components of the motion field
Courtesy: E. Trucco and A. Verri, “Introductory techniques for 3D Computer Vision”

8. Aperture Problem We can measure
Terms that can be measured
Terms to be computed
Number of equations - 1
The component of the motion field that is orthogonal to the spatial image gradient is not constrained by the image brightness constancy assumption
Intuitively
The component of the flow in the gradient direction is determined
The component of the flow parallel to an edge is unknown
Courtesy: E. Trucco and A. Verri, “Introductory techniques for 3D Computer Vision”

10. Different physical motion but same measurable motion within a fixed window

11. Roadmap Optic Flow Constraint Optic Flow Computation
Gradient Based Approach
Feature Based Approach
Estimation Criterion
Block Matching algorithms
Conclusion
Some slides and illustrations are from M. Pollefeys and M. Shah

12. Optic Flow Constraint How to get more equations for a pixel?
Basic idea: impose additional constraints
Most common is to assume that the flow field is smooth locally
One method: pretend the pixel’s neighbors have the same (u,v)
If we use a 5x5 window, that gives us 25 equations per pixel!

13. Lucas-Kanade Optic Flow
We now have more equations than unknowns
Solve the least squares problem
Minimum least squares solution (in d) is given by
First proposed by Lucas-Kanade in 1981
Summation performed over all the pixels in the window

14. Lucas-Kanade Optic Flow
When is the Lucas-Kanade equations solvable
ATA should be invertible
ATA should not be too small (effects of noise)
Eigenvalues of ATA, 1 and 2 should not be small
ATA should be well conditioned
1/2 should not be large (1 = larger eigenvalue)

15. Edge
Gradient is large in magnitude
Large 1 but small 2

16. Low texture region
Gradients has small magnitude
Small 1 and small 2

17. High texture region Gradients are different with large magnitudes
Large 1 and large 2

18. Improving the Lucas-Kanade method
When our assumptions are violated
Brightness constancy is not satisfied
The motion is not small
A point does not move like its neighbors
Iterative Lucas-Kanade Algorithm
Estimate velocity at each pixel by solving Lucas-Kanade equations
Warp H towards I using the estimated flow field
use image warping techniques
Repeat until convergence

19. Iterative Lucas-Kanade method
Gaussian pyramid of image H
Gaussian pyramid of image I
image I
image H
u=10 pixels
u=5 pixels
u=2.5 pixels
u=1.25 pixels
image H
image I

20. Iterative Lucas-Kanade method
Gaussian pyramid of image H
Gaussian pyramid of image I
image I
image H
run iterative L-K
warp & upsample
run iterative L-K .
image J
image I

21. Roadmap Optic Flow Constraint Optic Flow Computation
Gradient Based Approach
Feature Based Approach
Estimation Criterion
Block Matching algorithms
Conclusion
Some slides and illustrations are from M. Pollefeys and M. Shah

22. Feature Based Method Feature Extraction
Maxima in first derivative of the Image
Local peak in the first derivative
Numerical Approximation
Compute the motion parameters from the best bipartite graph
Correspondence between the feature points in one image with those in the other
For more information: Ramesh Jain, Rangachar Kasturi, Brian Schunck: Machine Vision 1995 ( )

23. Roadmap Optic Flow Constraint Optic Flow Computation
Gradient Based Approach
Feature Based Approach
Estimation Criterion
Block Matching algorithms
Conclusion
Some slides and illustrations are from M. Pollefeys and M. Shah

24. Estimation Criterion Pixel domain Criterion Frequency Domain Criterion
MAE/MSE
Lorentzian
Correlation
Frequency Domain Criterion
Cross Correlation
Phase Correlation

25. Estimation Criterion(contd.)
Pixel Domain Criterion
Estimation criterion aim at minimizing
prediction error is sensitive to noise if number of pixels is not large or if region is poorly textured
Common choice of estimation criterion
Quadratic function is not good since a single large error can bias the estimate of the field
Absolute value function is better than the quadratic since cost grows linearly with error
Does not require multiplications and is better suited for real-time video encoders

26. Estimation Criterion(contd.)
A more robust criterion is based on the Lorentzian function
Grows slower than |x| for larger errors
Similarity measure using Correlation
Computationally complex because of the multiplications
This criterion requires maximization
Usually the normalized Cross correlation is computed
For more details: M. Black, “Robust Incremental Optical Flow”

27. Estimation Criterion(contd.)

28. Estimation Criterion(contd.)
Frequency Domain Criterion
Amplitudes of both the FT are independent of z
Argument difference depends linearly on translation
Global motion is recovered by evaluating the phase difference over a number of frequencies and solving the resulting system of equations
In practice, this method will work only for a single object moving across a uniform background

29. Estimation Criterion(contd.)
Phase Correlation
In the case of a single global translation, the correlation surface becomes a Kronecker delta function
In practice, there are numerous peaks which correspond to the dominant displacements between the two images
The locations are relatively independent to illumination changes

30. Roadmap Optic Flow Constraint Optic Flow Computation
Gradient Based Approach
Feature Based Approach
Estimation Criterion
Block Matching algorithms
Conclusion
Some slides and illustrations are from M. Pollefeys and M. Shah

31. Block Matching Algorithms
Sparse motion measurements
Motion is spatially constant and temporally linear over a rectangular region of support
The minimization problem is
is an M x N block of pixels with the top-left corner co-ordinate at

32. Block Matching Algorithms(contd.)p -p
Current Picture
Reference Frame
(x,y)
(x+u,y+v) u v

33. Block Matching Algorithms(contd.)
Principle of Locality of Reference
Block Matching algorithms
Exhaustive Search
Always finds the “deepest” minimum
Computationally very expensive
If I x J is the picture resolution and rate is F fps the overall operations in comparing MxN blocks would be
This corresponds to GOPS for p=15 at 30fps for a 720x480 image (3 operations per pixel of one subtraction, one absolute value and one addition)

34. Block Matching Algorithms(contd.)
Logarithmic Search
Sub-optimal and may get trapped in a local minima
Computationally feasible for real-time video encoders
Search Method
Divide the search space at [-p/2, -p/2]
Search at (0,0) and at 8 major points at the perimeter of the rectangle at [-p/2, -p/2]
Using best match position as starting point, search in the eight perimeter points at the half distance window
If I x J is the picture resolution and rate is F fps the overall operations in comparing MxN blocks would be
This corresponds to 1.03 GOPS for p=15 at 30fps for a 720x480 image
For more information refer the work by Dr. Lai-Man Po and C. K. Cheung (

35. Block Matching Algorithms(contd.)
Hierarchical Search
Sub-optimal for regions containing detail and increased storage requirements
Computationally feasible for real-time video encoders
Search method
Form several low resolution images by low pass filtering
At the lowest resolution perform a sub-optimal search like log search
Propagate search vectors to higher resolution images and perform search
If I x J is the picture resolution and rate is F fps the overall operations in comparing MxN blocks would be
This corresponds to MOPS for p=15 at 30fps for a 720x480 image

36. Conclusion Motion estimation Aperture problem
Different algorithms to perform motion analysis
Lucas-Kanade algorithm
Estimation criterion for motion field computation
Block Matching Algorithms
Computational complexity of motion analysis