1. Estimating Parametric and Layered Motion
CS 636 Computer Vision
Estimating Parametric and Layered Motion
Nathan Jacobs

2. Overview
Motion fields and parallax
Parametric Motion
Layered Motion

3. Motion field and parallax
X(t) is a moving 3D point
Velocity of scene point: V = dX/dt
x(t) = (x(t),y(t)) is the projection of X in the image
Apparent velocity v in the image: given by components vx = dx/dt and vy = dy/dt
These components are known as the motion field of the image
X(t+dt) V
X(t) v
x(t+dt)
x(t)
Lazebnik

4. Examples of Motion Fields
(a) (b)
(a) Motion field of a pilot looking straight ahead while approaching a fixed point on a landing strip. (b) Pilot is looking to the right in level flight.

5. Examples of Motion Fields
(a) (b)
(c) (d)
(a) Translation perpendicular to a surface. (b) Rotation about axis perpendicular to image plane. (c) Translation parallel to a surface at a constant distance. (d) Translation parallel to an obstacle in front of a more distant background.

6. Motion field and parallax
X(t+dt)
To find image velocity v, differentiate x=(x,y) with respect to t (using quotient rule): V
X(t) v
x(t+dt)
Quotient rule for differentiation: D(f/g) = (g f’ – g’ f)/g^2
x(t)
Image motion is a function of both the 3D motion (V) and the depth of the 3D point (Z)
Lazebnik

7. Motion field and parallax
Pure translation camera motion: V is constant everywhere
Lazebnik

8. Motion field and parallax
Pure translation: V is constant everywhere
The length of the motion vectors is inversely proportional to the depth Z
if Vz is nonzero:
Every motion vector points toward (or away from) the vanishing point of the translation direction
At the vanishing point of the motion location, the visual motion is zero
We have v = 0 when V_z x = v_0 or x = (f V_x / V_z, f V_y / V_z)
Lazebnik

9. Motion field and parallax
Pure translation: V is constant everywhere
The length of the motion vectors is inversely proportional to the depth Z
Vz is nonzero:
Every motion vector points toward (or away from) the vanishing point of the translation direction
Vz is zero:
Motion is parallel to the image plane, all the motion vectors are parallel
Lazebnik

10. Why estimate motion? Lots of uses Track object behavior
Correct for camera jitter (stabilization)
Align images (mosaics)
3D shape reconstruction
Special effects
1998 Oscar for Visual Effects

11. Types of motion estimation methods
Input: sequence of images
Output: point correspondences
Benchmark:
Types:
Correspondence based methods:
E.g. SIFT matching
Feature tracking:
E.g. Lucas-Kanade tracking of harris corners
Optical Flow:
Like feature tracking, but use all pixels
Parametric flow:
Track all pixels, but use a global motion model

12. feature-based vs. dense
dense (simple translational motion)
extract features
extract features
minimize residual
minimize residual
optimal motion estimate
optimal motion estimate

13. What is the difference? feature-based
dense (simple translational motion)

14. Discussion: features vs. flow?
Features are better for:
Flow is better for:

15. The brightness constancy constraint
How many equations and unknowns per pixel?
One equation, two unknowns
Intuitively, what does this constraint mean?
The component of the flow perpendicular to the gradient (i.e., parallel to the edge) is unknown
A: u and v are unknown, 1 equation
gradient
(u,v)
If (u, v) satisfies the equation, so does (u+u’, v+v’) if
(u’,v’)
(u+u’,v+v’)
edge

16. Lucas-Kanade flow How to get more equations for a pixel?
Basic idea: impose additional constraints
most common is to assume that the flow field is smooth locally
one method: pretend the pixel’s neighbors have the same (u,v)
If we use a 5x5 window, that gives us 25 equations per pixel!

17. Iterative Refinement Iterative Lucas-Kanade Algorithm
Estimate velocity at each pixel by solving Lucas-Kanade equations
Warp H towards I using the estimated flow field
- use image warping techniques
Repeat until convergence

18. Optical flow: iterative estimation
estimate update
Initial guess:
Estimate: x0 x
(using d for displacement here instead of u)

19. Optical flow: iterative estimationx x0
estimate update
Initial guess:
Estimate:

20. Optical flow: iterative estimationx x0
estimate update
Initial guess:
Estimate:

21. Optical flow: iterative estimationx0 x

22. Motion models Translation 2 unknowns Affine 6 unknowns Perspective
3D rotation
3 unknowns

23. Revisiting: translational motion
Substituting into the B.C. equation
Each pixel provides 1 linear constraint in 2 unknowns (this can be for a local region, or the entire image!)
Least square minimization (over all pixels):

24. Solving for parameters
A least squares problem:
With following gradients:
Since f is linear

25. Solving for parameters
Since f is linear
In this case, the terms are Ix and Iy.
Note: this can be minimized for arbitrary image regions

26. Example: affine motion
Substituting into the B.C. equation
Each pixel provides 1 linear constraint in 6 global unknowns
Least square minimization (over all pixels):

27. Parametric motion
i.e., the product of image gradient with Jacobian of the correspondence field

28. Parametric motion
the expressions inside the brackets are the same as the cases for simpler translation motion case

29. Other 2D Motion Models
From Szeliski book

30. Generalizing Lucas-Kanade
extends to all differentiable warp functions!
Lucas-Kanade 20 Years On: A Unifying Framework, Simon Baker and Iain Matthews

32. image approximations to explore linear to matlab…
imSteepest = imJacobian, cat(4, gx, gy)), 4); tmpJacobian = scale2rgb(repmat(reshape(imJacobian, [sz 1 Np*2]), [ ])); tmpSteepest = scale2rgb(repmat(reshape(imSteepest, [sz 1 Np]), [ ])); figure(1); clf; montage(tmpJacobian, 'Size', [2 Np]) title({'Jacobian Images: change in the warp field w.r.t. parameters'}) figure(2); clf; montage(tmpSteepest, 'Size', [1 Np]) title('Steepest Descent Images: inner product with image to update parameters') [Tx Ty] = ndgrid(linspace(-1, 1, 20), linspace(-1,1,20)); p = randn(100, Np); p = p, [ ]); for ix = 1:size(p,1) % make the approximation using the steepest descent images (the first % order approximation of the image) figure(3); clf; subplot(121); imagesc(im + imSteepest,reshape(p(ix,:), [1 1 Np])),3), [0 1]); axis image off; title(p(ix,:)) subplot(122) imagesc(im + imSteepest,reshape(p(ix,:), [1 1 Np])),3), [0 1]); axis image off; xlim([40 100]), ylim([ ]); colormap gray pause(eps) end
image approximations
to explore linear
to matlab…
% load data im = im2double(rgb2gray(imread('peppers.png'))); [gx gy] = gradient(im); sz = size(im); [X Y] = meshgrid((1:sz(2)) -sz(2)/2, (1:sz(1)) -sz(1)/2); motionModel = 'translation'; motionModel = 'affine'; % make jacobian images % i x j x Np x 2 switch motionModel case 'translation’ Np = 2; imJacobian = zeros([sz Np 2]); imJacobian(:,:,1,1) = 1; imJacobian(:,:,2,2) = 1; case 'affine’ Np = 6; imJacobian(:,:,1,1) = 1; imJacobian(:,:,2,2) = 1; imJacobian(:,:,3,1) = X; imJacobian(:,:,4,1) = Y; imJacobian(:,:,5,2) = X; imJacobian(:,:,6,2) = Y; end

33. Motion representations
Is there something between global parametric motion and optical flow?

34. Block-based motion prediction
Break image up into square blocks
Estimate translation for each block
Use this to predict next frame, code difference (MPEG-2)

35. Layered motion Break image sequence up into “layers”:   =
  =
Then describe motion of each layer

36. Layered motion Advantages: Difficulties:
can represent occlusions / dis-occlusions
each layer’s motion can be smooth
video segmentation for semantic processing
Difficulties:
how do we determine the correct number?
how do we assign pixels?
how do we model the motion?

37. Layers for video summarization

38. Background modeling (MPEG-4)
Convert masked images into a background sprite for layered video coding =

39. What are layers? Layers or Sprites Intensities Velocities
Alpha matting
Intensities
Velocities
Wang and Adelson, “Representing moving images with layers”, IEEE Transactions on Image Processing, 1994

40. How do we form them?

41. How do we form them?

42. How do we estimate the layers?
compute coarse-to-fine flow
estimate affine motion in blocks (regression)
cluster with k-means
assign pixels to best fitting affine region
re-estimate affine motions in each region…

43. Layer synthesis For each layer: Determine occlusion relationships
stabilize the sequence with the affine motion
compute median value at each pixel
Determine occlusion relationships

44. Results

45. Why is are layers useful?

46. summary parametric motion: generalized Lucas-Kanade
layered motion models