Estimating Parametric and Layered Motion CS 636 Computer Vision Estimating Parametric and Layered Motion Nathan Jacobs

Overview Motion fields and parallax Parametric Motion Layered Motion

Motion field and parallax X(t) is a moving 3D point Velocity of scene point: V = dX/dt x(t) = (x(t),y(t)) is the projection of X in the image Apparent velocity v in the image: given by components vx = dx/dt and vy = dy/dt These components are known as the motion field of the image X(t+dt) V X(t) v x(t+dt) x(t) Lazebnik

Examples of Motion Fields (a) (b) (a) Motion field of a pilot looking straight ahead while approaching a fixed point on a landing strip. (b) Pilot is looking to the right in level flight.

Examples of Motion Fields (a) (b) (c) (d) (a) Translation perpendicular to a surface. (b) Rotation about axis perpendicular to image plane. (c) Translation parallel to a surface at a constant distance. (d) Translation parallel to an obstacle in front of a more distant background.

Motion field and parallax X(t+dt) To find image velocity v, differentiate x=(x,y) with respect to t (using quotient rule): V X(t) v x(t+dt) Quotient rule for differentiation: D(f/g) = (g f’ – g’ f)/g^2 x(t) Image motion is a function of both the 3D motion (V) and the depth of the 3D point (Z) Lazebnik

Motion field and parallax Pure translation camera motion: V is constant everywhere Lazebnik

Motion field and parallax Pure translation: V is constant everywhere The length of the motion vectors is inversely proportional to the depth Z if Vz is nonzero: Every motion vector points toward (or away from) the vanishing point of the translation direction At the vanishing point of the motion location, the visual motion is zero We have v = 0 when V_z x = v_0 or x = (f V_x / V_z, f V_y / V_z) Lazebnik

Motion field and parallax Pure translation: V is constant everywhere The length of the motion vectors is inversely proportional to the depth Z Vz is nonzero: Every motion vector points toward (or away from) the vanishing point of the translation direction Vz is zero: Motion is parallel to the image plane, all the motion vectors are parallel Lazebnik

Why estimate motion? Lots of uses Track object behavior Correct for camera jitter (stabilization) Align images (mosaics) 3D shape reconstruction Special effects 1998 Oscar for Visual Effects http://www.youtube.com/watch?v=au0ibcWUxPI

Types of motion estimation methods Input: sequence of images Output: point correspondences Benchmark: http://vision.middlebury.edu/flow/ Types: Correspondence based methods: E.g. SIFT matching Feature tracking: E.g. Lucas-Kanade tracking of harris corners Optical Flow: Like feature tracking, but use all pixels Parametric flow: Track all pixels, but use a global motion model

feature-based vs. dense dense (simple translational motion) extract features extract features minimize residual minimize residual optimal motion estimate optimal motion estimate

What is the difference? feature-based dense (simple translational motion)

Discussion: features vs. flow? Features are better for: Flow is better for:

The brightness constancy constraint How many equations and unknowns per pixel? One equation, two unknowns Intuitively, what does this constraint mean? The component of the flow perpendicular to the gradient (i.e., parallel to the edge) is unknown A: u and v are unknown, 1 equation gradient (u,v) If (u, v) satisfies the equation, so does (u+u’, v+v’) if (u’,v’) (u+u’,v+v’) edge

Lucas-Kanade flow How to get more equations for a pixel? Basic idea: impose additional constraints most common is to assume that the flow field is smooth locally one method: pretend the pixel’s neighbors have the same (u,v) If we use a 5x5 window, that gives us 25 equations per pixel!

Iterative Refinement Iterative Lucas-Kanade Algorithm Estimate velocity at each pixel by solving Lucas-Kanade equations Warp H towards I using the estimated flow field - use image warping techniques Repeat until convergence

Optical flow: iterative estimation estimate update Initial guess: Estimate: x0 x (using d for displacement here instead of u)

Optical flow: iterative estimation x x0 estimate update Initial guess: Estimate:

Optical flow: iterative estimation x x0 estimate update Initial guess: Estimate:

Optical flow: iterative estimation x0 x

Motion models Translation 2 unknowns Affine 6 unknowns Perspective 3D rotation 3 unknowns

Revisiting: translational motion Substituting into the B.C. equation Each pixel provides 1 linear constraint in 2 unknowns (this can be for a local region, or the entire image!) Least square minimization (over all pixels):

Solving for parameters A least squares problem: With following gradients: Since f is linear http://en.wikipedia.org/wiki/Least_squares

Solving for parameters Since f is linear In this case, the terms are Ix and Iy. Note: this can be minimized for arbitrary image regions http://en.wikipedia.org/wiki/Least_squares

Example: affine motion Substituting into the B.C. equation Each pixel provides 1 linear constraint in 6 global unknowns Least square minimization (over all pixels):

Parametric motion i.e., the product of image gradient with Jacobian of the correspondence field

Parametric motion the expressions inside the brackets are the same as the cases for simpler translation motion case

Other 2D Motion Models From Szeliski book

Generalizing Lucas-Kanade extends to all differentiable warp functions! Lucas-Kanade 20 Years On: A Unifying Framework, Simon Baker and Iain Matthews

image approximations to explore linear to matlab… imSteepest = sum(bsxfun(@times, imJacobian, cat(4, gx, gy)), 4); tmpJacobian = scale2rgb(repmat(reshape(imJacobian, [sz 1 Np*2]), [1 1 3 1])); tmpSteepest = scale2rgb(repmat(reshape(imSteepest, [sz 1 Np]), [1 1 3 1])); figure(1); clf; montage(tmpJacobian, 'Size', [2 Np]) title({'Jacobian Images: change in the warp field w.r.t. parameters'}) figure(2); clf; montage(tmpSteepest, 'Size', [1 Np]) title('Steepest Descent Images: inner product with image to update parameters') [Tx Ty] = ndgrid(linspace(-1, 1, 20), linspace(-1,1,20)); p = randn(100, Np); p = bsxfun(@times, p, [1 1 .01 .01 .01 .01]); for ix = 1:size(p,1) % make the approximation using the steepest descent images (the first % order approximation of the image) figure(3); clf; subplot(121); imagesc(im + sum(bsxfun(@times, imSteepest,reshape(p(ix,:), [1 1 Np])),3), [0 1]); axis image off; title(p(ix,:)) subplot(122) imagesc(im + sum(bsxfun(@times, imSteepest,reshape(p(ix,:), [1 1 Np])),3), [0 1]); axis image off; xlim([40 100]), ylim([160 210]); colormap gray pause(eps) end image approximations to explore linear to matlab… % load data im = im2double(rgb2gray(imread('peppers.png'))); [gx gy] = gradient(im); sz = size(im); [X Y] = meshgrid((1:sz(2)) -sz(2)/2, (1:sz(1)) -sz(1)/2); motionModel = 'translation'; motionModel = 'affine'; % make jacobian images % i x j x Np x 2 switch motionModel case 'translation’ Np = 2; imJacobian = zeros([sz Np 2]); imJacobian(:,:,1,1) = 1; imJacobian(:,:,2,2) = 1; case 'affine’ Np = 6; imJacobian(:,:,1,1) = 1; imJacobian(:,:,2,2) = 1; imJacobian(:,:,3,1) = X; imJacobian(:,:,4,1) = Y; imJacobian(:,:,5,2) = X; imJacobian(:,:,6,2) = Y; end

Motion representations Is there something between global parametric motion and optical flow?

Block-based motion prediction Break image up into square blocks Estimate translation for each block Use this to predict next frame, code difference (MPEG-2)

Layered motion Break image sequence up into “layers”:   =   = Then describe motion of each layer

Layered motion Advantages: Difficulties: can represent occlusions / dis-occlusions each layer’s motion can be smooth video segmentation for semantic processing Difficulties: how do we determine the correct number? how do we assign pixels? how do we model the motion?

Layers for video summarization

Background modeling (MPEG-4) Convert masked images into a background sprite for layered video coding + + + + + + =

What are layers? Layers or Sprites Intensities Velocities Alpha matting Intensities Velocities Wang and Adelson, “Representing moving images with layers”, IEEE Transactions on Image Processing, 1994

How do we form them?

How do we form them?

How do we estimate the layers? compute coarse-to-fine flow estimate affine motion in blocks (regression) cluster with k-means assign pixels to best fitting affine region re-estimate affine motions in each region…

Layer synthesis For each layer: Determine occlusion relationships stabilize the sequence with the affine motion compute median value at each pixel Determine occlusion relationships

Results

Why is are layers useful?

summary parametric motion: generalized Lucas-Kanade layered motion models