Algorithms & Applications in Computer Vision Lihi Zelnik-Manor Lecture 9: Optical Flow

Last Time: Alignment Homographies Rotational Panoramas RANSAC Global alignment Warping Blending

Today: Motion and Flow Motion estimation Patch-based motion (optic flow) Regularization and line processes Parametric (global) motion Layered motion models

Optical Flow Where did each pixel in image 1 go to in image 2

Optical Flow Pierre Kornprobst's Demo

Introduction Given a video sequence with camera/objects moving we can better understand the scene if we find the motions of the camera/objects.

Scene Interpretation How is the camera moving? How many moving objects are there? Which directions are they moving in? How fast are they moving? Can we recognize their type of motion (e.g. walking, running, etc.)?

Applications Recover camera ego-motion. Result by MobilEye (www.mobileye.com)

Applications Motion segmentation Result by: L.Zelnik-Manor, M.Machline, M.Irani “Multi-body Segmentation: Revisiting Motion Consistency” To appear, IJCV

Applications Structure from Motion Input Reconstructed shape Result by: L. Zhang, B. Curless, A. Hertzmann, S.M. Seitz “Shape and motion under varying illumination: Unifying structure from motion, photometric stereo, and multi-view stereo” ICCV’03

Examples of Motion fields Forward motion Rotation Horizontal translation Closer objects appear to move faster!!

Motion Field & Optical Flow Field Motion Field = Real world 3D motion Optical Flow Field = Projection of the motion field onto the 2d image 3D motion vector 2D optical flow vector CCD

When does it break? The screen is stationary yet displays motion Homogeneous objects generate zero optical flow. Fixed sphere. Changing light source. Non-rigid texture motion

The Optical Flow Field Still, in many cases it does work…. Goal: Find for each pixel a velocity vector which says: How quickly is the pixel moving across the image In which direction it is moving

How do we actually do that?

Estimating Optical Flow Assume the image intensity is constant Time = t Time = t+dt

Brightness Constancy Equation First order Taylor Expansion Simplify notations: Gradient and flow are orthogonal Divide by dt and denote: Problem I: One equation, two unknowns

Problem II: “The Aperture Problem” For points on a line of fixed intensity we can only recover the normal flow Time t Time t+dt Time t+dt ? Where did the blue point move to? We need additional constraints

Use Local Information Sometimes enlarging the aperture can help

Local smoothness Lucas Kanade (1984) Assume constant (u,v) in small neighborhood

Lucas Kanade (1984) Goal: Minimize Method: Least-Squares 2x2 2x1

How does Lucas-Kanade behave? We want this matrix to be invertible. i.e., no zero eigenvalues

How does Lucas-Kanade behave? Edge  becomes singular

How does Lucas-Kanade behave? Homogeneous   0 eigenvalues

How does Lucas-Kanade behave? Textured regions  two high eigenvalues

How does Lucas-Kanade behave? Edge  becomes singular Homogeneous regions  low gradients High texture 

Iterative Refinement Estimate velocity at each pixel using one iteration of Lucas and Kanade estimation Warp one image toward the other using the estimated flow field (easier said than done) Refine estimate by repeating the process

Optical Flow: Iterative Estimation estimate update Initial guess: Estimate: x0 x

Optical Flow: Iterative Estimation x x0 estimate update Initial guess: Estimate:

Optical Flow: Iterative Estimation x x0 estimate update Initial guess: Estimate: Initial guess: Estimate:

Optical Flow: Iterative Estimation x0 x

Optical Flow: Iterative Estimation Some Implementation Issues: Warping is not easy (ensure that errors in warping are smaller than the estimate refinement) Warp one image, take derivatives of the other so you don’t need to re-compute the gradient after each iteration. Often useful to low-pass filter the images before motion estimation (for better derivative estimation, and linear approximations to image intensity)

Other break-downs Correlation based methods Brightness constancy is not satisfied A point does not move like its neighbors what is the ideal window size? The motion is not small (Taylor expansion doesn’t hold) Aliasing Correlation based methods Regularization based methods Use multi-scale estimation

Optical Flow: Aliasing Temporal aliasing causes ambiguities in optical flow because images can have many pixels with the same intensity. I.e., how do we know which ‘correspondence’ is correct? actual shift estimated shift nearest match is correct (no aliasing) nearest match is incorrect (aliasing) To overcome aliasing: coarse-to-fine estimation.

Multi-Scale Flow Estimation u=10 pixels u=5 pixels u=2.5 pixels u=1.25 pixels image It-1 image I image It image It+1 Gaussian pyramid of image It Gaussian pyramid of image It+1

Multi-Scale Flow Estimation run Lucas-Kanade warp & upsample run Lucas-Kanade . image It-1 image I image It image It+1 Gaussian pyramid of image It Gaussian pyramid of image It+1

Other break-downs Brightness constancy is not satisfied A point does not move like its neighbors what is the ideal window size? The motion is not small (Taylor expansion doesn’t hold) Aliasing Correlation based methods Regularization based methods Use multi-scale estimation

Spatial Coherence Assumption * Neighboring points in the scene typically belong to the same surface and hence typically have similar motions. * Since they also project to nearby points in the image, we expect spatial coherence in image flow. Black

Formalize this Idea Noisy 1D signal: u x Noisy measurements p(x) Black

Regularization Find the “best fitting” smoothed function v(x) u q(x) x Noisy measurements p(x) Black

Membrane model Find the “best fitting” smoothed function v(x) u q(x) x Black

Membrane model Find the “best fitting” smoothed function q(x) u q(x) x Black

Membrane model Find the “best fitting” smoothed function q(x) u q(x) Black

Regularization u q(x) x Spatial smoothness Minimize: assumption Faithful to the data Black

Bayesian Interpretation Black

Discontinuities u q(x) x What about this discontinuity? What is happening here? What can we do? Black

Robust Estimation Noise distributions are often non-Gaussian, having much heavier tails. Noise samples from the tails are called outliers. Sources of outliers (multiple motions): specularities / highlights jpeg artifacts / interlacing / motion blur multiple motions (occlusion boundaries, transparency) velocity space u1 u2 + Black

Occlusion occlusion disocclusion shear Multiple motions within a finite region. Black

Coherent Motion Possibly Gaussian. Black

Multiple Motions Definitely not Gaussian. Black

Weak membrane model u v(x) x Black

Analog line process Penalty function Family of quadratics Black

Analog line process Infimum defines a robust error function. Minima are the same: Black

Least Squares Estimation Standard Least Squares Estimation allows too much influence for outlying points

Robust Regularization v(x) x Treat large spatial derivatives as outliers. Minimize: Black

Robust Estimation Problem: Least-squares estimators penalize deviations between data & model with quadratic error fn (extremely sensitive to outliers) error penalty function influence function Redescending error functions (e.g., Geman-McClure) help to reduce the influence of outlying measurements. error penalty function influence function Black

Regularization Horn and Schunk (1981) Add global smoothness term Smoothness error: Error in brightness constancy equation Minimize: Solve by calculus of variations

Robust Estimation Black & Anandan (1993) Regularization can over-smooth across edges Use “smarter” regularization

Robust Estimation Black & Anandan (1993) Regularization can over-smooth across edges Use “smarter” regularization Minimize: Brightness constancy Smoothness

Optimization Gradient descent Coarse-to-fine (pyramid) Deterministic annealing Black

Example

Multi-resolution registration

Optical Flow Results

Optical Flow Results

Parametric motion estimation

Global (parametric) motion models 2D Models: Affine Quadratic Planar projective transform (Homography) 3D Models: Instantaneous camera motion models Homography+epipole Plane+Parallax Szeliski

Motion models Translation 2 unknowns Affine 6 unknowns Perspective 3D rotation 3 unknowns Szeliski

Example: Affine Motion For panning camera or planar surfaces: Only 6 parameters to solve for Better results Least Square Minimization (over all pixels): [ ] 2 å = a Err ) ( t y x I + p r

Last lecture: Alignment / motion warping “Alignment”: Assuming we know the correspondences, how do we get the transformation? Expressed in terms of absolute coordinates of corresponding points… Generally presumed features separately detected in each frame e.g., affine model in abs. coords…

Today: “flow”, “parametric motion” Two views presumed in temporal sequence…track or analyze spatio-temporal gradient Sparse or dense in first frame Search in second frame Motion models expressed in terms of position change

Today: “flow”, “parametric motion” Two views presumed in temporal sequence…track or analyze spatio-temporal gradient Sparse or dense in first frame Search in second frame Motion models expressed in terms of position change

Today: “flow”, “parametric motion” Two views presumed in temporal sequence…track or analyze spatio-temporal gradient Sparse or dense in first frame Search in second frame Motion models expressed in terms of position change

Today: “flow”, “parametric motion” Two views presumed in temporal sequence…track or analyze spatio-temporal gradient (ui,vi) Sparse or dense in first frame Search in second frame Motion models expressed in terms of position change

Today: “flow”, “parametric motion” Two views presumed in temporal sequence…track or analyze spatio-temporal gradient (ui,vi) Sparse or dense in first frame Search in second frame Motion models expressed in terms of position change

Today: “flow”, “parametric motion” Two views presumed in temporal sequence…track or analyze spatio-temporal gradient Previous Alignment model: (ui,vi) Now, Displacement model: Sparse or dense in first frame Search in second frame Motion models expressed in terms of position change

Other 2D Motion Models Quadratic – instantaneous approximation to planar motion Projective – exact planar motion Szeliski

3D Motion Models Instantaneous camera motion: Homography+Epipole Local Parameter: Instantaneous camera motion: Global parameters: Global parameters: Homography+Epipole Local Parameter: Residual Planar Parallax Motion Global parameters: Local Parameter:

Segmentation of Affine Motion = + Input Segmentation result Result by: L.Zelnik-Manor, M.Irani “Multi-frame estimation of planar motion”, PAMI 2000

Panoramas Input Motion estimation by Andrew Zisserman’s group

Stabilization Result by: L.Zelnik-Manor, M.Irani “Multi-frame estimation of planar motion”, PAMI 2000

Discrete Search vs. Gradient Based Consider image I translated by The discrete search method simply searches for the best estimate. The gradient method linearizes the intensity function and solves for the estimate Szeliski

Correlation and SSD For larger displacements, do template matching Define a small area around a pixel as the template Match the template against each pixel within a search area in next image. Use a match measure such as correlation, normalized correlation, or sum-of-squares difference Choose the maximum (or minimum) as the match Sub-pixel estimate (Lucas-Kanade) Szeliski

Shi-Tomasi feature tracker Find good features (min eigenvalue of 22 Hessian) Use Lucas-Kanade to track with pure translation Use affine registration with first feature patch Terminate tracks whose dissimilarity gets too large Start new tracks when needed Szeliski

Tracking result

How well do these techniques work?

A Database and Evaluation Methodology for Optical Flow Simon Baker, Daniel Scharstein, J.P Lewis, Stefan Roth, Michael Black, and Richard Szeliski ICCV 2007 http://vision.middlebury.edu/flow/

Limitations of Yosemite Only sequence used for quantitative evaluation Limitations: Very simple and synthetic Small, rigid motion Minimal motion discontinuities/occlusions Image 7 Image 8 Yosemite Ground-Truth Flow Flow Color Coding http://www.youtube.com/watch?v=0xRDFf7UtQE Szeliski

Limitations of Yosemite Only sequence used for quantitative evaluation Current challenges: Non-rigid motion Real sensor noise Complex natural scenes Motion discontinuities Need more challenging and more realistic benchmarks Image 7 Image 8 Yosemite Ground-Truth Flow Flow Color Coding Szeliski

Realistic synthetic imagery Randomly generate scenes with “trees” and “rocks” Significant occlusions, motion, texture, and blur Rendered using Mental Ray and “lens shader” plugin Rock Grove Motion estimation Szeliski

Modified stereo imagery Recrop and resample ground-truth stereo datasets to have appropriate motion for OF Venus Moebius Szeliski

Dense flow with hidden texture Paint scene with textured fluorescent paint Take 2 images: One in visible light, one in UV light Move scene in very small steps using robot Generate ground-truth by tracking the UV images Setup Visible UV Lights Image Cropped Szeliski

Experimental results http://vision.middlebury.edu/flow/ Motion estimation Szeliski

Conclusions Difficulty: Data substantially more challenging than Yosemite Diversity: Substantial variation in difficulty across the various datasets Motion GT vs Interpolation: Best algorithms for one are not the best for the other Comparison with Stereo: Performance of existing flow algorithms appears weak Szeliski

Layered Scene Representations

Motion representations How can we describe this scene? Szeliski

Block-based motion prediction Break image up into square blocks Estimate translation for each block Use this to predict next frame, code difference (MPEG-2) Szeliski

Layered motion Break image sequence up into “layers”:  =  = Describe each layer’s motion Szeliski

Layered Representation For scenes with multiple affine motions Estimate dominant motion parameters Reject pixels which do not fit Convergence Restart on remaining pixels

Layered motion Advantages: can represent occlusions / disocclusions each layer’s motion can be smooth video segmentation for semantic processing Difficulties: how do we determine the correct number? how do we assign pixels? how do we model the motion? Szeliski

How do we form them? Szeliski

How do we estimate the layers? compute coarse-to-fine flow estimate affine motion in blocks (regression) cluster with k-means assign pixels to best fitting affine region re-estimate affine motions in each region… Szeliski

Layer synthesis For each layer: stabilize the sequence with the affine motion compute median value at each pixel Determine occlusion relationships Szeliski

Results Szeliski

Recent results: SIFT Flow

Recent results: SIFT Flow

Recent GPU Implementation http://gpu4vision.icg.tugraz.at/ Real time flow exploiting robust norm + regularized mapping

Today: Motion and Flow Motion estimation Patch-based motion (optic flow) Regularization and line processes Parametric (global) motion Layered motion models

Slide Credits Trevor Darrell Rick Szeliski Michael Black