Optical Flow Digital Photography CSE558, Spring 2003 Richard Szeliski (notes cribbed from P. Anandan)

3/9/2003Optical Flow2 Classes of Techniques Feature-based methods Extract salient visual features (corners, textured areas) and track them over multiple frames Analyze the global pattern of motion vectors of these features Sparse motion fields, but possibly robust tracking Suitable especially when image motion is large (10-s of pixels) Direct-methods Directly recover image motion from spatio-temporal image brightness variations Global motion parameters directly recovered without an intermediate feature motion calculation Dense motion fields, but more sensitive to appearance variations Suitable for video and when image motion is small (< 10 pixels)

3/9/2003Optical Flow3 Brightness Constancy Equation: The Brightness Constraint Or, better still, Minimize : Linearizing (assuming small (u,v)):

3/9/2003Optical Flow4 Gradient Constraint (or the Optical Flow Constraint) Minimizing: In general Hence,

3/9/2003Optical Flow5 Local Patch Analysis

3/9/2003Optical Flow6 Patch Translation [Lucas-Kanade] Minimizing Assume a single velocity for all pixels within an image patch LHS: sum of the 2x2 outer product tensor of the gradient vector

3/9/2003Optical Flow7 The Aperture Problem Let Algorithm: At each pixel compute by solving M is singular if all gradient vectors point in the same direction e.g., along an edge of course, trivially singular if the summation is over a single pixel or there is no texture i.e., only normal flow is available (aperture problem) Corners and textured areas are OK and

3/9/2003Optical Flow8 Aperture Problem and Normal Flow

3/9/2003Optical Flow9 Local Patch Analysis

3/9/2003Optical Flow10 Iterative Refinement Estimate velocity at each pixel using one iteration of Lucas and Kanade estimation Warp one image toward the other using the estimated flow field (easier said than done) Refine estimate by repeating the process

3/9/2003Optical Flow11 Iterative refinement x t x t t x BUT!!

3/9/2003Optical Flow12 Limits of the gradient method Fails when intensity structure in window is poor Fails when the displacement is large (typical operating range is motion of 1 pixel) Linearization of brightness is suitable only for small displacements Also, brightness is not strictly constant in images actually less problematic than it appears, since we can pre-filter images to make them look similar

3/9/2003Optical Flow13 Pyramids Pyramids were introduced as a multi-resolution image computation paradigm in the early 80s. The most popular pyramid is the Burt pyramid, which foreshadows wavelets Two kinds of pyramids: Low pass or “Gaussian pyramid” Band-pass or “Laplacian pyramid”

3/9/2003Optical Flow14 Gaussian Pyramid Convolve image with a small Gaussian kernel Typically 5x5 Subsample (decimate by 2) to get lower resolution image Repeat for more levels A sequence of low-pass filtered images

3/9/2003Optical Flow15 Laplacian pyramid Laplacian as Difference of Gaussian Band-pass filtered images Highlights edges at different spatial scales For matching, this is less sensitive to image illumination changes But more noisy than using Gaussians

3/9/2003Optical Flow16

3/9/2003Optical Flow17 image I image J JwJw warp refine + Pyramid of image JPyramid of image I image I image J Coarse-to-Fine Estimation u=10 pixels u=5 pixels u=2.5 pixels u=1.25 pixels

3/9/2003Optical Flow18 J JwJw I warp refine + J JwJw I warp refine + J pyramid construction J JwJw I warp refine + I pyramid construction Coarse-to-Fine Estimation

3/9/2003Optical Flow19 Global Motion Models 2D Models: Affine Quadratic Planar projective transform (Homography) 3D Models: Instantaneous camera motion models Homography+epipole Plane+Parallax

3/9/2003Optical Flow20 Example: Affine Motion Substituting into the B.C. Equation: Each pixel provides 1 linear constraint in 6 global unknowns Least Square Minimization (over all pixels):

3/9/2003Optical Flow21 Quadratic – instantaneous approximation to planar motion Other 2D Motion Models Projective – exact planar motion

3/9/2003Optical Flow22 3D Motion Models Local Parameter: Instantaneous camera motion: Global parameters: Homography+Epipole Local Parameter: Residual Planar Parallax Motion Global parameters: Local Parameter:

3/9/2003Optical Flow23 Correlation and SSD For larger displacements, do template matching Define a small area around a pixel as the template Match the template against each pixel within a search area in next image. Use a match measure such as correlation, normalized correlation, or sum-of-squares difference Choose the maximum (or minimum) as the match Sub-pixel interpolation also possible

3/9/2003Optical Flow24 SSD Surface – Textured area

3/9/2003Optical Flow25 SSD Surface -- Edge

3/9/2003Optical Flow26 SSD – homogeneous area

3/9/2003Optical Flow27 Discrete Search vs. Gradient Based Estimation Consider image I translated by The discrete search method simply searches for the best estimate. The gradient method linearizes the intensity function and solves for the estimate

3/9/2003Optical Flow28 Consider image I translated by Uncertainty in Local Estimation Now, This assumes uniform priors on the velocity field

3/9/2003Optical Flow29 Quadratic Approximation When are small After some fiddling around, we can show

3/9/2003Optical Flow30 Posterior uncertainty At edges is singular, but just take pseudo-inverse Note that the error is always convex, since is positive semi-definite i.e., even for occluded points and other false matches, this is the case… seems a bit odd!

3/9/2003Optical Flow31 Match plus confidence Numerically compute error for various Search for the peak Numerically fit a qudratic to around the peak Find sub-pixel estimate for and covariance If the matrix is negative, it is false match Or even better, if you can afford it, simply maintain a discrete sampling of and

3/9/2003Optical Flow32 Quadratic Approximation and Covariance Estimation When where

3/9/2003Optical Flow33 Correlation Window Size Small windows lead to more false matches Large windows are better this way, but… Neighboring flow vectors will be more correlated (since the template windows have more in common) Flow resolution also lower (same reason) More expensive to compute Another way to look at this: Small windows are good for local search but more precise and less smooth Large windows good for global search but less precise and more smooth method

3/9/2003Optical Flow34 Robust Estimation Standard Least Squares Estimation allows too much influence for outlying points

3/9/2003Optical Flow35 Robust Estimation Robust gradient constraint Robust SSD

3/9/2003Optical Flow36 References J. R. Bergen, P. Anandan, K. J. Hanna, and R. Hingorani. Hierarchical model-based motion estimation. In ECCV’92, pp. 237–252, Italy, May M. J. Black and P. Anandan. The robust estimation of multiple motions: Parametric and piecewise-smooth flow fields. Comp. Vis. Image Understanding, 63(1):75–104, H. S. Sawhney and S. Ayer. Compact representation of videos through dominant multiple motion estimation. IEEE Trans. Patt. Anal. Mach. Intel., 18(8):814–830, Aug Y. Weiss. Smoothness in layers: Motion segmentation using nonparametric mixture estimation. In CVPR’97, pp. 520–526, June 1997.