Optical Flow Digital Photography CSE558, Spring 2003 Richard Szeliski (notes cribbed from P. Anandan)

Slides:



Advertisements
Similar presentations
Optical Flow Estimation
Advertisements

Investigation Into Optical Flow Problem in the Presence of Spatially-varying Motion Blur Mohammad Hossein Daraei June 2014 University.
Motion Estimation I What affects the induced image motion? Camera motion Object motion Scene structure.
Computer Vision Optical Flow
C280, Computer Vision Prof. Trevor Darrell Lecture 9: Motion.
Algorithms & Applications in Computer Vision
Automatic Image Alignment (direct) : Computational Photography Alexei Efros, CMU, Fall 2006 with a lot of slides stolen from Steve Seitz and Rick.
Motion Estimation I What affects the induced image motion? Camera motion Object motion Scene structure.
Announcements Quiz Thursday Quiz Review Tomorrow: AV Williams 4424, 4pm. Practice Quiz handout.
Optical Flow Methods 2007/8/9.
Direct Methods for Visual Scene Reconstruction Paper by Richard Szeliski & Sing Bing Kang Presented by Kristin Branson November 7, 2002.
Announcements Project1 artifact reminder counts towards your grade Demos this Thursday, 12-2:30 sign up! Extra office hours this week David (T 12-1, W/F.
Feature matching and tracking Class 5 Read Section 4.1 of course notes Read Shi and Tomasi’s paper on.
Announcements Project 1 test the turn-in procedure this week (make sure your folder’s there) grading session next Thursday 2:30-5pm –10 minute slot to.
Caltech, Oct Lihi Zelnik-Manor
Visual motion Many slides adapted from S. Seitz, R. Szeliski, M. Pollefeys.
Motion Computing in Image Analysis
Optical Flow Estimation
Lecture 19: Optical flow CS6670: Computer Vision Noah Snavely
Visual motion Many slides adapted from S. Seitz, R. Szeliski, M. Pollefeys.
Motion Estimation Today’s Readings Trucco & Verri, 8.3 – 8.4 (skip 8.3.3, read only top half of p. 199) Numerical Recipes (Newton-Raphson), 9.4 (first.
1 Stanford CS223B Computer Vision, Winter 2006 Lecture 7 Optical Flow Professor Sebastian Thrun CAs: Dan Maynes-Aminzade, Mitul Saha, Greg Corrado Slides.
Linearizing (assuming small (u,v)): Brightness Constancy Equation: The Brightness Constraint Where:),(),(yxJyxII t  Each pixel provides 1 equation in.
COMP 290 Computer Vision - Spring Motion II - Estimation of Motion field / 3-D construction from motion Yongjik Kim.
3D Rigid/Nonrigid RegistrationRegistration 1)Known features, correspondences, transformation model – feature basedfeature based 2)Specific motion type,
Matching Compare region of image to region of image. –We talked about this for stereo. –Important for motion. Epipolar constraint unknown. But motion small.
KLT tracker & triangulation Class 6 Read Shi and Tomasi’s paper on good features to track
Computational Photography: Image Registration Jinxiang Chai.
Motion Estimation I What affects the induced image motion? Camera motion Object motion Scene structure.
Announcements Project1 due Tuesday. Motion Estimation Today’s Readings Trucco & Verri, 8.3 – 8.4 (skip 8.3.3, read only top half of p. 199) Supplemental:
CSCE 641 Computer Graphics: Image Registration Jinxiang Chai.
Motion estimation Computer Vision CSE576, Spring 2005 Richard Szeliski.
Optical flow Combination of slides from Rick Szeliski, Steve Seitz, Alyosha Efros and Bill Freeman.
CSE 473/573 Computer Vision and Image Processing (CVIP) Ifeoma Nwogu Lecture 19 – Dense motion estimation (OF) 1.
Linearizing (assuming small (u,v)): Brightness Constancy Equation: The Brightness Constraint Where:),(),(yxJyxII t  Each pixel provides 1 equation in.
Optical Flow Donald Tanguay June 12, Outline Description of optical flow General techniques Specific methods –Horn and Schunck (regularization)
The Brightness Constraint
Motion estimation Digital Visual Effects Yung-Yu Chuang with slides by Michael Black and P. Anandan.
Visual Motion Estimation Problems & Techniques Harpreet S. Sawhney Princeton University COS 429 Lecture Feb. 12, 2004.
The Measurement of Visual Motion P. Anandan Microsoft Research.
EECS 274 Computer Vision Motion Estimation.
CSE 185 Introduction to Computer Vision Feature Tracking and Optical Flow.
Visual motion Many slides adapted from S. Seitz, R. Szeliski, M. Pollefeys.
Motion estimation Computer Vision CSE576, Spring 2005 Richard Szeliski.
Uses of Motion 3D shape reconstruction Segment objects based on motion cues Recognize events and activities Improve video quality Track objects Correct.
Effective Optical Flow Estimation
Computer Vision Lecture #10 Hossam Abdelmunim 1 & Aly A. Farag 2 1 Computer & Systems Engineering Department, Ain Shams University, Cairo, Egypt 2 Electerical.
Optical Flow. Distribution of apparent velocities of movement of brightness pattern in an image.
Motion Estimation Today’s Readings Trucco & Verri, 8.3 – 8.4 (skip 8.3.3, read only top half of p. 199) Newton's method Wikpedia page
Motion Estimation I What affects the induced image motion?
Linearizing (assuming small (u,v)): Brightness Constancy Equation: The Brightness Constraint Where:),(),(yxJyxII t  Each pixel provides 1 equation in.
Motion Estimation Today’s Readings Trucco & Verri, 8.3 – 8.4 (skip 8.3.3, read only top half of p. 199) Newton's method Wikpedia page
Motion / Optical Flow II Estimation of Motion Field Avneesh Sud.
Motion estimation Digital Visual Effects, Spring 2006 Yung-Yu Chuang 2005/4/12 with slides by Michael Black and P. Anandan.
Optical flow and keypoint tracking Many slides adapted from S. Seitz, R. Szeliski, M. Pollefeys.
Linearizing (assuming small (u,v)): Brightness Constancy Equation: The Brightness Constraint Where:),(),(yxJyxII t  Each pixel provides 1 equation in.
Image Motion. The Information from Image Motion 3D motion between observer and scene + structure of the scene –Wallach O’Connell (1953): Kinetic depth.
MOTION Model. Road Map Motion Model Non Parametric Motion Field : Algorithms 1.Optical flow field estimation. 2.Block based motion estimation. 3.Pel –recursive.
Motion estimation Parametric motion (image alignment) Tracking Optical flow.
Motion estimation Digital Visual Effects, Spring 2005 Yung-Yu Chuang 2005/3/23 with slides by Michael Black and P. Anandan.
CPSC 641: Image Registration
Motion and Optical Flow
Digital Visual Effects Yung-Yu Chuang
The Brightness Constraint
The Brightness Constraint
The Brightness Constraint
Announcements more panorama slots available now
Announcements Questions on the project? New turn-in info online
Announcements more panorama slots available now
Optical flow and keypoint tracking
Presentation transcript:

Optical Flow Digital Photography CSE558, Spring 2003 Richard Szeliski (notes cribbed from P. Anandan)

3/9/2003Optical Flow2 Classes of Techniques Feature-based methods Extract salient visual features (corners, textured areas) and track them over multiple frames Analyze the global pattern of motion vectors of these features Sparse motion fields, but possibly robust tracking Suitable especially when image motion is large (10-s of pixels) Direct-methods Directly recover image motion from spatio-temporal image brightness variations Global motion parameters directly recovered without an intermediate feature motion calculation Dense motion fields, but more sensitive to appearance variations Suitable for video and when image motion is small (< 10 pixels)

3/9/2003Optical Flow3 Brightness Constancy Equation: The Brightness Constraint Or, better still, Minimize : Linearizing (assuming small (u,v)):

3/9/2003Optical Flow4 Gradient Constraint (or the Optical Flow Constraint) Minimizing: In general Hence,

3/9/2003Optical Flow5 Local Patch Analysis

3/9/2003Optical Flow6 Patch Translation [Lucas-Kanade] Minimizing Assume a single velocity for all pixels within an image patch LHS: sum of the 2x2 outer product tensor of the gradient vector

3/9/2003Optical Flow7 The Aperture Problem Let Algorithm: At each pixel compute by solving M is singular if all gradient vectors point in the same direction e.g., along an edge of course, trivially singular if the summation is over a single pixel or there is no texture i.e., only normal flow is available (aperture problem) Corners and textured areas are OK and

3/9/2003Optical Flow8 Aperture Problem and Normal Flow

3/9/2003Optical Flow9 Local Patch Analysis

3/9/2003Optical Flow10 Iterative Refinement Estimate velocity at each pixel using one iteration of Lucas and Kanade estimation Warp one image toward the other using the estimated flow field (easier said than done) Refine estimate by repeating the process

3/9/2003Optical Flow11 Iterative refinement x t x t t x BUT!!

3/9/2003Optical Flow12 Limits of the gradient method Fails when intensity structure in window is poor Fails when the displacement is large (typical operating range is motion of 1 pixel) Linearization of brightness is suitable only for small displacements Also, brightness is not strictly constant in images actually less problematic than it appears, since we can pre-filter images to make them look similar

3/9/2003Optical Flow13 Pyramids Pyramids were introduced as a multi-resolution image computation paradigm in the early 80s. The most popular pyramid is the Burt pyramid, which foreshadows wavelets Two kinds of pyramids: Low pass or “Gaussian pyramid” Band-pass or “Laplacian pyramid”

3/9/2003Optical Flow14 Gaussian Pyramid Convolve image with a small Gaussian kernel Typically 5x5 Subsample (decimate by 2) to get lower resolution image Repeat for more levels A sequence of low-pass filtered images

3/9/2003Optical Flow15 Laplacian pyramid Laplacian as Difference of Gaussian Band-pass filtered images Highlights edges at different spatial scales For matching, this is less sensitive to image illumination changes But more noisy than using Gaussians

3/9/2003Optical Flow16

3/9/2003Optical Flow17 image I image J JwJw warp refine + Pyramid of image JPyramid of image I image I image J Coarse-to-Fine Estimation u=10 pixels u=5 pixels u=2.5 pixels u=1.25 pixels

3/9/2003Optical Flow18 J JwJw I warp refine + J JwJw I warp refine + J pyramid construction J JwJw I warp refine + I pyramid construction Coarse-to-Fine Estimation

3/9/2003Optical Flow19 Global Motion Models 2D Models: Affine Quadratic Planar projective transform (Homography) 3D Models: Instantaneous camera motion models Homography+epipole Plane+Parallax

3/9/2003Optical Flow20 Example: Affine Motion Substituting into the B.C. Equation: Each pixel provides 1 linear constraint in 6 global unknowns Least Square Minimization (over all pixels):

3/9/2003Optical Flow21 Quadratic – instantaneous approximation to planar motion Other 2D Motion Models Projective – exact planar motion

3/9/2003Optical Flow22 3D Motion Models Local Parameter: Instantaneous camera motion: Global parameters: Homography+Epipole Local Parameter: Residual Planar Parallax Motion Global parameters: Local Parameter:

3/9/2003Optical Flow23 Correlation and SSD For larger displacements, do template matching Define a small area around a pixel as the template Match the template against each pixel within a search area in next image. Use a match measure such as correlation, normalized correlation, or sum-of-squares difference Choose the maximum (or minimum) as the match Sub-pixel interpolation also possible

3/9/2003Optical Flow24 SSD Surface – Textured area

3/9/2003Optical Flow25 SSD Surface -- Edge

3/9/2003Optical Flow26 SSD – homogeneous area

3/9/2003Optical Flow27 Discrete Search vs. Gradient Based Estimation Consider image I translated by The discrete search method simply searches for the best estimate. The gradient method linearizes the intensity function and solves for the estimate

3/9/2003Optical Flow28 Consider image I translated by Uncertainty in Local Estimation Now, This assumes uniform priors on the velocity field

3/9/2003Optical Flow29 Quadratic Approximation When are small After some fiddling around, we can show

3/9/2003Optical Flow30 Posterior uncertainty At edges is singular, but just take pseudo-inverse Note that the error is always convex, since is positive semi-definite i.e., even for occluded points and other false matches, this is the case… seems a bit odd!

3/9/2003Optical Flow31 Match plus confidence Numerically compute error for various Search for the peak Numerically fit a qudratic to around the peak Find sub-pixel estimate for and covariance If the matrix is negative, it is false match Or even better, if you can afford it, simply maintain a discrete sampling of and

3/9/2003Optical Flow32 Quadratic Approximation and Covariance Estimation When where

3/9/2003Optical Flow33 Correlation Window Size Small windows lead to more false matches Large windows are better this way, but… Neighboring flow vectors will be more correlated (since the template windows have more in common) Flow resolution also lower (same reason) More expensive to compute Another way to look at this: Small windows are good for local search but more precise and less smooth Large windows good for global search but less precise and more smooth method

3/9/2003Optical Flow34 Robust Estimation Standard Least Squares Estimation allows too much influence for outlying points

3/9/2003Optical Flow35 Robust Estimation Robust gradient constraint Robust SSD

3/9/2003Optical Flow36 References J. R. Bergen, P. Anandan, K. J. Hanna, and R. Hingorani. Hierarchical model-based motion estimation. In ECCV’92, pp. 237–252, Italy, May M. J. Black and P. Anandan. The robust estimation of multiple motions: Parametric and piecewise-smooth flow fields. Comp. Vis. Image Understanding, 63(1):75–104, H. S. Sawhney and S. Ayer. Compact representation of videos through dominant multiple motion estimation. IEEE Trans. Patt. Anal. Mach. Intel., 18(8):814–830, Aug Y. Weiss. Smoothness in layers: Motion segmentation using nonparametric mixture estimation. In CVPR’97, pp. 520–526, June 1997.