COMP 290 Computer Vision - Spring Motion II - Estimation of Motion field / 3-D construction from motion Yongjik Kim

COMP 290 Computer Vision - Spring Outline Estimating the motion field –Differential technique Optical flow algorithm (Trucco) Filling in optical flow / segmentation of multiple objects (Horn) Validity of optical flow –Feature-based technique Feature matching / Feature tracking –Recovering 3-D motion & structure

COMP 290 Computer Vision - Spring Estimating the Motion Field –Ref) Trucco, Chapter 8.4 –Differential Techniques : based on spatial & temporal variations of the image at all pixels. –Matching (feature-based) techniques : rely on special image points (features) and track them through frames.

COMP 290 Computer Vision - Spring Differential Techniques : Optical Flow Optical Flow Algorithm (Trucco, p196) –For each pixel p, we must satisfy  I v + dI/dt = 0 Assumption : we assume that this equation holds in the neighborhood of p with constant v. we write this equation for a small (typically 5x5) patch centered at p. Then we find least square fit of v - this is the calculated optical flow for pixel p.

COMP 290 Computer Vision - Spring Differential Techniques : Optical Flow –We assumed that  I v + dI/dt = 0 holds in the neighborhood of p with constant v. –Can we justify it? –Yes : In case of rigid motion, the motion field of a moving plane is a quadratic polynomial in the coordinates (x, y, f) of the image points. Therefore, if the object is smooth & rigid, we can assume the motion field varies smoothly. cf) Trucco p187

COMP 290 Computer Vision - Spring Filling in Optical Flow Information Ref) Horn : Chapter 12 Ref) Determining Optical Flow, Horn & Schunck (Artificial Intelligence 17(1981) p ) –For any point, we can have optical flow information in only one direction (parallel to  I : orthogonal to the boundary). –We must ‘fill’ the image with these information from boundaries.

COMP 290 Computer Vision - Spring Filling in Optical Flow Information How? –Two criteria e s : The optical flow must be smooth e c : The error in the optical flow constraint equation must be small –Iteratively minimize e s + k e c cf) Horn, p284

COMP 290 Computer Vision - Spring Filling in Optical Flow Information Results –Horn, Chapter 12, p –Horn & Schunck

COMP 290 Computer Vision - Spring Discontinuities in Optical Flow If the scene is composed of multiple objects, there are discontinuities in the optical flow. –We need discontinuity information (= object segmentation) to refine optical flow. –On the other hand, we need optical flow to find discontinuities. –Solution : iteratively refine both segmentation and optical flow

COMP 290 Computer Vision - Spring Validity of Optical Flow –The optical flow equation assumes that image brightness remains constant. Is that valid? Ref) Trucco, p194 –Even with simple Lambertian reflectance, image brightness is constant only in case of pure translation, or when the illumination direction is parallel to the angular velocity (i.e. the axis of rotation).

COMP 290 Computer Vision - Spring Validity of Optical Flow –Therefore, in general, the optical flow is almost always different from the motion field! –The error is small if image gradient is high.

COMP 290 Computer Vision - Spring Feature-based Techniques –Ref) Trucco, p –They only get ‘sparse motion field’ - motion vectors are known only at feature points. Two-frame method : Feature matching How to find features? Use optical flow algorithm (least square fit) - if the calculated optical flow v is confident enough (i.e., if its covariance matrix is smaller enough), we consider it a feature point.

COMP 290 Computer Vision - Spring Feature Matching (continued) Algorithm Initially set ‘displacement field’ using optical flow alrogithm. For each feature point p in image 1, –1. ‘Warp’ its neighborhood Q1 according to current displacement vector in order to get Q’. –2. From Q’ and corresponding section Q2 of image B, find optical flow at p (= new displacement vector) and image difference between Q’ and Q2. –3. If image difference is below threshold, exit. Otherwise go to step 1.

COMP 290 Computer Vision - Spring Feature Tracking Multiple-frame Method : Feature Tracking –similar to feature matching, iterated between frame 0 & 1, and then 1 & 2, and then 2 & 3, and so on... –use knowledge of prior frames to estimate the position of feature points at next frame cf) Trucco, p201 - see the uncertainty (white cross) decreasing cf) Trucco, p203 - correspondence problem

COMP 290 Computer Vision - Spring D Motion & Structure from Sparce Motion Field –Feature-based technique gives us only sparse motion field : we have to extract information about (dense) 3-D motion field & 3-D structure from it! –Factorization method : If the camera model is orthographic, and the motion is rigid, the (2N * n) matrix of n feature points at N frames has at most rank 3 : huge intercorrelation! Use SVD to calculate 3D motion & structure.

COMP 290 Computer Vision - Spring Motion of Rigid Objects –Ref) Trucco, Chapter 8.2 –‘Any’ rigid body motion, at a given instant, can be decomposed into translation & rotation with respect to a given point. Ex) a rolling ball

COMP 290 Computer Vision - Spring Motion of Rigid Objects –In particular, a rigid body motion can be decomposed into: translation rotation about the origin in the camera reference frame –cf) Trucco, p183 –From now on, we will mean by ‘rotation’ a rotation defined like above.

COMP 290 Computer Vision - Spring Motion Parallax –Imagine two points instantaneously coincident in the image coordinate. –In general, their apparent motion (motion field) doesn’t have to be the same. –But the difference in their apprent motion will always point toward the epipole! epipole : vanishing point of the motion field if there were no rotation cf) Trucco, p

COMP 290 Computer Vision - Spring –If we assume the motion field is continuous, we can use adjacent points instead of coinciding points to calculate direction to epipole. –A pair of adjacent points determines a line intersecting epipole - with many pairs, we can use least square fit to find epipole. –Once we find the epipole, we can calculate angular velocity : then we have 3D motion. 3-D Motion & Structure from Dense Motion Field (sketch)