Computer Vision Optical Flow Some slides from K. H. Shafique [http://www.cs.ucf.edu/courses/cap6411/cap5415/] and T. Darrell

Correspondence Which pixel went where? Time: t Time: t + dt Bahadir K. Gunturk EE 7730 - Image Analysis II

Motion Field vs. Optical Flow Scene flow: 3D velocities of scene points. Motion field: 2D projection of scene flow. Optical flow: Approximation of motion field derived from apparent motion of brightness patterns in image. Bahadir K. Gunturk EE 7730 - Image Analysis II

Motion Field vs. Optical Flow Consider perfectly uniform sphere rotating in front of camera. Motion field follows surface points. Optical flow is zero since motion is not visible. Now consider stationary sphere with moving light source. Motion field is zero. But optical flow results from changing shading. But, in general, optical flow is a reliable indicator of motion field. Bahadir K. Gunturk EE 7730 - Image Analysis II

Applications Correct for camera jitter (stabilization) Object tracking Video compression Structure from motion Segmentation Correct for camera jitter (stabilization) Combining overlapping images (panoramic image construction) … Bahadir K. Gunturk EE 7730 - Image Analysis II

Optical Flow Problem How to estimate pixel motion from one image to another? Bahadir K. Gunturk EE 7730 - Image Analysis II

Computing Optical Flow Assumption 1: Brightness is constant. Assumption 2: Motion is small. (from Taylor series expansion) Bahadir K. Gunturk EE 7730 - Image Analysis II

Computing Optical Flow Combine In the limit as u and v goes to zero, the equation becomes exact (optical flow equation) Bahadir K. Gunturk EE 7730 - Image Analysis II

Computing Optical Flow At each pixel, we have one equation, two unknowns. This means that only the flow component in the gradient direction can be determined. (optical flow equation) The motion is parallel to the edge, and it cannot be determined. This is called the aperture problem. Bahadir K. Gunturk EE 7730 - Image Analysis II

Computing Optical Flow We need more constraints. The most commonly used assumption is that optical flow changes smoothly locally. Bahadir K. Gunturk EE 7730 - Image Analysis II

Computing Optical Flow One method: The (u,v) is constant within a small neighborhood of a pixel. Optical flow equation: Use a 5x5 window: Two unknowns, 25 equation ! Bahadir K. Gunturk EE 7730 - Image Analysis II

Computing Optical Flow Find minimum least squares solution Lucas – Kanade method Bahadir K. Gunturk EE 7730 - Image Analysis II

Computing Optical Flow Lucas – Kanade When is This Solvable? ATA should be invertible ATA should not be too small. Other wise noise will be amplified when inverted.) Bahadir K. Gunturk EE 7730 - Image Analysis II

Computing Optical Flow What are the potential causes of errors in this procedure? Brightness constancy is not satisfied The motion is not small A point does not move like its neighbors window size is too large Bahadir K. Gunturk EE 7730 - Image Analysis II

Improving accuracy Recall our small motion assumption This is not exact To do better, we need to add higher order terms back in: This is a polynomial root finding problem Can solve using Newton’s method Also known as Newton-Raphson method Lucas-Kanade method does one iteration of Newton’s method Better results are obtained via more iterations Bahadir K. Gunturk EE 7730 - Image Analysis II

Iterative Refinement Iterative Lucas-Kanade Algorithm Estimate velocity at each pixel by solving Lucas-Kanade equations Warp H towards I using the estimated flow field - use image warping techniques Repeat until convergence Bahadir K. Gunturk EE 7730 - Image Analysis II

Revisiting the small motion assumption What can we do when the motion is not small? Bahadir K. Gunturk EE 7730 - Image Analysis II

Reduce the resolution! Bahadir K. Gunturk EE 7730 - Image Analysis II

Coarse-to-fine optical flow estimation Gaussian pyramid of image H Gaussian pyramid of image I image I image H u=10 pixels u=5 pixels u=2.5 pixels u=1.25 pixels image H image I Bahadir K. Gunturk EE 7730 - Image Analysis II

Coarse-to-fine optical flow estimation Gaussian pyramid of image H Gaussian pyramid of image I image I image H run iterative L-K warp & upsample run iterative L-K . Bahadir K. Gunturk EE 7730 - Image Analysis II

Multi-resolution Lucas-Kanade Algorithm Bahadir K. Gunturk EE 7730 - Image Analysis II

Block-Based Motion Estimation Bahadir K. Gunturk EE 7730 - Image Analysis II

Block-Based Motion Estimation Bahadir K. Gunturk EE 7730 - Image Analysis II

Block-Based Motion Estimation Gaussian pyramid of image H Gaussian pyramid of image I image I image H Hierarchical search image H image I Bahadir K. Gunturk EE 7730 - Image Analysis II

Parametric (Global) Motion Sometimes few parameters are enough to represent the motion of whole image translation rotation scale Bahadir K. Gunturk EE 7730 - Image Analysis II

Parametric (Global) Motion Affine Flow Bahadir K. Gunturk EE 7730 - Image Analysis II

Parametric (Global) Motion Affine Flow Bahadir K. Gunturk EE 7730 - Image Analysis II

Parametric (Global) Motion Affine Flow – Approach Bahadir K. Gunturk EE 7730 - Image Analysis II

Iterative Refinement Iterative Estimation Estimate parameters by solving the linear system. Warp H towards I using the estimated flow field - use image warping techniques Repeat until convergence or a fixed number of iterations Bahadir K. Gunturk EE 7730 - Image Analysis II

Coarse-to-fine global flow estimation Gaussian pyramid of image H Gaussian pyramid of image I image I image H Compute Flow Iteratively warp & upsample Compute Flow Iteratively . image J image I Bahadir K. Gunturk EE 7730 - Image Analysis II

Global Flow Find features Match features Fit parametric model Application: Mosaic construction Bahadir K. Gunturk EE 7730 - Image Analysis II

Image Warping Bahadir K. Gunturk EE 7730 - Image Analysis II

Image Warping Interpolate to find the intensity at (x’,y’) Pixel values are known in this image Pixel values are to found in this image Bahadir K. Gunturk EE 7730 - Image Analysis II

Other Parametric Motion Models Perspective and Pseudo-Perspective: Approximation to perspective. (Planar scene + Perspective projection) Bahadir K. Gunturk EE 7730 - Image Analysis II