Optical Flow

Brightness Constancy The Aperture problem Regularization Lucas-Kanade Coarse-to-fine Parametric motion models Direct depth SSD tracking Robust flow Textured motion

Optical Flow: Where do pixels move to?

Motion is a basic cue Motion can be the only cue for segmentation

Applications tracking structure from motion motion segmentation stabilization compression mosaicing …

Optical Flow Brightness Constancy The Aperture problem Regularization Lucas-Kanade Coarse-to-fine Parametric motion models Direct depth SSD tracking Robust flow Textured motion

Definition of optical flow OPTICAL FLOW = apparent motion of brightness patterns Ideally, the optical flow is the projection of the three-dimensional velocity vectors on the image 

Caution required ! Two examples : 1. Uniform, rotating sphere  O.F. = 0 2. No motion, but changing lighting  O.F.  0 

Mathematical formulation I (x,y,t) = brightness at (x,y) at time t Optical flow constraint equation :  Brightness constancy assumption:

Optical Flow Brightness Constancy The Aperture problem Regularization Lucas-Kanade Coarse-to-fine Parametric motion models Direct depth SSD tracking Robust flow Textured motion

The aperture problem 1 equation in 2 unknowns 

The Aperture Problem 0

What is Optical Flow, anyway? Estimate of observed projected motion field Not always well defined! Compare: Motion Field (or Scene Flow) projection of 3-D motion field Normal Flow observed tangent motion Optical Flow apparent motion of the brightness pattern (hopefully equal to motion field) Consider Barber pole illusion

Optical Flow Brightness Constancy The Aperture problem Regularization Lucas-Kanade Coarse-to-fine Parametric motion models Direct depth SSD tracking Robust flow Textured motion

Horn & Schunck algorithm Additional smoothness constraint : besides OF constraint equation term minimize e s + e c 

The calculus of variations look for functions that extremize functionals and 

Calculus of variations Suppose 1. f(x) is a solution 2.  (x) is a test function with  (x 1 )= 0 and  (x 2 ) = 0 for the optimum : 

Calculus of variations Using integration by parts : Therefore regardless of  (x), then Euler-Lagrange equation 

Calculus of variations Generalizations 1. Simultaneous Euler-Lagrange equations : 2. 2 independent variables x and y 

Hence Now by Gauss’s integral theorem, so that = 0 Calculus of variations 

Given the boundary conditions, Consequently, for all test functions . is the Euler-Lagrange equation 

Horn & Schunck The Euler-Lagrange equations : In our case, so the Euler-Lagrange equations are is the Laplacian operator 

Horn & Schunck Remarks : 1. Coupled PDEs solved using iterative methods and finite differences 2. More than two frames allow a better estimation of I t 3. Information spreads from corner-type patterns 



Horn & Schunck, remarks 1. Errors at boundaries 2. Example of regularization (selection principle for the solution of ill-posed problems) 

Results of an enhanced system

Structure from motion with OF

Optical Flow Brightness Constancy The Aperture problem Regularization Lucas-Kanade Coarse-to-fine Parametric motion models Direct depth SSD tracking Robust flow Textured motion

Optical Flow Brightness Constancy The Aperture problem Regularization Lucas-Kanade Coarse-to-fine Parametric motion models Direct depth SSD tracking Robust flow Textured motion

Optical Flow Brightness Constancy The Aperture problem Regularization Lucas-Kanade Coarse-to-fine Parametric motion models Direct depth SSD tracking Robust flow Textured motion

Optical Flow Brightness Constancy The Aperture problem Regularization Lucas-Kanade Coarse-to-fine Parametric motion models Direct depth SSD tracking Robust flow Bayesian flow

Optical Flow Brightness Constancy The Aperture problem Regularization Lucas-Kanade Coarse-to-fine Parametric motion models Direct depth SSD tracking Robust flow Textured motion

Optical Flow Brightness Constancy The Aperture problem Regularization Lucas-Kanade Coarse-to-fine Parametric motion models Direct depth SSD tracking Robust flow Textured motion

Optical Snow snow.html Optical snow is the type of motion an observer sees when watching a snow fall. Flakes that are closer to the observer appear to move faster than flakes which are farther away.

Textured Motion xturedmotion.htm xturedmotion.htm Natural scenes contain rich stochastic motion patterns which are characterized by the movement of a large amount of particle and wave elements, such as falling snow, water waves, dancing grass, etc. We call these motion patterns "textured motion".