Optical Flow

Distribution of apparent velocities of movement of brightness pattern in an image

Optical Flow Perspective projection (pinhole camera) Instantaneous 2D velocity

4 Definitions optical flow [ Horn&Schunck ] The optical flow is a velocity field in the image which transforms one image into the next image in a sequence [ Horn&Schunck ] [ Horn&Schunck ] The motion field … is the projection into the image of three-dimensional motion vectors [ Horn&Schunck ] + = frame #1 frame #2 flow field

5 Ambiguity of optical flow Frame 1 flow (1): true motion flow (2)

Determining Optical Flow Berthold K.P. Horn and Brian G. Rhunck Artificial Intelligence Laboratory, MIT.

Optical Flow Optical flow field: Estimate the 2D motion field, which are the 2D velocities for all visible points Two key problems – Determine what image property to track – Determine how to track it

8 Brightness Constancy frame t frame t+1 v u

Brightness Constancy Track points of constant brightness, assuming that surface radiance is constant over time Brightness constancy is often violated by Mother Nature, so the resulting optical flow field is sometimes a very poor approximation to the 2D motion field

Optical Flow The optical flow cannot be computed at a point independently of neighboring points without introducing additional constraints

Brightness Constancy Constraint The brightness of a particular point in the pattern is constant Cannot determine the flow velocity locally without additional constraints Constraint line

Brightness Constancy Constraint Normal Velocity No constraint when the gradient magnitude is zero

13 Optical Flow Assumptions: * Slide from Michael Black, CS

14 Optical Flow Assumptions: * Slide from Michael Black, CS

Smoothness Constraint Constraint Assumption – Neighboring points on the objects have similar velocities and – The velocity field of the brightness patterns in the image varies smoothly almost everywhere – Discontinuities in flow can be expected where one object occludes another Algorithms based on smoothness constraint are likely to have difficulties with occluding edges

Smoothness Constraint Minimize the square of the gradient magnitude of the optical flow velocity Sum of the squares of Laplacians of x- and y- components of the flow

Estimate the Partial Derivatives

Estimate the Laplacian of Flow Velocities

Minimization Minimize the sum of errors in rate of change of image brightness And measure of departure from smoothness Total error to be minimized

Minimization Good (u,v) occurs at

Difference of Flow from Local Average (u,v) lies in the direction towards the constraint line along a line that perpendicular to it a^2 plays significant role when Ex, Ey is small: roughly equal to the expected noise in the estimate of

Constrained Minimization Minimizing While maintaining = 0 The (u,v) here is just the intersection on the constraint line

Iterative Solution

24 Optical Flow: 1D Case Brightness Constancy Assumption: IxIx vItIt  Because no change in brightness with time

25? Tracking in the 1D case:

26 Spatial derivative Temporal derivative Tracking in the 1D case:Assumptions: Brightness constancy Brightness constancy Small motion Small motion

27 Tracking in the 1D case: Temporal derivative at 2 nd iteration Iterating helps refining the velocity vector Can keep the same estimate for spatial derivative Converges in about 5 iterations

28 Algorithm for 1D tracking:  Compute local image derivative at :  Compute local image derivative at p:  Initialize velocity vector:  Repeat until convergence:  Compensate for current velocity vector:  Compute temporal derivative:  Update velocity vector: For all pixel of interest : For all pixel of interest p:  Need access to neighborhood pixels round p to compute  Need access to the second image patch, for velocity compensation:  The pixel data to be accessed in next image depends on current velocity estimate (bad?)  Compensation stage requires a bilinear interpolation (because is not integer)  Compensation stage requires a bilinear interpolation (because v is not integer)  The image derivative needs to be kept in memory throughout the iteration process Requirements:

Iterative Solution Very costly to solve these equations simultaneously Iterative algorithms, such as Gauss-Seidel method It is interesting to note that the new estimates at a particular point do not depend directly on the previous estimate at the same point

Fill in Uniform Region Uniform region: brightness gradient is zero No local information to constrain the vecocity Fill in uniform region – Given the values on region boudnary – Solve the Laplace equation under Dirichlet boundary condition

Tightness of Constraint In general, the solution is most accurately determined in the region where the brightness is not too small and varies in direction from point to point – Constraint information is available in a relative small neighborhood

Choice of Iterative Scheme How to interlace the iterations with the time step? – Get stable values before going to next frame – Only one iteration per time-step gives good initial guess Ability to deal with more images per unit time Better estimation in uniform regions because of the noise in measurements of images will be independent and tend to cancel out

Experiment Very low resolution video, say, 32*32 A rotating ball with smoothly varying patterns, but no shading

Results I Simple linear translation of the entire pattern Estimate between two images Showing the velocity Few changes occur after 32 iterations when errors ~ 10%

Results I 1 time step4 time steps

Results I 16 time steps64 time steps

Results II Only one iteration per time-step More rapid Few changes after 16 iteration when errors ~ 7%

Results II 1 time steps4 time steps

Results II 16 time steps64 time steps

Results III Simple rotation and contraction of the brightness pattern

Result III initial32 time steps

Result IV Rigid body motion – Laplacian for one of the velocity components becomes infinite on the occluding bound – Reasonable results are still obtained with finite velocities

Results IV Rotating cylinderRotating sphere

Results IV Exact flow pattern for a rotating cylinder and rotating sphere

Summary A method for computing optical flow from a sequence of images Two components of flow velocity Additional constraint: smoothness Iterative method

46 Horn&Schunck Optical Flow Research: Timeline Lucas&Kanade now Benchmark: Barron et.al. Benchmark: Galvin et.al. many methods more methods A slow and not very consistent improvement in results, but a lot of useful ingredients were developed Seminal papers