1. Recent progress in optical flow
Presented by: Darya Frolova and Denis Simakov

2. Optical Flow is not in favor
Very popular slide:
Often not using Optical Flow is stated as one of the main advantages of a method
Optical Flow methods have a reputation of either unreliable or slow
Recent works claim:
Optical Flow can be computed fast and accurately

3. Optical Flow Research: Timeline
Horn&Schunck
many methods
more methods
Lucas&Kanade
1981
1992
1998
now
Benchmark: Barron et.al.
Benchmark: Galvin et.al.
Seminal papers
A slow and not very consistent improvement in results, but a lot of useful ingredients were developed

4. In This Lecture We will describe :
Ingredients for an accurate and robust optical flow
How people combine these ingredients
Fast algorithms
Papers:
Combining the advantages of local and global optic flow methods (“Lucas/Kanade meets Horn/Schunck”)
A. Bruhn, J. Weickert, C. Schnörr,
High accuracy optical flow estimation based on a theory for warping
T. Brox, A. Bruhn, N. Papenberg, J. Weickert,
Real-Time Optic Flow Computation with Variational Methods
A. Bruhn, J. Weickert, C. Feddern, T. Kohlberger, C. Schnörr,
Towards ultimate motion estimation: Combining highest accuracy with real-time performance A. Bruhn, J. Weickert, 2005
Bilateral filtering-based optical flow estimation with occlusion detection J.Xiao, H.Cheng, H.Sawhney, C.Rao, M.Isnardi, 2006

5. What is Optical Flow?
explain: arrows and dots of flow field …

6. Definitions
The optical flow is a velocity field in the image which transforms one image into the next image in a sequence [ Horn&Schunck ] + =
frame #1
flow field
frame #2
The motion field … is the projection into the image of three-dimensional motion vectors [ Horn&Schunck ]

7. Ambiguity of optical flow
flow (1): true motion
flow (2)
Frame 1

8. Applications optical flow video compression 3D reconstruction
segmentation
object detection
activity detection
key frame extraction
interpolation in time
motion field
We are usually interested in actual motion

9. Outline Ingredients for an accurate and robust optical flow
Local image constraints on motion
Robust statistics
Spatial coherence
How people combine these ingredients
Fast algorithms

10. Local image constraints

11. Brightness Constancy u
frame t+1 v
frame t

12. Linearized brightness constancy
Deviation from brightness constancy (we want it to be zero)
Complex dependence on
Linearize:

13. Linearized brightness constancy
Let us square the difference:
J – “motion tensor”, or “structure tensor”

14. Averaged linearized constraint
J is a function of x, y (a matrix for every point)
Combine over small neighborhoods (more robust to noise): J = *

15. Method of Lucas&Kanade
Solve independently for each point [ Lucas&Kanade 1981 ]
linear system
Can be solved for every point where matrix is not degenerate

16. Lukas&Kanade - Results
Rubik cube
Hamburg taxi
flow field
flow field

17. Brightness is not always constant
Rotating cylinder
Brightness constancy does not always hold
Gradient constancy holds
intensity
intensity derivative
position
position

18. Local constraints - Summary
We have seen
linearized
brightness constancy
averaged linearized
averaged linearized
gradient constancy

19. Local constraints are not enough!

20. Local constraints work poorly
Optical flow direction using only local constraints
input video
color encodes direction as marked on the boundary

21. Where local constraints fail
Uniform regions
Motion is not observable in the image (locally)

22. Where local constraints fail
“Aperture problem”
We can estimate only one flow component (normal)

23. Where local constraints fail
Occlusions
We have not seen where some points moved
Occluded regions are marked in red

24. Obtaining support from neighbors
Two main problems with local constraints:
information about motion is missing in some points => need spatial coherency
constraints do not hold everywhere => need methods to combine them robustly
good
missing
wrong

25. Robust combination of partially reliable data
or: How to hold elections

26. Toy example xi xi → xi + ∆ L2: L1:
Find “best” representative for the set of numbers xi
L2:
L1:
Influence of xi on E:
xi → xi + ∆
proportional to
equal for all xi
Outliers influence the most
Majority decides

27. Elections and robust statistics
many ordinary people
a very rich man
wealth
Oligarchy
Democracy
ordinary people – 1 vote
a rich man – many votes
If you know what to do – choose oligarchy, if don’t know - democracy
Votes proportional to the wealth
One vote per person
like in L2 norm minimization
like in L1 norm minimization

28. Combination of two flow constraints
usual: L2
easy to analyze and minimize
sensitive to outliers
robust: L1
robust in presence of outliers
non-smooth: hard to analyze
robust regularized
smooth: easy to analyze
robust in presence of outliers ε
[A. Bruhn, J. Weickert, 2005] Towards ultimate motion estimation: Combining highest accuracy with real-time performance

29. Spatial Propagation

30. Obtaining support from neighbors
Two main problems with local constraints:
information about motion is missing in some points => need spatial coherency
constraints do not hold everywhere => need methods to combine them robustly
good
missing
wrong

31. Homogeneous propagation
- flow in the x direction
flow in the y direction
gradient
This constraint is not correct on motion boundaries => over-smoothing of the resulting flow
[Horn&Schunck 1981]

32. Robustness to flow discontinuitiesε
(also known as isotropic flow-driven regularization)
[T. Brox, A. Bruhn, N. Papenberg, J. Weickert, 2004]
High accuracy optical flow estimation based on a theory for warping

33. Selective flow filtering
We want to propagate information
without crossing image and flow discontinuities
from “good” points only (not occluded)
Solution:
use “bilateral” filter in space, intensity, flow;
taking into account occlusions
[J.Xiao, H.Cheng, H.Sawhney, C.Rao, M.Isnardi, 2006]
Bilateral filtering-based optical flow estimation with occlusion detection

34. Bilateral filter Unilateral (usual) Bilateral x I I x
Preserves discontinuities! x I x I
[C. Tomasi, R. Manduchi, 1998] Bilateral filtering for gray and color images.

35. Using of bilateral filter - Example
occluded regions
cyan rectangle moves to the right and occludes background region marked by red
[J.Xiao, H.Cheng, H.Sawhney, C.Rao, M.Isnardi, 2006]
Bilateral filtering-based optical flow estimation with occlusion detection

36. Learning of spatial coherence
Come to the next lecture …

37. Spatial coherence: Summary
Homogeneous propagation - oversmoothing
Robust statistics with homogeneous propagation - preserves flow discontinuities
Bilateral filtering - combines information from regions with similar flow and similar intensities
Handles occlusions

38. Two more useful ingredients
in brief – one slide each

39. 2D vs. 3D Several frames allow more accurate optical flow estimation

40. Multiscale Optical Flow
Linearization: valid only for small flow
pyramid for frame 1
pyramid for frame 2
frame 1 warped ? +
upsample +
(other names: “warping”, “coarse-to-fine”, “multiresolution”)

41. How to make tasty soup with these ingredients: several recipes
Methods
How to make tasty soup with these ingredients: several recipes

42. Outline Ingredients for an accurate and robust optical flow
How people combine these ingredients
Lukas & Kanade meet Horn & Schunck
The more ingredients – the better
Bilateral filtering and occlusions
Fast algorithms

43. Combining ingredients
Spatial coherency
Homogeneous
Flow-driven
Bilateral filtering + occlusions
Local constraints
Brightness constancy
Image gradient constancy
Energy = ∫ϕ (Data) + ∫ϕ (“Smoothness”)
Combined using robust statistics
Computed coarse-to-fine
Use several frames

44. Combining Local and Global
Remember:
Lucas&Kanade
Horn&Schunk
Basic “Combining local and global”
[A. Bruhn, J. Weickert, C. Schnörr, 2002 ]

45. Sensitivity to noise – quantitative results
frame t+1
Error measure: angle between true and computed flow in (x,y,t) space
frame t
ground truth flow

46. The more ingredients - the better
brightness constancy
spatial coherence
gradient constancy
[Bruhn, Weickert, 2005] Towards ultimate motion estimation: Combining highest accuracy with real-time performance

47. Combining local and global
Quantitative results
Average
Standard Deviation
Lucas&Kanade
4.3
(density 35%)
Horn&Schunk
9.8
16.2
Combining local and global
4.2
7.7
“Towards ultimate …”
2.4
6.7
Angular error
Method
Yosemite sequence with clouds
Average error decreases, but standard deviation is still high….

48. Influence of each ingredient
For Yosemite sequence with clouds

49. Handling occlusions
bilateral filtering of flow: preserve intensity and flow discontinuities;
model occlusions
[J.Xiao, H.Cheng, H.Sawhney, C.Rao, M.Isnardi, ECCV 2006]
Bilateral filtering-based optical flow estimation with occlusion detection

50. Qualitative results

51. Combining local and global Bilateral+occlusions
Quantitative results
Average
Standard Deviation
Lucas&Kanade
4.3
(density 35%)
Horn&Schunk
9.8
16.2
Combining local and global
4.2
7.7
“Towards ultimate …”
2.4
6.7
Bilateral+occlusions
2.6
6.1
Angular error
Method
Yosemite sequence with clouds

52. Outline Ingredients for an accurate and robust optical flow
How people combine these ingredients
Fast algorithms
Energy functional => discrete equation
Multigrid solver: nearly real-time

53. Euler-Lagrange equation
How to minimize energy
Analogy:
Necessary condition:
Necessary condition
Euler-Lagrange equation

54. An example
Let us see how to derive discretized equation for 1D Horn & Schuhck
Horn&Schunk
1D version (simplified):

55. Iterative minimization (simple example)
Euler-Lagrange
Linear system of equation for u
Discretized:
Local iterations:

56. Life is not a picnic Linear discretized system
Non-linear in u, non-linear discretized system
Even more complicated

57. Optimization algorithms
Simple iterative minimization
Multigrid: much faster convergence

58. Solving the system How to solve? Start with some initial guess
and apply some iterative method
fast convergence
good initial guess
2 components of success:

59. Relaxation smoothes the error
Relaxation schemes have smoothing property:
It may take thousands of iterations to propagate information to large distance
oscillatory modes of the error are eliminated effectively, but smooth modes are damped slowly
Only neighboring pixels are coupled in relaxation scheme

60. Relaxation smoothes the error Examples
1D case:
2D case:
Error of initial guess
Error after 5 relaxation
Error after 15 relaxations

61. Idea: coarser grid On a coarser grid low frequencies become higher
initial grid – fine grid
On a coarser grid low frequencies become higher
Hence, relaxations can be more effective
coarse grid – we take every second point

62. Multigrid 2-Level V-Cycle
5. Correct the previous solution
6. Iterate ⇒ remove interpolation artifacts
1. Iterate ⇒ error becomes smooth
2. Transfer error equation to the coarse level ⇒ low frequencies become high
4. Transfer error to the fine level
3. Solve for the error on the coarse level ⇒ good error estimation

63. Coarse grid - advantages
Coarsening allows:
make iteration process faster (on the coarse grid we can effectively minimize the error)
obtain better initial guess (solve directly on the coarsest grid)
go to the coarsest grid
interpolate to the finer grid
solve here the equation
to find

64. Multigrid approach – Full scheme

65. Non-linear: Full Approximation Scheme
A(·) non-linear
Linear
Non-linear
fine level:
coarse level:
Difference from the linear case:
Equation for error involves current solution u0:
⇒ Need to transfer current solution to the coarser level

66. Multigrid: Summary Used to solve linear or non-linear equations
Method: combine two techniques
Basic iterative solver: quickly removes high frequencies of the error
Coarsening: makes low frequencies high
Contribution: fast minimization of loosely coupled equations

67. Fast Optimization: Results
Time [sec]
frames/sec
Gauss-Seidel
1.15
0.87
Full multigrid
0.016
62.8
9.52
0.11
FAS-multigrid
0.087
11.5
34.5
0.03
0.396
2.5
Horn&Schunck
CLG
“Towards ultimate…”
image size: 160 x 120

68. Summary of the Talk
25 years of Optical Flow : a lot of useful ingredients were developed:
local constraints:
brightness constancy
gradient constancy
smoothing techniques:
homogeneous
flow-driven (preserving discontinuities)
bilateral filters
handling of occlusions
robust functions
multiscale
All ingredients are combined an a global Energy Minimization approach
This difficult global optimization can be done very fast using Multigrid

69. Thank you!