1. Motion
Extraction 

2. Motion is a basic cue Motion can be the only cue for segmentation
Biologically favoured because of camouflage

3. Motion is a basic cue
… which set in motion a constant, evolutionary race,
encouraging animals to sit still, only moving in bursts

4. Motion is a basic cue
Motion can be the only cue for segmentation

5. Motion is a basic cue
Even impoverished motion data can elicit a strong percept

6. Some applications of motion extraction
Change / shot cut detection
Surveillance / traffic monitoring
Autonomous driving
Analyzing game dynamics in sports
Motion capture / gesture analysis (HCI)
Image stabilisation
Motion compensation (e.g. medical robotics)
Feature tracking for 3D reconstruction
Etc. ! 

7. Shot cut detection & Keyframes

8. Human-Machine Interfacing

9. 3D: Structure-from-Motion
Tracked Points gives correspondences

10. Temple of the Masks, Edzna, Mexico
3D: Structure-from-Motion
Temple of the Masks, Edzna, Mexico

11.
Asymmetric patterns like non-aligned tiles or mezzanines cannot be handled by facade structure detection
Thick window frames will be wrongly interpreted as a split i.e. The user has to reverse the split manually
Heavy image noise or small irregular elements on non-repetitive tiles causes incorrect splits

12. in this lecture... Several techniques, but…
this lecture is restricted to the
1. detection of the “optical flow”
2. tracking with the “Condensation filter” 

13. optical flow 

14. Definition of optical flow
Ideally, the optical flow is the projection of the 3D velocity vectors onto the image
A motion vector is sought at every pixel of the
image
OF = apparent motion of
brightness patterns
In practice 

15. OF may fail to capture the real motion !
Two examples :
1. Uniform, rotating sphere ⇓
O.F. = 0
2. No motion, but changing lighting ⇓
O.F. ≠ 0 

16. Caution required !

17. Qualitative formulation
Suppose a point of the scene projects to a
certain pixel of the current video frame. Our
task is to figure out to which pixel in the next
frame it moves…
That question needs answering for all pixels of
the current image.
In order to find these corresponding pixels, we
need to come up with a reasonable assumption
on how we can detect them among the many.
We assume these corresponding pixels have
the same intensities as the pixels the scene
points came from in the previous frame.
That will only hold approximately… 

18. Mathematical formulation
I (x,y,t) = brightness at (x,y) at time t
Optical flow constraint equation :
This equation states that if one were to track the image
projections of a physical point through the video, it would
not change its intensity… This tends to be true over short
lapses of time. (Note the different types of time derivatives!) 

19. Mathematical formulation
We will use as
shorthand
notation for 

20. I, but that u and v are unknown
The aperture problem
Note that we can measure the 3 derivatives of
I, but that u and v are unknown
1 equation in 2 unknowns 

21. The aperture problem Aperture problem : only the component along the
gradient can be retrieved 

22. The aperture problem

23. Remarks

24. Remarks 1. The underdetermined nature could be
solved using higher derivatives of intensity
2. for some intensity patterns, e.g. patches with
a planar intensity profile, the aperture problem
cannot be resolved anyway.
For many images, large parts have planar intensity
profiles… higher-order derivatives than 1st order are
typically not used (also because they are noisy) 

25. Horn & Schunck algorithm
Breaking the spell via an …
additional smoothness constraint :
to be minimized,
besides the OF constraint equation term
minimize es+λec
(also reduces influence of noise) 

26. The calculus of variations
look for functions that extremize functionals
(a functional is a function that takes a vector as its input
argument, and returns a scalar)
like for our functional:
what are the optimal u(x,y) and v(x,y) ? 

27. The calculus of variations
look for functions that extremize functionals
with ,
and 

28. Calculus of variations
Suppose
1. f(x) is a solution
2. η(x) is a test function with η(x1)= 0
and η(x2) = 0
We then consider
Rationale: supposed f is the solution, then any deviation should result in a worse I;
when applying classical optimization over the values of the scalar e the optimum should be e = 0 

29. Calculus of variations
Suppose
1. f(x) is a solution
2. η(x) is a test function with η(x1)= 0
and η(x2) = 0
We then consider
With this trick, we reformulate an optimization
over a function into a classical optimization over
a scalar… a problem we know how to solve 

30. Calculus of variations
Suppose
1. f(x) is a solution
2. η(x) is a test function with η(x1)= 0
and η(x2) = 0
for the optimum :
Around the optimum, the
derivative should be zero 

31. Calculus of variations
Suppose
1. f(x) is a solution
2. η(x) is a test function with η(x1)= 0
and η(x2) = 0
for the optimum :
with
with 

32. Calculus of variations
Using integration by parts:
Using integration by parts :
where 

33. Calculus of variations
Using integration by parts : 

34. Calculus of variations
Using integration by parts :
Therefore
regardless of η(x), then
Euler-Lagrange equation 

35. Calculus of variations
Generalizations 1.
Simultaneous Euler-Lagrange equations,
i.c. one for u and one for v :
independent variables x and y 

36. Calculus of variations
Hence
Now by Gauss’s integral theorem,
such that
= 0 

37. Calculus of variations

38. Calculus of variations
Consequently,
for all test functions η , thus
is the Euler-Lagrange equation 

39. Horn & Schunck The Euler-Lagrange equations : In our case ,
so the Euler-Lagrange equations are
is the Laplacian operator 

40. Horn & Schunck i i Remarks : 1. Coupled PDEs solved using iterative
methods and finite differences (iteration i) i i
2. More than two frames allow for a better
estimation of It
3. Information spreads from edge- and corner-type
patterns 

41. 

42. 1. Errors at object boundaries
Horn & Schunck, remarks
1. Errors at object boundaries
2. Example of regularisation
(selection principle for the solution of
ill-posed problems by imposing an extra
generic constraint, like here smoothness) 

43. condensation filter 

44. condensation filter as an example of a `tracker’,
shifting the emphasis from
pixels to objects… 

45. Condensation tracker state vector noise in system model
observation vector
noise in measurement
model
System. M.
Measur. M.

46. Condensation tracker 1. Prediction , based on the system model
( f = system transition function)
2. Update , based on the measurement model
( h = measurement function)
is the history of observations

47. Condensation tracker Example System model position velocity
Measurement model

48. Condensation tracker Recursive Bayesian filter
Object not as a single state but a prob. distribution

49. Condensation tracker Recursive Bayesian filter
Object not as a single state but a prob. Distribution
(p here means probability…)
1. Prediction
2. Update

50. Condensation tracker Recursive Bayesian filter
Object not as a single state but a prob. distribution
1. Prediction
2. Update
can be considered a normalization factor

51. Condensation tracker Recursive Bayesian filter
Object not as a single state but a prob. distribution
Bayes´ rule
2. Update
can be considered a normalization factor

52. Condensation tracker Recursive Bayesian filter
Object not as a single state but a prob. distribution
Bayes´ rule
2. Update
can be considered a normalization factor

53. Condensation tracker Recursive Bayesian filter
Object not as a single state but a prob. distribution
1. Prediction
2. Update

54. Condensation tracker Example cont’d a posteriori prob. distr. at t-1
prediction
a priori prob.
distr. at t
update

55. Condensation tracker Recursive Bayesian filter Calculating
numerically is very time consuming, and the prob.
distributions have to be known…
Analytic solutions are only available for the simplest
of cases, e.g. when distr. are Gaussian and the
system and measurement models are linear…
(Kalman filter, Kalman was prof. at ETH, D-ITET)

56. Condensation tracker Recursive Bayesian filter Calculating
numerically is very time consuming, and the prob.
distributions have to be known…
Analytic solutions are only available for the simplest
of cases, e.g. when distr. are Gaussian and the
system and measurement models are linear…
That’s where CONDENSATION comes in, acronym for
CONditional DENSity propagATION

57. distributions GaussianK A L M N F I LT E R
In our example
model is linear,
distributions Gaussian
System model
Measurement model

58. K A L M N F I LT E R

59. Condensation tracker The probability distribution is represented by a
sample set S (set of selected states s)
With a weight determining the sampling
probability

60. Condensation tracker 1. prediction
Start with , the sample set of the previous
step, and apply the system model to each sample,
yielding predicted samples
2. update
Sample from the predicted set, where samples
are drawn with replacement and with probability
(i.e. using meas. model)
In the limit (large N) equivalent to Bayesian tracker

61. Condensation tracker
prediction
weighing
update

62. Condensation tracker NOTE Sample may be drawn multiple times, but
different noise will yield different predictions
under the system model, for samples
corresponding to the same state after drawing.
This diversification through noise is important,
as otherwise fewer and fewer different samples
would survive

63. C O N D E S A T I
Example cont’d
prediction
weighing with
update

64. Condensation tracker Comparison with Kalman filter
Condensation Kalman-Bucy
Unrestricted system models
Unrestricted noise models
Multiple hypotheses
Discretisation error
Postprocessing for interpret.
Linear system models
Gaussian noise
Unimodal
Exact solution
Direct interpretation

65. Condensation tracker

66. Condensation tracker

67. Condensation tracker

68. Condensation tracker

69. Condensation tracker

70. Condensation tracker

71. Condensation tracker

72. Condensation tracker

73. Condensation tracker

74. Condensation tracker

75. Condensation tracker Elliptical region with prescribed color histogram
System model
position
velocity
size
size chance

76. where p and q are the color histograms
Condensation tracker
Measurement model
with
where p and q are the color histograms
of a sample and the target, resp.

77. Condensation tracker

78. Mean shift tracker

79. Mean shift tracker

80. Condensation tracker

81. Other approaches 1. Model-based tracking (application-specific)
- active contours (discussed with segmentation)
- analysis/synthesis schemes
2. Feature tracking (more generic)
- corner tracking (shown when we discuss 3D)
- blob/contour tracking
- intensity profile tracking
- region tracking 

82. Model-based tracker
(EPFL)

83. Model-based tracker
(EPFL)

84. Motion capture for special effects