2. Optical Flow

3. Brightness Constancy The Aperture problem Regularization Lucas-Kanade Coarse-to-fine Parametric motion models Direct depth SSD tracking Robust flow Textured motion

4. Optical Flow: Where do pixels move to?

5. Motion is a basic cue Motion can be the only cue for segmentation

6. Applications tracking structure from motion motion segmentation stabilization compression mosaicing …

7. Optical Flow Brightness Constancy The Aperture problem Regularization Lucas-Kanade Coarse-to-fine Parametric motion models Direct depth SSD tracking Robust flow Textured motion

8. Definition of optical flow OPTICAL FLOW = apparent motion of brightness patterns Ideally, the optical flow is the projection of the three-dimensional velocity vectors on the image 

9. Caution required ! Two examples : 1. Uniform, rotating sphere  O.F. = 0 2. No motion, but changing lighting  O.F.  0 

10. Mathematical formulation I (x,y,t) = brightness at (x,y) at time t Optical flow constraint equation :  Brightness constancy assumption:

11. Optical Flow Brightness Constancy The Aperture problem Regularization Lucas-Kanade Coarse-to-fine Parametric motion models Direct depth SSD tracking Robust flow Textured motion

12. The aperture problem 1 equation in 2 unknowns 

13. The Aperture Problem 0

15. What is Optical Flow, anyway? Estimate of observed projected motion field Not always well defined! Compare: Motion Field (or Scene Flow) projection of 3-D motion field Normal Flow observed tangent motion Optical Flow apparent motion of the brightness pattern (hopefully equal to motion field) Consider Barber pole illusion

17. Optical Flow Brightness Constancy The Aperture problem Regularization Lucas-Kanade Coarse-to-fine Parametric motion models Direct depth SSD tracking Robust flow Textured motion

18. Horn & Schunck algorithm Additional smoothness constraint : besides OF constraint equation term minimize e s + e c 

19. The calculus of variations look for functions that extremize functionals and 

20. Calculus of variations Suppose 1. f(x) is a solution 2.  (x) is a test function with  (x 1 )= 0 and  (x 2 ) = 0 for the optimum : 

21. Calculus of variations Using integration by parts : Therefore regardless of  (x), then Euler-Lagrange equation 

22. Calculus of variations Generalizations 1. Simultaneous Euler-Lagrange equations : 2. 2 independent variables x and y 

23. Hence Now by Gauss’s integral theorem, so that = 0 Calculus of variations 

24. Given the boundary conditions, Consequently, for all test functions . is the Euler-Lagrange equation 

25. Horn & Schunck The Euler-Lagrange equations : In our case, so the Euler-Lagrange equations are is the Laplacian operator 

26. Horn & Schunck Remarks : 1. Coupled PDEs solved using iterative methods and finite differences 2. More than two frames allow a better estimation of I t 3. Information spreads from corner-type patterns 

27. 

28. Horn & Schunck, remarks 1. Errors at boundaries 2. Example of regularization (selection principle for the solution of ill-posed problems) 

29. Results of an enhanced system

30. Structure from motion with OF

31. Optical Flow Brightness Constancy The Aperture problem Regularization Lucas-Kanade Coarse-to-fine Parametric motion models Direct depth SSD tracking Robust flow Textured motion

36. Optical Flow Brightness Constancy The Aperture problem Regularization Lucas-Kanade Coarse-to-fine Parametric motion models Direct depth SSD tracking Robust flow Textured motion

40. Optical Flow Brightness Constancy The Aperture problem Regularization Lucas-Kanade Coarse-to-fine Parametric motion models Direct depth SSD tracking Robust flow Textured motion

70. Optical Flow Brightness Constancy The Aperture problem Regularization Lucas-Kanade Coarse-to-fine Parametric motion models Direct depth SSD tracking Robust flow Bayesian flow

74. Optical Flow Brightness Constancy The Aperture problem Regularization Lucas-Kanade Coarse-to-fine Parametric motion models Direct depth SSD tracking Robust flow Textured motion

80. Optical Flow Brightness Constancy The Aperture problem Regularization Lucas-Kanade Coarse-to-fine Parametric motion models Direct depth SSD tracking Robust flow Textured motion

87. Optical Snow http://www.cim.mcgill.ca/~langer/research-optical- snow.html Optical snow is the type of motion an observer sees when watching a snow fall. Flakes that are closer to the observer appear to move faster than flakes which are farther away.

88. Textured Motion http://civs.stat.ucla.edu/Yizhou_Research/Te xturedmotion.htm http://civs.stat.ucla.edu/Yizhou_Research/Te xturedmotion.htm Natural scenes contain rich stochastic motion patterns which are characterized by the movement of a large amount of particle and wave elements, such as falling snow, water waves, dancing grass, etc. We call these motion patterns "textured motion".