2. Computer Vision Optical Flow Marc Pollefeys COMP 256 Some slides and illustrations from L. Van Gool, T. Darell, B. Horn, Y. Weiss, P. Anandan, M. Black, K. Toyama

3. Computer Vision last week: polar rectification

4. Computer Vision Last week: polar rectification

5. Computer Vision Last week: Stereo matching Optimal path (dynamic programming ) Similarity measure (SSD or NCC) Constraints epipolar ordering uniqueness disparity limit disparity gradient limit Trade-off Matching cost (data) Discontinuities (prior) (Cox et al. CVGIP’96; Koch’96; Falkenhagen´97; Van Meerbergen,Vergauwen,Pollefeys,VanGool IJCV‘02)

6. Computer Vision Questions for assignment?

7. Computer Vision Aug 26/28-Introduction Sep 2/4CamerasRadiometry Sep 9/11Sources & ShadowsColor Sep 16/18Linear filters & edges(Isabel hurricane) Sep 23/25Pyramids & TextureMulti-View Geometry Sep30/Oct2StereoProject proposals Oct 7/9Tracking (Welch)Optical flow Oct 14/16-- Oct 21/23Silhouettes/carving(Fall break) Oct 28/30-Structure from motion Nov 4/6Project updateCamera calibration Nov 11/13SegmentationFitting Nov 18/20Prob. segm.&fit.Matching templates Nov 25/27Matching relations(Thanksgiving) Dec 2/4Range dataFinal project Tentative class schedule

8. Computer Vision Optical Flow Brightness Constancy The Aperture problem Regularization Lucas-Kanade Coarse-to-fine Parametric motion models Direct depth SSD tracking Robust flow Bayesian flow

9. Computer Vision Optical Flow: Where do pixels move to?

10. Computer Vision Motion is a basic cue Motion can be the only cue for segmentation

11. Computer Vision Motion is a basic cue Motion can be the only cue for segmentation

12. Computer Vision Motion is a basic cue Even impoverished motion data can elicit a strong percept

13. Computer Vision Applications tracking structure from motion motion segmentation stabilization compression mosaicing …

14. Computer Vision Optical Flow Brightness Constancy The Aperture problem Regularization Lucas-Kanade Coarse-to-fine Parametric motion models Direct depth SSD tracking Robust flow Bayesian flow

15. Computer Vision Definition of optical flow OPTICAL FLOW = apparent motion of brightness patterns Ideally, the optical flow is the projection of the three-dimensional velocity vectors on the image 

16. Computer Vision Caution required ! Two examples : 1. Uniform, rotating sphere  O.F. = 0 2. No motion, but changing lighting  O.F.  0 

17. Computer Vision Caution required !

18. Computer Vision Mathematical formulation I (x,y,t) = brightness at (x,y) at time t Optical flow constraint equation :  Brightness constancy assumption:

19. Computer Vision Optical Flow Brightness Constancy The Aperture problem Regularization Lucas-Kanade Coarse-to-fine Parametric motion models Direct depth SSD tracking Robust flow Bayesian flow

20. Computer Vision The aperture problem 1 equation in 2 unknowns 

21. Computer Vision The aperture problem 0

22. Computer Vision

23. Computer Vision The aperture problem

24. Computer Vision Remarks

25. Computer Vision Apparently an aperture problem 

26. Computer Vision What is Optic Flow, anyway? Estimate of observed projected motion field Not always well defined! Compare: –Motion Field (or Scene Flow) projection of 3-D motion field –Normal Flow observed tangent motion –Optic Flow apparent motion of the brightness pattern (hopefully equal to motion field) Consider Barber pole illusion

27. Computer Vision

28. Computer Vision Optical Flow Brightness Constancy The Aperture problem Regularization Lucas-Kanade Coarse-to-fine Parametric motion models Direct depth SSD tracking Robust flow Bayesian flow

29. Computer Vision Horn & Schunck algorithm Additional smoothness constraint : besides OF constraint equation term minimize e s + e c 

30. Computer Vision The calculus of variations look for functions that extremize functionals and 

31. Computer Vision Calculus of variations Suppose 1. f(x) is a solution 2.  (x) is a test function with  (x 1 )= 0 and  (x 2 ) = 0 for the optimum : 

32. Computer Vision Calculus of variations Using integration by parts : Therefore regardless of  (x), then Euler-Lagrange equation 

33. Computer Vision Calculus of variations Generalizations 1. Simultaneous Euler-Lagrange equations : 2. 2 independent variables x and y 

34. Computer Vision Hence Now by Gauss’s integral theorem, so that = 0 Calculus of variations 

35. Computer Vision Calculus of variations Given the boundary conditions, Consequently, for all test functions . is the Euler-Lagrange equation 

36. Computer Vision Horn & Schunck The Euler-Lagrange equations : In our case, so the Euler-Lagrange equations are is the Laplacian operator 

37. Computer Vision Horn & Schunck Remarks : 1. Coupled PDEs solved using iterative methods and finite differences 2. More than two frames allow a better estimation of I t 3. Information spreads from corner-type patterns 

38. Computer Vision 

39. Computer Vision Horn & Schunck, remarks 1. Errors at boundaries 2. Example of regularisation (selection principle for the solution of illposed problems) 

40. Computer Vision Results of an enhanced system

41. Computer Vision Structure from motion with OF

42. Computer Vision Optical Flow Brightness Constancy The Aperture problem Regularization Lucas-Kanade Coarse-to-fine Parametric motion models Direct depth SSD tracking Robust flow Bayesian flow

43. Computer Vision

44. Computer Vision

45. Computer Vision

46. Computer Vision

47. Computer Vision

48. Computer Vision Optical Flow Brightness Constancy The Aperture problem Regularization Lucas-Kanade Coarse-to-fine Parametric motion models Direct depth SSD tracking Robust flow Bayesian flow

49. Computer Vision

50. Computer Vision

51. Computer Vision

52. Computer Vision Optical Flow Brightness Constancy The Aperture problem Regularization Lucas-Kanade Coarse-to-fine Parametric motion models Direct depth SSD tracking Robust flow Bayesian flow

53. Computer Vision

54. Computer Vision

55. Computer Vision

56. Computer Vision

57. Computer Vision

58. Computer Vision

59. Computer Vision

60. Computer Vision

61. Computer Vision

62. Computer Vision

63. Computer Vision

64. Computer Vision

65. Computer Vision

66. Computer Vision

67. Computer Vision

68. Computer Vision

69. Computer Vision

70. Computer Vision

71. Computer Vision

72. Computer Vision

73. Computer Vision

74. Computer Vision

75. Computer Vision

76. Computer Vision

77. Computer Vision

78. Computer Vision

79. Computer Vision

80. Computer Vision

81. Computer Vision

82. Computer Vision Optical Flow Brightness Constancy The Aperture problem Regularization Lucas-Kanade Coarse-to-fine Parametric motion models Direct depth SSD tracking Robust flow Bayesian flow

83. Computer Vision

84. Computer Vision

85. Computer Vision

86. Computer Vision Optical Flow Brightness Constancy The Aperture problem Regularization Lucas-Kanade Coarse-to-fine Parametric motion models Direct depth SSD tracking Robust flow Bayesian flow

87. Computer Vision

88. Computer Vision

89. Computer Vision

90. Computer Vision

91. Computer Vision

92. Computer Vision Optical Flow Brightness Constancy The Aperture problem Regularization Lucas-Kanade Coarse-to-fine Parametric motion models Direct depth SSD tracking Robust flow Bayesian flow

93. Computer Vision

94. Computer Vision

95. Computer Vision

96. Computer Vision

97. Computer Vision

98. Computer Vision

99. Computer Vision Optical Flow Brightness Constancy The Aperture problem Regularization Lucas-Kanade Coarse-to-fine Parametric motion models Direct depth SSD tracking Robust flow Bayesian flow

100. Computer Vision

101. Computer Vision Rhombus Displays http://www.cs.huji.ac.il/~yweiss/Rhombus/

102. Computer Vision

103. Computer Vision

104. Computer Vision

105. Computer Vision

106. Computer Vision

107. Computer Vision