2. Computer Vision Optical Flow Marc Pollefeys COMP 256 Some slides and illustrations from L. Van Gool, T. Darell, B. Horn, Y. Weiss, P. Anandan, M. Black, K. Toyama

3. Computer Vision 2 last week: polar rectification

4. Computer Vision 3 Last week: polar rectification

5. Computer Vision 4 Stereo matching Optimal path (dynamic programming ) Similarity measure (SSD or NCC) Constraints epipolar ordering uniqueness disparity limit disparity gradient limit Trade-off Matching cost (data) Discontinuities (prior) (Cox et al. CVGIP’96; Koch’96; Falkenhagen´97; Van Meerbergen,Vergauwen,Pollefeys,VanGool IJCV‘02)

6. Computer Vision 5 Hierarchical stereo matching Downsampling (Gaussian pyramid) Disparity propagation Allows faster computation Deals with large disparity ranges ( Falkenhagen´97;Van Meerbergen,Vergauwen,Pollefeys,VanGool IJCV‘02)

7. Computer Vision 6 Disparity map image I(x,y) image I´(x´,y´) Disparity map D(x,y) (x´,y´)=(x+D(x,y),y)

8. Computer Vision 7 Example: reconstruct image from neighboring images

9. Computer Vision 8 Multi-view depth fusion Compute depth for every pixel of reference image –Triangulation –Use multiple views –Up- and down sequence –Use Kalman filter (Koch, Pollefeys and Van Gool. ECCV‘98) Allows to compute robust texture

10. Computer Vision 9 Real-time stereo on graphics hardware Computes Sum-of-Square-Differences Hardware mip-map generation used to aggregate results over support region Trade-off between small and large support window Yang and Pollefeys CVPR03 140M disparity hypothesis/sec on Radeon 9700pro e.g. 512x512x20disparities at 30Hz Shape of a kernel for summing up 6 levels

11. Computer Vision 10 Sample Re-Projections near far

12. Computer Vision 11 (1x1) (1x1+2x2) (1x1+2x2 +4x4+8x8) (1x1+2x2 +4x4+8x8 +16x16) Combine multiple aggregation windows using hardware mipmap and multiple texture units in single pass video

13. Computer Vision 12 Cool ideas Space-time stereo (varying illumination, not shape)

14. Computer Vision 13 More on stereo … The Middleburry Stereo Vision Research Page http://cat.middlebury.edu/stereo/ Recommended reading D. Scharstein and R. Szeliski. A Taxonomy and Evaluation of Dense Two-Frame Stereo Correspondence Algorithms. IJCV 47(1/2/3):7-42, April-June 2002. PDF file (1.15 MB) - includes current evaluation. Microsoft Research Technical Report MSR-TR-2001-81, November 2001. PDF file (1.27 MB).PDF file

15. Computer Vision 14 Jan 16/18-Introduction Jan 23/25CamerasRadiometry Jan 30/Feb1Sources & ShadowsColor Feb 6/8Linear filters & edgesTexture Feb 13/15Multi-View GeometryStereo Feb 20/22Optical flowProject proposals Feb27/Mar1Affine SfMProjective SfM Mar 6/8Camera CalibrationSilhouettes and Photoconsistency Mar 13/15Springbreak Mar 20/22SegmentationFitting Mar 27/29Prob. SegmentationProject Update Apr 3/5Tracking Apr 10/12Object Recognition Apr 17/19Range data Apr 24/26Final project Tentative class schedule

16. Computer Vision 15 Optical Flow Brightness Constancy The Aperture problem Regularization Lucas-Kanade Coarse-to-fine Parametric motion models Direct depth SSD tracking Robust flow Bayesian flow

17. Computer Vision 16 Optical Flow: Where do pixels move to?

18. Computer Vision 17 Motion is a basic cue Motion can be the only cue for segmentation

19. Computer Vision 18 Motion is a basic cue Motion can be the only cue for segmentation

20. Computer Vision 19 Motion is a basic cue Even impoverished motion data can elicit a strong percept

21. Computer Vision 20 Applications tracking structure from motion motion segmentation stabilization compression Mosaicing …

22. Computer Vision 21 Optical Flow Brightness Constancy The Aperture problem Regularization Lucas-Kanade Coarse-to-fine Parametric motion models Direct depth SSD tracking Robust flow Bayesian flow

23. Computer Vision 22 Definition of optical flow OPTICAL FLOW = apparent motion of brightness patterns Ideally, the optical flow is the projection of the three-dimensional velocity vectors on the image 

24. Computer Vision 23 Caution required ! Two examples : 1. Uniform, rotating sphere  O.F. = 0 2. No motion, but changing lighting  O.F.  0 

25. Computer Vision 24 Caution required !

26. Computer Vision 25 Mathematical formulation I (x,y,t) = brightness at (x,y) at time t Optical flow constraint equation :  Brightness constancy assumption:

27. Computer Vision 26 Optical Flow Brightness Constancy The Aperture problem Regularization Lucas-Kanade Coarse-to-fine Parametric motion models Direct depth SSD tracking Robust flow Bayesian flow

28. Computer Vision 27 The aperture problem 1 equation in 2 unknowns 

29. Computer Vision 28 The aperture problem 0

30. Computer Vision 29

31. Computer Vision 30 The aperture problem

32. Computer Vision 31 Remarks

33. Computer Vision 32 Apparently an aperture problem 

34. Computer Vision 33 What is Optic Flow, anyway? Estimate of observed projected motion field Not always well defined! Compare: –Motion Field (or Scene Flow) projection of 3-D motion field –Normal Flow observed tangent motion –Optic Flow apparent motion of the brightness pattern (hopefully equal to motion field) Consider Barber pole illusion

35. Computer Vision 34

36. Computer Vision 35 Optical Flow Brightness Constancy The Aperture problem Regularization Lucas-Kanade Coarse-to-fine Parametric motion models Direct depth SSD tracking Robust flow Bayesian flow

37. Computer Vision 36 Horn & Schunck algorithm Additional smoothness constraint : besides OF constraint equation term minimize e s + e c 

38. Computer Vision 37 Horn & Schunck The Euler-Lagrange equations : In our case, so the Euler-Lagrange equations are is the Laplacian operator 

39. Computer Vision 38 Horn & Schunck Remarks : 1. Coupled PDEs solved using iterative methods and finite differences 2. More than two frames allow a better estimation of I t 3. Information spreads from corner-type patterns 

40. Computer Vision 39 

41. Computer Vision 40 Horn & Schunck, remarks 1. Errors at boundaries 2. Example of regularisation (selection principle for the solution of illposed problems) 

42. Computer Vision 41 Results of an enhanced system

43. Computer Vision 42 Structure from motion with OF

44. Computer Vision 43 Optical Flow Brightness Constancy The Aperture problem Regularization Lucas-Kanade Coarse-to-fine Parametric motion models Direct depth SSD tracking Robust flow Bayesian flow

45. Computer Vision 44

46. Computer Vision 45

47. Computer Vision 46

48. Computer Vision 47

49. Computer Vision 48

50. Computer Vision 49 Optical Flow Brightness Constancy The Aperture problem Regularization Lucas-Kanade Coarse-to-fine Parametric motion models Direct depth SSD tracking Robust flow Bayesian flow

51. Computer Vision 50

52. Computer Vision 51

53. Computer Vision 52

54. Computer Vision 53 Optical Flow Brightness Constancy The Aperture problem Regularization Lucas-Kanade Coarse-to-fine Parametric motion models Direct depth SSD tracking Robust flow Bayesian flow

55. Computer Vision 54

56. Computer Vision 55

57. Computer Vision 56

58. Computer Vision 57

59. Computer Vision 58

60. Computer Vision 59

61. Computer Vision 60

62. Computer Vision 61

63. Computer Vision 62

64. Computer Vision 63

65. Computer Vision 64

66. Computer Vision 65

67. Computer Vision 66

68. Computer Vision 67

69. Computer Vision 68

70. Computer Vision 69

71. Computer Vision 70

72. Computer Vision 71

73. Computer Vision 72

74. Computer Vision 73

75. Computer Vision 74

76. Computer Vision 75

77. Computer Vision 76

78. Computer Vision 77

79. Computer Vision 78

80. Computer Vision 79

81. Computer Vision 80

82. Computer Vision 81

83. Computer Vision 82

84. Computer Vision 83 Optical Flow Brightness Constancy The Aperture problem Regularization Lucas-Kanade Coarse-to-fine Parametric motion models Direct depth SSD tracking Robust flow Bayesian flow

85. Computer Vision 84

86. Computer Vision 85

87. Computer Vision 86

88. Computer Vision 87 Optical Flow Brightness Constancy The Aperture problem Regularization Lucas-Kanade Coarse-to-fine Parametric motion models Direct depth SSD tracking Robust flow Bayesian flow

89. Computer Vision 88

90. Computer Vision 89

91. Computer Vision 90

92. Computer Vision 91

93. Computer Vision 92

94. Computer Vision 93 Optical Flow Brightness Constancy The Aperture problem Regularization Lucas-Kanade Coarse-to-fine Parametric motion models Direct depth SSD tracking Robust flow Bayesian flow

95. Computer Vision 94

96. Computer Vision 95

97. Computer Vision 96

98. Computer Vision 97

99. Computer Vision 98

100. Computer Vision 99

101. Computer Vision 100 Optical Flow Brightness Constancy The Aperture problem Regularization Lucas-Kanade Coarse-to-fine Parametric motion models Direct depth SSD tracking Robust flow Bayesian flow

102. Computer Vision 101

103. Computer Vision 102 Rhombus Displays http://www.cs.huji.ac.il/~yweiss/Rhombus/

104. Computer Vision 103

105. Computer Vision 104

106. Computer Vision 105

107. Computer Vision 106

108. Computer Vision 107

109. Computer Vision 108