1. 1 Recent progress in optical flow progress Presented by: Darya Frolova and Denis Simakov

2. 2 Optical Flow is not in favor Often not using Optical Flow is stated as one of the main advantages of a method Optical Flow methods have a reputation of either unreliable or slow Very popular slide: Optical Flow can be computed fast and accurately Recent works claim Recent works claim:

3. 3 Horn&Schunck Optical Flow Research: Timeline Lucas&Kanade 1981 1992 1998now Benchmark: Barron et.al. Benchmark: Galvin et.al. many methods more methods no significant improvement, but a lot of useful ingredients were developed Seminal papers

4. 4 In This Lecture We will describe : Ingredients for an accurate and robust optical flow How people combine these ingredients Fast algorithms Combining the advantages of local and global optic flow methods (“Lucas/Kanade meets Horn/Schunck”) A. Bruhn, J. Weickert, C. Schnörr, 2002 - 2005 High accuracy optical flow estimation based on a theory for warping T. Brox, A. Bruhn, N. Papenberg, J. Weickert, 2004 - 2005 Real-Time Optic Flow Computation with Variational Methods A. Bruhn, J. Weickert, C. Feddern, T. Kohlberger, C. Schnörr, 2003 - 2005 Towards ultimate motion estimation: Combining highest accuracy with real-time performance A. Bruhn, J. Weickert, 2005 Bilateral filtering-based optical flow estimation with occlusion detection J.Xiao, H.Cheng, H.Sawhney, C.Rao, M.Isnardi, 2006 Papers:

5. 5 What is Optical Flow? …

6. 6 Definitions optical flow [ Horn&Schunck ] The optical flow is a velocity field in the image which transforms one image into the next image in a sequence [ Horn&Schunck ] [ Horn&Schunck ] The motion field … is the projection into the image of three-dimensional motion vectors [ Horn&Schunck ] + = frame #1 frame #2 flow field

7. 7 Ambiguity of optical flow Frame 1 flow (1): true motion flow (2)

8. 8 Applications video compression 3D reconstruction segmentation object detection activity detection key frame extraction interpolation in time optical flow We are usually interested in actual motion motion field

9. 9 Outline Ingredients for an accurate and robust optical flow Local image constraints on motion Robust statistics Spatial coherence How people combine these ingredients Fast algorithms

10. 10 Local image constraints

11. 11 Brightness Constancy frame t frame t+1 v u

12. 12 Complex dependence on Linearize: Linearized brightness constancy Deviation from brightness constancy (we want to minimize it)

13. 13 Linearized brightness constancy J – “motion tensor”, or “structure tensor” Let us square the difference:

14. 14 Averaged linearized constraint J is a function of x, y (a matrix for every point) J * = Combine over small neighborhoods (more robust to noise):

15. 15 Method of Lucas&Kanade Solve independently for each point [ Lucas&Kanade 1981 ] linear system Can be solved for every point where matrix is not degenerate

16. 16 Lukas&Kanade - Results flow field Hamburg taxi Rubik cube

17. 17 Brightness is not always constant Rotating cylinder Brightness constancy does not always hold intensity position Gradient constancy holds intensity derivative position

18. 18 Local constraints - Summary We have seen brightness constancy linearized averaged linearized gradient constancy averaged linearized

19. 19 Local constraints are not enough!

20. 20 Local constraints work poorly input video Optical flow direction using only local constraints color encodes direction as marked on the boundary

21. 21 Where local constraints fail Uniform regions Motion is not observable in the image (locally)

22. 22 Where local constraints fail “Aperture problem” We can estimate only one flow component (normal)

23. 23 Where local constraints fail Occlusions We have not seen where some points moved Occluded regions are marked in red

24. 24 Obtaining support from neighbors Two main problems with local constraints: information about motion is missing in some points => need spatial coherency constraints do not hold everywhere => need methods to combine them robustly good missing wrong

25. 25 Robust combination of partially reliable data or: How to hold elections

26. 26 Toy example Find “best” representative for the set of numbers L2: L1: xixi Influence of x i on E : x i → x i + ∆ Outliers influence the most proportional to Majority decides equal for all x i

27. 27 Elections and robust statistics many ordinary people a very rich man Oligarchy Votes proportional to the wealth Democracy One vote per person wealth like in L2 norm minimization like in L1 norm minimization

28. 28 Combination of two flow constraints robust: L1 robust in presence of outliers – non-smooth: hard to analyze usual: L2 easy to analyze and minimize – sensitive to outliers robust regularized smooth: easy to analyze robust in presence of outliers ε [A. Bruhn, J. Weickert, 2005] Towards ultimate motion estimation: Combining highest accuracy with real-time performance

29. 29 Spatial Propagation

30. 30 Obtaining support from neighbors good missing wrong Two main problems with local constraints: information about motion is missing in some points => need spatial coherency constraints do not hold everywhere => need methods to combine them robustly

31. 31 Homogeneous propagation - flow in the x direction - flow in the y direction - gradient [Horn&Schunck 1981] This constraint is not correct on motion boundaries => over-smoothing of the resulting flow

32. 32 Robustness to flow discontinuities (also known as isotropic flow-driven regularization) [T. Brox, A. Bruhn, N. Papenberg, J. Weickert, 2004] High accuracy optical flow estimation based on a theory for warping ε

33. 33 Selective flow filtering [J.Xiao, H.Cheng, H.Sawhney, C.Rao, M.Isnardi, 2006] Bilateral filtering-based optical flow estimation with occlusion detection Solution: use “bilateral” filter in space, intensity, flow; taking into account occlusions We want to propagate information without crossing image and flow discontinuities from “good” points only (not occluded)

34. 34 Bilateral filter x I x I x I x I Unilateral (usual)Bilateral Preserves discontinuities! [C. Tomasi, R. Manduchi, 1998] Bilateral filtering for gray and color images.

35. 35 Using of bilateral filter - Example occluded regions cyan rectangle moves to the right and occludes background region marked by red [J.Xiao, H.Cheng, H.Sawhney, C.Rao, M.Isnardi, 2006] Bilateral filtering-based optical flow estimation with occlusion detection

36. 36 Learning of spatial coherence Come to the next lecture …

37. 37 Spatial coherence: Summary Homogeneous propagation - oversmoothing Robust statistics with homogeneous propagation - preserves flow discontinuities Bilateral filtering - combines information from regions with similar flow and similar intensities Handles occlusions

38. 38 Two more useful ingredients in brief – one slide each

39. 2D vs. 3D 2 frames: Several frames: Several frames allow more accurate optical flow estimation

40. 40 Multiscale Optical Flow (other names: “warping”, “coarse-to-fine”, “multiresolution”) pyramid for frame 1 Linearization : valid only for small flow + + frame 1 warped ? pyramid for frame 2 upsample

41. 41 Methods How to make tasty soup with these ingredients: several recipes

42. 42 Outline Ingredients for an accurate and robust optical flow How people combine these ingredients Lukas & Kanade meet Horn & Schunck The more ingredients – the better Bilateral filtering and occlusions Fast algorithms

43. 43 Combining ingredients ∫ϕ (∫ϕ ( Energy = ∫ϕ ( Data) + ∫ϕ ( “Smoothness”) Local constraints –Brightness constancy –Image gradient constancy Spatial coherency –Homogeneous –Flow-driven –Bilateral filtering + occlusions Combined using robust statistics Computed coarse-to-fine Use several frames

44. 44 Combining Local and Global Lucas&Kanade Horn&Schunk Basic “Combining local and global” [ A. Bruhn, J. Weickert, C. Schnörr, 2002 ] Remember:

45. 45 Sensitivity to noise – quantitative results ground truth flow frame t+1 frame t Error measure: angle between true and computed flow in (x,y,t) space

46. 46 The more ingredients - the better brightness constancy gradient constancy spatial coherence [ Bruhn, Weickert, 2005 ] Towards ultimate motion estimation: Combining highest accuracy with real-time performance

47. 47 Quantitative results Average Standard Deviation Lucas&Kanade 4.3 (density 35%) Horn&Schunk 9.816.2 Combining local and global 4.27.7 “Towards ultimate …” 2.46.7 Angular error Method Yosemite sequence with clouds Average error decreases, but standard deviation is still high….

48. 48 Influence of each ingredient For Yosemite sequence with clouds

49. 49 Handling occlusions bilateral filtering bilateral filtering of flow: preserve intensity and flow discontinuities; model occlusions [J.Xiao, H.Cheng, H.Sawhney, C.Rao, M.Isnardi, ECCV 2006] Bilateral filtering-based optical flow estimation with occlusion detection

50. 50 Qualitative results

51. 51 Quantitative results Average Standard Deviation Lucas&Kanade 4.3 (density 35%) Horn&Schunk 9.816.2 Combining local and global 4.27.7 “Towards ultimate …” 2.46.7 Bilateral+occlusions 2.66.1 Angular error Method Yosemite sequence with clouds

52. 52 Outline Ingredients for an accurate and robust optical flow How people combine these ingredients Fast algorithms Energy functional => discrete equation Multigrid solver: nearly real-time

53. 53 How to minimize energy Necessary condition: Necessary condition Analogy: Euler-Lagrange equation

54. 54 An example Horn&Schunk Let us see how to derive discretized equation for 1D Horn & Schuhck 1D version (simplified):

55. 55 Iterative minimization (simple example) Euler-Lagrange Discretized: Local iterations: Linear system of equation for u

56. 56 Life is not a picnic Linear discretized system Even more complicated Non-linear in u, non-linear discretized system

57. 57 Optimization algorithms Simple iterative minimization Multigrid : much faster convergence

58. 58 Solving the system How to solve? 2 components of success: fast convergence good initial guess Start with some initial guess and apply some iterative method

59. 59 smoothing Relaxation schemes have smoothing property: Only neighboring pixels are coupled in relaxation scheme It may take thousands of iterations to propagate information to large distance...... Relaxation smoothes the error

60. 60 Relaxation smoothes the error Examples 2D case: 1D case: Error of initial guess Error after 5 relaxation Error after 15 relaxations

61. 61 Idea: coarser grid On a coarser grid low frequencies become higher Hence, relaxations can be more effective initial grid – fine grid coarse grid – we take every second point

62. 62 Multigrid 2-Level V-Cycle 1. Iterate ⇒ error becomes smooth 2. Transfer error equation to the coarse level ⇒ low frequencies become high 3. Solve for the error on the coarse level ⇒ good error estimation 4. Transfer error to the fine level 5. Correct the previous solution 6. Iterate ⇒ remove interpolation artifacts

63. 63 make iteration process faster (on the coarse grid we can effectively minimize the error) obtain better initial guess (solve directly on the coarsest grid) Coarse grid - advantages Coarsening allows: go to the coarsest grid solve here the equation to find interpolate to the finer grid

64. 64 Multigrid approach – Full scheme

65. 65 Non-linear: Full Approximation Scheme ⇒ Need to transfer current solution to the coarser level A(·) non-linear Equation for error involves current solution u 0 : Difference from the linear case: LinearNon-linear fine level: coarse level:

66. 66 Multigrid: Summary Used to solve linear or non-linear equations Method: combine two techniques –Basic iterative solver: quickly removes high frequencies of the error –Coarsening: makes low frequencies high Contribution: fast minimization of loosely coupled equations

67. 67 Fast Optimization: Results Gauss-Seidel 1.150.87 Full multigrid 0.01662.8 Gauss-Seidel 9.520.11 FAS-multigrid 0.08711.5 Gauss-Seidel 34.50.03 FAS-multigrid 0.3962.5 Horn&Schunck CLG “Towards ultimate…” Time [sec] frames/sec

68. 68 Summary of the Talk 25 years of Optical Flow : a lot of useful ingredients were developed: local constraints: brightness constancy gradient constancy smoothing techniques: homogeneous flow-driven (preserving discontinuities) bilateral filters handling of occlusions robust functions multiscale All ingredients are combined an a global Energy Minimization approach This difficult global optimization can be done very fast using Multigrid

69. 69 Thank you!

70. 70 Coarse grid - advantages Another advantages of using coarse grids: Relaxations are cheaper (because we have less poins) (1/2 as many points in 1D, 1/4 in 2D, 1/8 in 3D) Convergence rate is better ( instead of )

71. 71 Definition of the error After each iteration sweep we have a solution: There are two measures of the correctness of the solution : The error: The residual:

72. 72 Some Approaches to Optical Flow Gradient-based techniques use spatiotemporal partial derivatives to estimate flow at each point frame 1 frame 2 Patch comparison techniques patch comparison ++ + uv – shown to perform poorly good performance – aperture problem

73. Image-driven isotropic image-driven image gradient Idea: relax smoothing constraint on strong image gradients

74. 74 Anisotropic image-driven anisotropic image-driven encourages smooth flow along edges and not across Intuition: penalize only along- the-edge non-smoothness of flow: Actually used more complex:

75. 75 Flow-driven regularization anisotropic flow-driven [Aubert, Deriche, Kornprobst, 1995 ] [Weickert, Schnorr 2000] an alternative (one of): data terms smoothness terms complete functionals

76. 76 Flow-driven regularization anisotropic flow-drivenisotropic flow-driven less penalty for non-smooth flow less penalty across flow edges data terms smoothness terms complete functionals

77. 77 We need smart smoothing homogeneous image-driven isotropic image-driven anisotropic flow-driven isotropic flow-driven anisotropic method degree of smoothness optimal level ? data terms smoothness terms complete functionals

78. 78 What we get homogeneous isotropic image-driven anisotropic image-driven isotropic flow-driven anisotropic flow-driven data terms smoothness terms complete functionals

79. 79 What we pay homogeneous anisotropic image-driven isotropic flow-driven isotropic image-driven anisotropic flow-driven regularization (smoothness) methods running time 60 sec 40 sec 20 sec (image size 160x120) data terms smoothness terms complete functionals

80. 80 energy functional Discretized equation Solve by linear non-linear local iterations Multigrid (simple for linear, FAS for non-linear) equation is a min for satisfies Euler-Lagrange equation (ELE) hence Energy minimization: overview

81. 81 Gauss-Seidel relaxations start from random initial guess solve for one point assuming its neighbors are fixed (Gauss-Seidel)