1. Motion estimation Digital Visual Effects, Spring 2005 Yung-Yu Chuang 2005/3/23 with slides by Michael Black and P. Anandan

2. Announcements Project #1 is due on next Tuesday, submission mechanism will be announced later this week. grading: report is important, results (good/bad), discussions on implementation, interface, features, etc.

3. Outline Motion estimation Lucas-Kanade algorithm Tracking Optical flow

4. Motion estimation Parametric motion (image alignment) Tracking Optical flow

5. Parametric motion

6. Tracking

7. Optical flow

8. Three assumptions Brightness consistency Spatial coherence Temporal persistence

9. Brightness consistency

10. Spatial coherence

11. Temporal persistence

12. Image registration Goal: register a template image J(x) and an input image I(x), where x=(x,y) T. Image alignment: I(x) and J(x) are two images Tracking: I(x) is the image at time t. J(x) is a small patch around the point p in the image at t+1. Optical flow: I(x) and J(x) are images of t and t+1.

13. Simple approach Minimize brightness difference

14. Simple SSD algorithm For each offset (u, v) compute E(u,v); Choose (u, v) which minimizes E(u,v); Problems: Not efficient No sub-pixel accuracy

15. Lucas-Kanade algorithm

16. Newton’s method Root finding for f(x)=0 Taylor’s expansion:

17. Lucas-Kanade algorithm

19. iterate shift I(x,y) with (u,v) compute gradient image I x, I y compute error image J(x,y)-I(x,y) compute Hessian matrix solve the linear system (u,v)=(u,v)+(∆u,∆v) until converge

20. Parametric model translation affine

21. Parametric model minimize with respect to minimize

22. Parametric model image gradient Jacobian of the warp warped image

23. Jacobian of the warp For example, for affine

24. Parametric model minimize Hessian

25. Lucas-Kanade algorithm iterate warp I with W(x;p) compute error image J(x,y)-I(W(x,p)) compute gradient image evaluate Jacobian at (x;p) compute compute Hessian compute solve update p by p+ until converge

27. Coarse-to-fine strategy J JwJw I warp refine + J JwJw I warp refine + J pyramid construction J JwJw I warp refine + I pyramid construction

28. Application of image alignment

29. Tracking

31. optical flow constraint equation brightness constancy

32. Optical flow constraint equation

33. Multiple constraint

34. Area-based method Assume spatial smoothness

35. Aperture problem

38. Demo for aperture problem http://www.sandlotscience.com/Distortions/Br eathing_objects.htmhttp://www.sandlotscience.com/Distortions/Br eathing_objects.htm http://www.sandlotscience.com/Ambiguous/ba rberpole.htmhttp://www.sandlotscience.com/Ambiguous/ba rberpole.htm

39. Aperture problem Larger window reduces ambiguity, but easily violates spatial smoothness assumption

40. Area-based method Assume spatial smoothness

41. Area-based method must be invertible

42. Area-based method The eigenvalues tell us about the local image structure. They also tell us how well we can estimate the flow in both directions Link to Harris corner detector

43. Textured area

44. Edge

45. Homogenous area

46. KLT tracking Select feature by Monitor features by measuring dissimilarity

50. KLT tracking http://www.ces.clemson.edu/~stb/klt/

51. KLT tracking http://www.ces.clemson.edu/~stb/klt/

52. SIFT tracking (matching actually)  Frame 0 Frame 10

53. SIFT tracking  Frame 0 Frame 100

54. SIFT tracking  Frame 0 Frame 200

55. KLT vs SIFT tracking KLT has larger accumulating error; partly because our KLT implementation doesn’t have affine transformation? SIFT is surprisingly robust

56. Tracking for rotoscoping

58. Waking life

59. Optical flow

60. Single-motion assumption Violated by Motion discontinuity Shadows Transparency Specular reflection …

61. Multiple motion

63. Simple problem: fit a line

64. Least-square fit

66. Robust statistics Recover the best fit for the majority of the data Detect and reject outliers

67. Approach

68. Robust weighting

69. Robust estimation

73. Regularization and dense optical flow

85. Input for the NPR algorithm

86. Brushes

87. Edge clipping

88. Gradient

89. Smooth gradient

90. Textured brush

91. Edge clipping

92. Temporal artifacts Frame-by-frame application of the NPR algorithm

93. Temporal coherence

94. RE:Vision

95. What dreams may come

96. Reference B.D. Lucas and T. Kanade, An Iterative Image Registration Technique with an Application to Stereo Vision, Proceedings of the 1981 DARPA Image Understanding Workshop, 1981, pp121-130.An Iterative Image Registration Technique with an Application to Stereo Vision Bergen, J. R. and Anandan, P. and Hanna, K. J. and Hingorani, R., Hierarchical Model-Based Motion Estimation, ECCV 1992, pp237-252. Hierarchical Model-Based Motion Estimation J. Shi and C. Tomasi, Good Features to Track, CVPR 1994, pp593-600.Good Features to Track Michael Black and P. Anandan, The Robust Estimation of Multiple Motions: Parametric and Piecewise-Smooth Flow Fields, Computer Vision and Image Understanding 1996, pp75-104.The Robust Estimation of Multiple Motions: Parametric and Piecewise-Smooth Flow Fields S. Baker and I. Matthews, Lucas-Kanade 20 Years On: A Unifying Framework, International Journal of Computer Vision, 56(3), 2004, pp221 - 255.Lucas-Kanade 20 Years On: A Unifying Framework Peter Litwinowicz, Processing Images and Video for An Impressionist Effects, SIGGRAPH 1997.Processing Images and Video for An Impressionist Effects Aseem Agarwala, Aaron Hertzman, David Salesin and Steven Seitz, Keyframe-Based Tracking for Rotoscoping and Animation, SIGGRAPH 2004, pp584-591. Keyframe-Based Tracking for Rotoscoping and Animation