2. Iterative Image Registration: Lucas & Kanade Revisited Kentaro Toyama Vision Technology Group Microsoft Research

3. Every writer creates his own precursors. His work modifies our conception of the past, as it will modify the future. Jorge Luis Borges

4. History Lucas & Kanade (IUW 1981) LK BAHHSTSBJHBBL GSICETSC Bergen, Anandan, Hanna, Hingorani (ECCV 1992) Shi & Tomasi (CVPR 1994) Szeliski & Coughlan (CVPR 1994) Szeliski (WACV 1994) Black & Jepson (ECCV 1996) Hager & Belhumeur (CVPR 1996) Bainbridge-Smith & Lane (IVC 1997) Gleicher (CVPR 1997) Sclaroff & Isidoro (ICCV 1998) Cootes, Edwards, & Taylor (ECCV 1998)

5. Image Registration

6. Applications

7. Stereo LK BAHHSTSBJHBBL GSICETSC

8. Applications Stereo Dense optic flow LK BAHHSTSBJHBBL GSICETSC

9. Applications Stereo Dense optic flow Image mosaics LK BAHHSTSBJHBBL GSICETSC

10. Applications Stereo Dense optic flow Image mosaics Tracking LK BAHHSTSBJHBBL GSICETSC

11. Applications Stereo Dense optic flow Image mosaics Tracking Recognition LK BAHHSTSBJHBBL GSICETSC ?

12. Lucas & Kanade #1 Derivation

13. L&K Derivation 1 I0(x)I0(x)

14. h I0(x)I0(x) I 0 (x+h)

15. L&K Derivation 1 h I0(x)I0(x) I(x)I(x)

16. h I0(x)I0(x) I(x)I(x)

17. I0(x)I0(x) R I(x)I(x)

18. I0(x)I0(x) I(x)I(x)

19. h0h0 I0(x)I0(x) I(x)I(x)

20. I 0 (x+h 0 ) I(x)I(x)

21. L&K Derivation 1 I 0 (x+h 1 ) I(x)I(x)

22. L&K Derivation 1 I 0 (x+h k ) I(x)I(x)

23. L&K Derivation 1 I 0 (x+h f ) I(x)I(x)

24. Lucas & Kanade Derivation #2

25. L&K Derivation 2 Sum-of-squared-difference (SSD) error E(h) =  [ I(x) - I 0 (x+h) ] 2 x e Rx e R E(h)  [ I(x) - I 0 (x) - hI 0 ’(x) ] 2 x e Rx e R

26. L&K Derivation 2  2[I 0 ’(x)(I(x) - I 0 (x) ) - hI 0 ’(x) 2 ] x e Rx e R  I 0 ’(x)(I(x) - I 0 (x)) x e Rx e R h  I 0 ’(x) 2 x e Rx e R = 0 = 0

27. Comparison  I 0 ’(x)[I(x) - I 0 (x)] h  I 0 ’(x) 2 x x h w(x)[I(x) - I 0 (x)]  w(x) x x  I 0 ’(x)

28. Comparison  I 0 ’(x)[I(x) - I 0 (x)] h  I 0 ’(x) 2 x h x w(x)[I(x) - I 0 (x)]  w(x) x x  I 0 ’(x)

29. Generalizations

30. Original h)=  x eR ( E [ I(x )-(x ] 2 ) + h  I 

31. Original Dimension of image h)=  x eR ( E [ I(x )-(x ] 2 ) + h 1-dimensional  I  LK BAHHSTSBJHBBL GSICETSC

32. Generalization 1a Dimension of image h)=  x eR ( E [ I(x )-(x ] 2 ) + h 2D:  I  LK BAHHSTSBJHBBL GSICETSC

33. Generalization 1b Dimension of image h)=  x eR ( E [ I(x )-(x ] 2 ) + h Homogeneous 2D:  I  LK BAHHSTSBJHBBL GSICETSC

34. Problem A LK BAHHSTSBJHBBL GSICETSC Does the iteration converge?

35. Problem A Local minima:

36. Problem A Local minima:

37. Problem B -  I 0 ’(x)(I(x) - I 0 (x)) x e Rx e R h  I 0 ’(x) 2 x e Rx e R h is undefined if  I 0 ’(x) 2 is zero x e Rx e R LK BAHHSTSBJHBBL GSICETSC Zero gradient:

38. Problem B Zero gradient: ?

39. Problem B’ -  (x)(I(x) - I 0 (x)) x e Rx e R h y  2 x e Rx e R Aperture problem: LK BAHHSTSBJHBBL GSICETSC

40. Problem B’ No gradient along one direction: ?

41. Solutions to A & B Possible solutions: –Manual intervention LK BAHHSTSBJHBBL GSICETSC

42. Possible solutions: –Manual intervention –Zero motion default LK BAHHSTSBJHBBL GSICETSC Solutions to A & B

43. Possible solutions: –Manual intervention –Zero motion default –Coefficient “dampening” LK BAHHSTSBJHBBL GSICETSC Solutions to A & B

44. Possible solutions: –Manual intervention –Zero motion default –Coefficient “dampening” –Reliance on good features LK BAHHSTSBJHBBL GSICETSC Solutions to A & B

45. Possible solutions: –Manual intervention –Zero motion default –Coefficient “dampening” –Reliance on good features –Temporal filtering LK BAHHSTSBJHBBL GSICETSC Solutions to A & B

46. Possible solutions: –Manual intervention –Zero motion default –Coefficient “dampening” –Reliance on good features –Temporal filtering –Spatial interpolation / hierarchical estimation LK BAHHSTSBJHBBL GSICETSC Solutions to A & B

47. Possible solutions: –Manual intervention –Zero motion default –Coefficient “dampening” –Reliance on good features –Temporal filtering –Spatial interpolation / hierarchical estimation –Higher-order terms LK BAHHSTSBJHBBL GSICETSC Solutions to A & B

48. Original h)=  x eR ( E [ I(x )-(x ] 2 ) + h  I 

49. Original Transformations/warping of image h)=  x eR ( E [ I(x )-  I  (x ] 2 ) + h Translations: LK BAHHSTSBJHBBL GSICETSC

50. Problem C What about other types of motion?

51. Generalization 2a Transformations/warping of image A, h)=  x eR ( E [ I(AxAx )-(x ] 2 ) + h Affine:  I  LK BAHHSTSBJHBBL GSICETSC

52. Generalization 2a Affine:

53. Generalization 2b Transformations/warping of image A)=  x eR ( E [ I(A x )-(x ] 2 ) Planar perspective:  I  LK BAHHSTSBJHBBL GSICETSC

54. Generalization 2b Planar perspective: Affine +

55. Generalization 2c Transformations/warping of image h)=  x eR ( E [ I(f(x, h) )-(x ] 2 ) Other parametrized transformations  I  LK BAHHSTSBJHBBL GSICETSC

56. Generalization 2c Other parametrized transformations

57. Problem B” -(J T J) -1 J (I(f(x,h)) - I 0 (x)) h ~ Generalized aperture problem: LK BAHHSTSBJHBBL GSICETSC -  I 0 ’(x)(I(x) - I 0 (x)) x e Rx e R h  I 0 ’(x) 2 x e Rx e R

58. Problem B” ? Generalized aperture problem:

59. Original h)=  x eR ( E [ I(x )-(x ] 2 ) + h  I 

60. Original Image type h)=  x eR ( E [ I(x )-(x ] 2 ) + h Grayscale images  I  LK BAHHSTSBJHBBL GSICETSC

61. Generalization 3 Image type h)=  x eR ( E || I(x )-  I  (x || 2 ) + h Color images LK BAHHSTSBJHBBL GSICETSC

62. Original h)=  x eR ( E [ I(x )-(x ] 2 ) + h  I 

63. Original Constancy assumption h)=  x eR ( E [ I(x )-  I  (x ] 2 ) + h Brightness constancy LK BAHHSTSBJHBBL GSICETSC

64. Problem C What if illumination changes?

65. Generalization 4a Constancy assumption h,h, )=  x eR ( E [ I(x )- II (x ] 2 )+  + h Linear brightness constancy LK BAHHSTSBJHBBL GSICETSC

66. Generalization 4a

67. Generalization 4b Constancy assumption h, )=  x eR (E [ I(x )-  B (x ] 2 ) + h Illumination subspace constancy LK BAHHSTSBJHBBL GSICETSC

68. Problem C’ What if the texture changes?

69. Generalization 4c Constancy assumption h, )=  x eR (E [ I(x )- ] 2 + h Texture subspace constancy  B (x) LK BAHHSTSBJHBBL GSICETSC

70. Problem D Convergence is slower as #parameters increases.

71. Faster convergence: –Coarse-to-fine, filtering, interpolation, etc. LK BAHHSTSBJHBBL GSICETSC Solutions to D

72. Faster convergence: –Coarse-to-fine, filtering, interpolation, etc. –Selective parametrization Solutions to D LK BAHHSTSBJHBBL GSICETSC

73. Faster convergence: –Coarse-to-fine, filtering, interpolation, etc. –Selective parametrization –Offline precomputation Solutions to D LK BAHHSTSBJHBBL GSICETSC

74. Faster convergence: –Coarse-to-fine, filtering, interpolation, etc. –Selective parametrization –Offline precomputation Difference decomposition LK BAHHSTSBJHB GSICETSC Solutions to D BL

75. Solutions to D Difference decomposition

76. Solutions to D Difference decomposition

77. Faster convergence: –Coarse-to-fine, filtering, interpolation, etc. –Selective parametrization –Offline precomputation Difference decomposition –Improvements in gradient descent LK BAHHSTSBJHB GSICETSC Solutions to D BL

78. Faster convergence: –Coarse-to-fine, filtering, interpolation, etc. –Selective parametrization –Offline precomputation Difference decomposition –Improvements in gradient descent Multiple estimates of spatial derivatives LK BAHHSTSBJHB GSICETSC Solutions to D BL

79. Solutions to D Multiple estimates / state-space sampling

80. Generalizations  x eR [ I(x )-(x ] 2 ) + h  I  Modifications made so far:

81. Original Error norm h)=  x eR ( E [ I(x )-  I  (x ] 2 ) + h Squared difference: LK BAHHSTSBJHBBL GSICETSC

82. Problem E What about outliers?

83. Generalization 5a Error norm h)=  x eR ( E ( I(x )-  I  (x ) ) + h Robust error norm:  LK BAHHSTSBJHBBL GSICETSC

84. Original h)=  x eR ( E [ I(x )-(x ] 2 ) + h  I 

85. Original Image region / pixel weighting h)=  x eR ( E [ I(x )-  I  (x ] 2 ) + h Rectangular: LK BAHHSTSBJHBBL GSICETSC

86. Problem E’ What about background clutter?

87. Generalization 6a Image region / pixel weighting h)=  x eR ( E [ I(x )-  I  (x ] 2 ) + h Irregular: LK BAHHSTSBJHBBL GSICETSC

88. Problem E” What about foreground occlusion?

89. Generalization 6b Image region / pixel weighting h)=  x eR ( E [ I(x )-  I  (x ] 2 ) + h Weighted sum: w(x)w(x) LK BAHHSTSBJHBBL GSICETSC

90. Generalizations  x eR [ I(x )-(x ] 2 ) + h  I  Modifications made so far:

91. Generalization 6c Image region / pixel weighting h)=  x eR ( E [ I(x )-  I  (x ] 2 ) + h Sampled: LK BAHHSTSBJHBBL GSICETSC

92. Generalizations: Summary =  x eR ( I( )- w(x)w(x)   (x ) ) h) ( E f(x, h) h)=  x eR ( E [ I(x )-(x ] 2 ) + h  I 

93. Foresight Lucas & Kanade (IUW 1981) Bergen, Anandan, Hanna, Hingorani (ECCV 1992) Shi & Tomasi (CVPR 1994) Szeliski & Coughlan (CVPR 1994) Szeliski (WACV 1994) Black & Jepson (ECCV 1996) Hager & Belhumeur (CVPR 1996) Bainbridge-Smith & Lane (IVC 1997) Gleicher (CVPR 1997) Sclaroff & Isidoro (ICCV 1998) Cootes, Edwards, & Taylor (ECCV 1998) LK BAHHSTSBJHBBL GSICETSC

94. Summary Generalizations –Dimension of image –Image transformations / motion models –Pixel type –Constancy assumption –Error norm –Image mask L&K ? Y n Y n Y

95. Summary Common problems: –Local minima –Aperture effect –Illumination changes –Convergence issues –Outliers and occlusions L&K ? Y maybe Y n

96. Mitigation of aperture effect: –Manual intervention –Zero motion default –Coefficient “dampening” –Elimination of poor textures –Temporal filtering –Spatial interpolation / hierarchical –Higher-order terms Summary L&K ? n Y n

97. Summary Better convergence: –Coarse-to-fine, filtering, etc. –Selective parametrization –Offline precomputation Difference decomposition –Improvements in gradient descent Multiple estimates of spatial derivatives L&K ? Y n maybe

98. Hindsight Lucas & Kanade (IUW 1981) Bergen, Anandan, Hanna, Hingorani (ECCV 1992) Shi & Tomasi (CVPR 1994) Szeliski & Coughlan (CVPR 1994) Szeliski (WACV 1994) Black & Jepson (ECCV 1996) Hager & Belhumeur (CVPR 1996) Bainbridge-Smith & Lane (IVC 1997) Gleicher (CVPR 1997) Sclaroff & Isidoro (ICCV 1998) Cootes, Edwards, & Taylor (ECCV 1998)