Iterative Image Registration: Lucas & Kanade Revisited Kentaro Toyama Vision Technology Group Microsoft Research

Every writer creates his own precursors. His work modifies our conception of the past, as it will modify the future. Jorge Luis Borges

History Lucas & Kanade (IUW 1981) LK BAHHSTSBJHBBL GSICETSC Bergen, Anandan, Hanna, Hingorani (ECCV 1992) Shi & Tomasi (CVPR 1994) Szeliski & Coughlan (CVPR 1994) Szeliski (WACV 1994) Black & Jepson (ECCV 1996) Hager & Belhumeur (CVPR 1996) Bainbridge-Smith & Lane (IVC 1997) Gleicher (CVPR 1997) Sclaroff & Isidoro (ICCV 1998) Cootes, Edwards, & Taylor (ECCV 1998)

Image Registration

Applications

Stereo LK BAHHSTSBJHBBL GSICETSC

Applications Stereo Dense optic flow LK BAHHSTSBJHBBL GSICETSC

Applications Stereo Dense optic flow Image mosaics LK BAHHSTSBJHBBL GSICETSC

Applications Stereo Dense optic flow Image mosaics Tracking LK BAHHSTSBJHBBL GSICETSC

Applications Stereo Dense optic flow Image mosaics Tracking Recognition LK BAHHSTSBJHBBL GSICETSC ?

Lucas & Kanade #1 Derivation

L&K Derivation 1 I0(x)I0(x)

h I0(x)I0(x) I 0 (x+h)

L&K Derivation 1 h I0(x)I0(x) I(x)I(x)

h I0(x)I0(x) I(x)I(x)

I0(x)I0(x) R I(x)I(x)

I0(x)I0(x) I(x)I(x)

h0h0 I0(x)I0(x) I(x)I(x)

I 0 (x+h 0 ) I(x)I(x)

L&K Derivation 1 I 0 (x+h 1 ) I(x)I(x)

L&K Derivation 1 I 0 (x+h k ) I(x)I(x)

L&K Derivation 1 I 0 (x+h f ) I(x)I(x)

Lucas & Kanade Derivation #2

L&K Derivation 2 Sum-of-squared-difference (SSD) error E(h) =  [ I(x) - I 0 (x+h) ] 2 x e Rx e R E(h)  [ I(x) - I 0 (x) - hI 0 ’(x) ] 2 x e Rx e R

L&K Derivation 2  2[I 0 ’(x)(I(x) - I 0 (x) ) - hI 0 ’(x) 2 ] x e Rx e R  I 0 ’(x)(I(x) - I 0 (x)) x e Rx e R h  I 0 ’(x) 2 x e Rx e R = 0 = 0

Comparison  I 0 ’(x)[I(x) - I 0 (x)] h  I 0 ’(x) 2 x x h w(x)[I(x) - I 0 (x)]  w(x) x x  I 0 ’(x)

Comparison  I 0 ’(x)[I(x) - I 0 (x)] h  I 0 ’(x) 2 x h x w(x)[I(x) - I 0 (x)]  w(x) x x  I 0 ’(x)

Generalizations

Original h)=  x eR ( E [ I(x )-(x ] 2 ) + h  I 

Original Dimension of image h)=  x eR ( E [ I(x )-(x ] 2 ) + h 1-dimensional  I  LK BAHHSTSBJHBBL GSICETSC

Generalization 1a Dimension of image h)=  x eR ( E [ I(x )-(x ] 2 ) + h 2D:  I  LK BAHHSTSBJHBBL GSICETSC

Generalization 1b Dimension of image h)=  x eR ( E [ I(x )-(x ] 2 ) + h Homogeneous 2D:  I  LK BAHHSTSBJHBBL GSICETSC

Problem A LK BAHHSTSBJHBBL GSICETSC Does the iteration converge?

Problem A Local minima:

Problem A Local minima:

Problem B -  I 0 ’(x)(I(x) - I 0 (x)) x e Rx e R h  I 0 ’(x) 2 x e Rx e R h is undefined if  I 0 ’(x) 2 is zero x e Rx e R LK BAHHSTSBJHBBL GSICETSC Zero gradient:

Problem B Zero gradient: ?

Problem B’ -  (x)(I(x) - I 0 (x)) x e Rx e R h y  2 x e Rx e R Aperture problem: LK BAHHSTSBJHBBL GSICETSC

Problem B’ No gradient along one direction: ?

Solutions to A & B Possible solutions: –Manual intervention LK BAHHSTSBJHBBL GSICETSC

Possible solutions: –Manual intervention –Zero motion default LK BAHHSTSBJHBBL GSICETSC Solutions to A & B

Possible solutions: –Manual intervention –Zero motion default –Coefficient “dampening” LK BAHHSTSBJHBBL GSICETSC Solutions to A & B

Possible solutions: –Manual intervention –Zero motion default –Coefficient “dampening” –Reliance on good features LK BAHHSTSBJHBBL GSICETSC Solutions to A & B

Possible solutions: –Manual intervention –Zero motion default –Coefficient “dampening” –Reliance on good features –Temporal filtering LK BAHHSTSBJHBBL GSICETSC Solutions to A & B

Possible solutions: –Manual intervention –Zero motion default –Coefficient “dampening” –Reliance on good features –Temporal filtering –Spatial interpolation / hierarchical estimation LK BAHHSTSBJHBBL GSICETSC Solutions to A & B

Possible solutions: –Manual intervention –Zero motion default –Coefficient “dampening” –Reliance on good features –Temporal filtering –Spatial interpolation / hierarchical estimation –Higher-order terms LK BAHHSTSBJHBBL GSICETSC Solutions to A & B

Original h)=  x eR ( E [ I(x )-(x ] 2 ) + h  I 

Original Transformations/warping of image h)=  x eR ( E [ I(x )-  I  (x ] 2 ) + h Translations: LK BAHHSTSBJHBBL GSICETSC

Problem C What about other types of motion?

Generalization 2a Transformations/warping of image A, h)=  x eR ( E [ I(AxAx )-(x ] 2 ) + h Affine:  I  LK BAHHSTSBJHBBL GSICETSC

Generalization 2a Affine:

Generalization 2b Transformations/warping of image A)=  x eR ( E [ I(A x )-(x ] 2 ) Planar perspective:  I  LK BAHHSTSBJHBBL GSICETSC

Generalization 2b Planar perspective: Affine +

Generalization 2c Transformations/warping of image h)=  x eR ( E [ I(f(x, h) )-(x ] 2 ) Other parametrized transformations  I  LK BAHHSTSBJHBBL GSICETSC

Generalization 2c Other parametrized transformations

Problem B” -(J T J) -1 J (I(f(x,h)) - I 0 (x)) h ~ Generalized aperture problem: LK BAHHSTSBJHBBL GSICETSC -  I 0 ’(x)(I(x) - I 0 (x)) x e Rx e R h  I 0 ’(x) 2 x e Rx e R

Problem B” ? Generalized aperture problem:

Original h)=  x eR ( E [ I(x )-(x ] 2 ) + h  I 

Original Image type h)=  x eR ( E [ I(x )-(x ] 2 ) + h Grayscale images  I  LK BAHHSTSBJHBBL GSICETSC

Generalization 3 Image type h)=  x eR ( E || I(x )-  I  (x || 2 ) + h Color images LK BAHHSTSBJHBBL GSICETSC

Original h)=  x eR ( E [ I(x )-(x ] 2 ) + h  I 

Original Constancy assumption h)=  x eR ( E [ I(x )-  I  (x ] 2 ) + h Brightness constancy LK BAHHSTSBJHBBL GSICETSC

Problem C What if illumination changes?

Generalization 4a Constancy assumption h,h, )=  x eR ( E [ I(x )- II (x ] 2 )+  + h Linear brightness constancy LK BAHHSTSBJHBBL GSICETSC

Generalization 4a

Generalization 4b Constancy assumption h, )=  x eR (E [ I(x )-  B (x ] 2 ) + h Illumination subspace constancy LK BAHHSTSBJHBBL GSICETSC

Problem C’ What if the texture changes?

Generalization 4c Constancy assumption h, )=  x eR (E [ I(x )- ] 2 + h Texture subspace constancy  B (x) LK BAHHSTSBJHBBL GSICETSC

Problem D Convergence is slower as #parameters increases.

Faster convergence: –Coarse-to-fine, filtering, interpolation, etc. LK BAHHSTSBJHBBL GSICETSC Solutions to D

Faster convergence: –Coarse-to-fine, filtering, interpolation, etc. –Selective parametrization Solutions to D LK BAHHSTSBJHBBL GSICETSC

Faster convergence: –Coarse-to-fine, filtering, interpolation, etc. –Selective parametrization –Offline precomputation Solutions to D LK BAHHSTSBJHBBL GSICETSC

Faster convergence: –Coarse-to-fine, filtering, interpolation, etc. –Selective parametrization –Offline precomputation Difference decomposition LK BAHHSTSBJHB GSICETSC Solutions to D BL

Solutions to D Difference decomposition

Solutions to D Difference decomposition

Faster convergence: –Coarse-to-fine, filtering, interpolation, etc. –Selective parametrization –Offline precomputation Difference decomposition –Improvements in gradient descent LK BAHHSTSBJHB GSICETSC Solutions to D BL

Faster convergence: –Coarse-to-fine, filtering, interpolation, etc. –Selective parametrization –Offline precomputation Difference decomposition –Improvements in gradient descent Multiple estimates of spatial derivatives LK BAHHSTSBJHB GSICETSC Solutions to D BL

Solutions to D Multiple estimates / state-space sampling

Generalizations  x eR [ I(x )-(x ] 2 ) + h  I  Modifications made so far:

Original Error norm h)=  x eR ( E [ I(x )-  I  (x ] 2 ) + h Squared difference: LK BAHHSTSBJHBBL GSICETSC

Problem E What about outliers?

Generalization 5a Error norm h)=  x eR ( E ( I(x )-  I  (x ) ) + h Robust error norm:  LK BAHHSTSBJHBBL GSICETSC

Original h)=  x eR ( E [ I(x )-(x ] 2 ) + h  I 

Original Image region / pixel weighting h)=  x eR ( E [ I(x )-  I  (x ] 2 ) + h Rectangular: LK BAHHSTSBJHBBL GSICETSC

Problem E’ What about background clutter?

Generalization 6a Image region / pixel weighting h)=  x eR ( E [ I(x )-  I  (x ] 2 ) + h Irregular: LK BAHHSTSBJHBBL GSICETSC

Problem E” What about foreground occlusion?

Generalization 6b Image region / pixel weighting h)=  x eR ( E [ I(x )-  I  (x ] 2 ) + h Weighted sum: w(x)w(x) LK BAHHSTSBJHBBL GSICETSC

Generalizations  x eR [ I(x )-(x ] 2 ) + h  I  Modifications made so far:

Generalization 6c Image region / pixel weighting h)=  x eR ( E [ I(x )-  I  (x ] 2 ) + h Sampled: LK BAHHSTSBJHBBL GSICETSC

Generalizations: Summary =  x eR ( I( )- w(x)w(x)   (x ) ) h) ( E f(x, h) h)=  x eR ( E [ I(x )-(x ] 2 ) + h  I 

Foresight Lucas & Kanade (IUW 1981) Bergen, Anandan, Hanna, Hingorani (ECCV 1992) Shi & Tomasi (CVPR 1994) Szeliski & Coughlan (CVPR 1994) Szeliski (WACV 1994) Black & Jepson (ECCV 1996) Hager & Belhumeur (CVPR 1996) Bainbridge-Smith & Lane (IVC 1997) Gleicher (CVPR 1997) Sclaroff & Isidoro (ICCV 1998) Cootes, Edwards, & Taylor (ECCV 1998) LK BAHHSTSBJHBBL GSICETSC

Summary Generalizations –Dimension of image –Image transformations / motion models –Pixel type –Constancy assumption –Error norm –Image mask L&K ? Y n Y n Y

Summary Common problems: –Local minima –Aperture effect –Illumination changes –Convergence issues –Outliers and occlusions L&K ? Y maybe Y n

Mitigation of aperture effect: –Manual intervention –Zero motion default –Coefficient “dampening” –Elimination of poor textures –Temporal filtering –Spatial interpolation / hierarchical –Higher-order terms Summary L&K ? n Y n

Summary Better convergence: –Coarse-to-fine, filtering, etc. –Selective parametrization –Offline precomputation Difference decomposition –Improvements in gradient descent Multiple estimates of spatial derivatives L&K ? Y n maybe

Hindsight Lucas & Kanade (IUW 1981) Bergen, Anandan, Hanna, Hingorani (ECCV 1992) Shi & Tomasi (CVPR 1994) Szeliski & Coughlan (CVPR 1994) Szeliski (WACV 1994) Black & Jepson (ECCV 1996) Hager & Belhumeur (CVPR 1996) Bainbridge-Smith & Lane (IVC 1997) Gleicher (CVPR 1997) Sclaroff & Isidoro (ICCV 1998) Cootes, Edwards, & Taylor (ECCV 1998)