Image Alignment by Image Averaging David Hong NCSSM, IE

Example Problem 1

Example Problem 2

Example Problem 3

Problem (Formal Statement)

Motivation Many Applications: – Special Effects (Movie) – Video Compression – Pattern Recognition – Image Stabilization (Digital Cameras) – Dead-reckoning (Mobile Robotics)

State of the Art Optical Flow Lucas-Kanade (1985) Optical Flow with Smoothness Constraint Horn-Schunck (1980) Phase Correlation

X Y u(x,y,t) U(X,Y) x y Lucas-Kanade

Then the floor-coordinate is (X 0,Y 0 ) and the sensor-coordinate is (x,y) at time t. Let us consider a point on the plane. From there, we can see:

Lucas-Kanade Differentiating on time gives us:

Lucas-Kanade Expressing (x,y) in terms of (X 0,Y 0 ) and the sensor position (X s,Y s,Θ s ) gives us:

Lucas-Kanade Putting the two together, we get: This is underdetermined!

Algorithm u u’

Improvement by Iteration u u’ u’’ u’’’

Improvement by Iteration u’ u u’’ u’(x’,y’) u’’(x’’,y’’)

Improvement by Iteration

u’ Places u’ is defined Place we need to evaluate u’ (x’, y’) (x’ 0, y’ 0 ) (x’ 1, y’ 0 ) (x’ 0, y’ 1 )(x’ 1, y’ 1 )

Improvement by Iteration

u’ u u’’ u’ not defined!

Improvement by Iteration u’ u’’ u’’ was not evaluated here Valid Region

Improvement by Iteration Places we need to evaluate u’ (i -1,0 ’, j -1,0 ’ ) (i 0,-1 ’, j 0,-1 ’ ) (i 0,1 ’, j 0,1 ’ ) (i 1,0 ’, j 1,0 ’ )

Performance of Algorithm Good Surface:Bad Surface: Algorithm Fails!

Performance of Algorithm Surface:

Performance of Algorithm Surface:

Performance of Algorithm Surface:

Assumptions Made The Error Function is locally quadratic The floor is linear

Weaknesses Many Iterations – Inherent to Technique “Fooled” by symmetry (Aliasing problem) – Inherent to Problem

Strengths Accurate Improvement by Iteration Finds Error Function Root by Newton’s

yuyu v(x v,y v ) u(x u,y u ) xvxv yvyv Phase Correlation xuxu

We consider the image to be like a 2-D wave. Then, displacement is simply a “phase shift” Rotation can similarly be found So, we “correlate” the “phases”

Phase Correlation

Weaknesses Inaccurate on first iteration Boundary Problem (Repetion Assumption) High complexity – FFT is “O(nlogn)”

Strengths Elegant Makes a big leap Works well on images with pattern Separates displacement and rotation (DFT)

yuyu v(x v,y v ) u(x u,y u ) xvxv yvyv Image Averaging xuxu

Find a “Center-of-Mass” of each image Track the motion of the center-of-mass

Weaknesses Boundary Problem (Average Point Moves) Average is affected by small discretization issues

Strengths Elegant Makes a big leap Very fast – Complexity of O(n) Yields itself well to Improvement by Iteration – Using same technique as in Lucas-Kanade

Handling the Weaknesses Here we decide to take an alternative approach Separate displacements from rotation Do this using FFT (as in Phase Correlation)

Handling the Weaknesses We handle rotation first – Post-FFT, only rotation remains

Handling the Weaknesses

Weaknesses We introduce an FFT ( O(nlogn) operation) However, only requires 2 – Phase correlation requires up to 3 or 4

Future Work Make Image Alignment Rigorous – Use complex numbers to notate displacement Smoothness Constraint Pre-processing the image Condition for Convergence

Thank You!