1. Image Alignment by Image Averaging David Hong NCSSM, IE364 2008

2. Example Problem 1

3. Example Problem 2

4. Example Problem 3

5. Problem (Formal Statement)

6. Motivation Many Applications: – Special Effects (Movie) – Video Compression – Pattern Recognition – Image Stabilization (Digital Cameras) – Dead-reckoning (Mobile Robotics)

7. State of the Art Optical Flow Lucas-Kanade (1985) Optical Flow with Smoothness Constraint Horn-Schunck (1980) Phase Correlation

8. X Y u(x,y,t) U(X,Y) x y Lucas-Kanade

9. Then the floor-coordinate is (X 0,Y 0 ) and the sensor-coordinate is (x,y) at time t. Let us consider a point on the plane. From there, we can see:

10. Lucas-Kanade Differentiating on time gives us:

11. Lucas-Kanade Expressing (x,y) in terms of (X 0,Y 0 ) and the sensor position (X s,Y s,Θ s ) gives us:

12. Lucas-Kanade Putting the two together, we get: This is underdetermined!

13. Algorithm u u’

14. Improvement by Iteration u u’ u’’ u’’’

15. Improvement by Iteration u’ u u’’ u’(x’,y’) u’’(x’’,y’’)

16. Improvement by Iteration

17. u’ Places u’ is defined Place we need to evaluate u’ (x’, y’) (x’ 0, y’ 0 ) (x’ 1, y’ 0 ) (x’ 0, y’ 1 )(x’ 1, y’ 1 )

18. Improvement by Iteration

19. u’ u u’’ u’ not defined!

20. Improvement by Iteration u’ u’’ u’’ was not evaluated here Valid Region

21. Improvement by Iteration Places we need to evaluate u’ (i -1,0 ’, j -1,0 ’ ) (i 0,-1 ’, j 0,-1 ’ ) (i 0,1 ’, j 0,1 ’ ) (i 1,0 ’, j 1,0 ’ )

22. Performance of Algorithm Good Surface:Bad Surface: Algorithm Fails!

23. Performance of Algorithm Surface:

24. Performance of Algorithm Surface:

25. Performance of Algorithm Surface:

26. Assumptions Made The Error Function is locally quadratic The floor is linear

27. Weaknesses Many Iterations – Inherent to Technique “Fooled” by symmetry (Aliasing problem) – Inherent to Problem

28. Strengths Accurate Improvement by Iteration Finds Error Function Root by Newton’s

29. yuyu v(x v,y v ) u(x u,y u ) xvxv yvyv Phase Correlation xuxu

30. We consider the image to be like a 2-D wave. Then, displacement is simply a “phase shift” Rotation can similarly be found So, we “correlate” the “phases”

31. Phase Correlation

35. Weaknesses Inaccurate on first iteration Boundary Problem (Repetion Assumption) High complexity – FFT is “O(nlogn)”

36. Strengths Elegant Makes a big leap Works well on images with pattern Separates displacement and rotation (DFT)

37. yuyu v(x v,y v ) u(x u,y u ) xvxv yvyv Image Averaging xuxu

38. Find a “Center-of-Mass” of each image Track the motion of the center-of-mass

39. Weaknesses Boundary Problem (Average Point Moves) Average is affected by small discretization issues

40. Strengths Elegant Makes a big leap Very fast – Complexity of O(n) Yields itself well to Improvement by Iteration – Using same technique as in Lucas-Kanade

41. Handling the Weaknesses Here we decide to take an alternative approach Separate displacements from rotation Do this using FFT (as in Phase Correlation)

42. Handling the Weaknesses We handle rotation first – Post-FFT, only rotation remains

43. Handling the Weaknesses

44. Weaknesses We introduce an FFT ( O(nlogn) operation) However, only requires 2 – Phase correlation requires up to 3 or 4

45. Future Work Make Image Alignment Rigorous – Use complex numbers to notate displacement Smoothness Constraint Pre-processing the image Condition for Convergence

46. Thank You!