1. Forward-Backward Correlation for Template-Based Tracking Xiao Wang ECE Dept. Clemson University

2. Introduction Object tracking: An important computer vision problem Security and surveillance Medical therapy Retail space instrumentation Video abstraction Traffic Management Video editing Template-Based Tracking A classic technique Idea of template-based tracker

3. Related Work Jepson et al. Robust Online Appearance Models for Visual Tracking, CVPR 2001 Ho et al. Visual Tracking Using Learned Subspaces, CVPR 2004 Davis et al. Tracking Rigid Motion using a Compact-Structure Constraint, ICCV 1999 Avidan et al. Ensemble Tracking, CVPR 2005

4. Overview of the Approach Next Frame Forward Correlation Module Backward Correlation Module Textured Background? Gradient ModuleUpdate Template YesNo

5.  Template Selection: first frame vs. previous frame  Motion Model:  Similarity transformation  Template Selection: first frame vs. previous frame  Motion Model:  Similarity transformation Template-Based Tracking scalingdisplacement

6. Template-Based Tracking  Cross Correlation: SSD displacement reference image search image

7. Template-Based Tracking  Similarity measure: s(Δx, Δy)  Correlation Coefficient: c(Δx, Δy)  Similarity measure: s(Δx, Δy)  Correlation Coefficient: c(Δx, Δy) Mean of template Mean of image region

8. Forward Correlation  Forward Correlation:  Reference frame: previous frame  Goal: find transformation vector (dx, dy, α)  Approach: cross-correlation  Forward Correlation:  Reference frame: previous frame  Goal: find transformation vector (dx, dy, α)  Approach: cross-correlation  Template Update: Put into correlation coefficient framework

9. Forward Correlation  Drifting Problem:  Forward correlation approximates rotation with translation.  Forward correlation does not check the reliability of the template.  We need a mechanism to question the assumption of forward correlation.  Drifting Problem:  Forward correlation approximates rotation with translation.  Forward correlation does not check the reliability of the template.  We need a mechanism to question the assumption of forward correlation. Out-of-plane rotation Previous frame Current Frame

10. Backward Correlation  Consider our problem as motion segmentation  Goal of motion segmentation  Why is motion segmentation of video sequences difficult?  Under-constrained  Occlusion & Disocclusion  Image noise  A two-step procedure:  Determine the motion vectors associate with each pixel or feature point.  Group pixels or feature points that perform common motion.

11. Backward Correlation  Kanade-Lucas-Tomasi (KLT) feature tracker  Idea: minimize the dissimilarity of feature windows in two images  Assumption: mutual correspondence  Kanade-Lucas-Tomasi (KLT) feature tracker  Idea: minimize the dissimilarity of feature windows in two images  Assumption: mutual correspondence

12. Backward Correlation  Now consider the dissimilarity under the template window.  Decompose the template window into 2 partitions:  Now consider the dissimilarity under the template window.  Decompose the template window into 2 partitions:  Rewrite dissimilarity as: foregroundbackground high low

13. Backward Correlation  Background is moving at a different velocity than the foreground.  Foreground pixels have similar velocity and generate low SSD error.  Correlation between background pixels using foreground velocity generates high SSD error.  Goal: group foreground pixels which are moving at similar velocities  Background is moving at a different velocity than the foreground.  Foreground pixels have similar velocity and generate low SSD error.  Correlation between background pixels using foreground velocity generates high SSD error.  Goal: group foreground pixels which are moving at similar velocities Reference frame I(x) Current image J(x) Difference image D(x)=[I(x)-J(x+d)] 2 Difference image D(x)=[I(x)-J(x+d)] 2

14. Backward Correlation  Formulations for backward correlation Set of template candidates Correlation coefficient (likelihood)

15. Untextured Backgrounds  Limitation of backward correlation:  Fails if background has little texture.  Why? --- Examine the assumption.  Backward correlation has no reason to prefer the foreground to the background which is untextured.  Limitation of backward correlation:  Fails if background has little texture.  Why? --- Examine the assumption.  Backward correlation has no reason to prefer the foreground to the background which is untextured. Also low if untextured low

16. Untextured Backgrounds  Likelihood of backward correlation: textured vs. untextured Textured background Untextured background Foreground Template containing background pixels

17. Gradient Module  Motivation:  Seek a module focusing on the boundary of the target being tracked.  An edge-based segmentation problem.  Prior information: an ellipse model.  Gradient Module:  Motivation:  Seek a module focusing on the boundary of the target being tracked.  An edge-based segmentation problem.  Prior information: an ellipse model.  Gradient Module: Unit vector normal at pixel i Intensity gradient

18. Combining Modules Gradient module and backward correlation module have orthogonal failure modes. Textured or Untextured? Use sum of the gradient magnitude of the neighborhood region. Combination of forward correlation module and backward correlation module is straightforward. Combination of forward correlation module and gradient module requires the normalization of the matching scores.

19. Combining Modules  Normalize the matching score (likelihood):  Finial state is decided by:

20. Adaptive Scale Vary the scale by ± 10 percent during search process. Filter the result to avoid oversensitive scale adaptation. Comaniciu et al. Kernel-based object tracking, TPAMI 2003 Vary the scale by ± 10 percent during search process. Filter the result to avoid oversensitive scale adaptation. Comaniciu et al. Kernel-based object tracking, TPAMI 2003 Size of the best state given by the alg. Size of the object in the previous frame

21. Experimental Results: Cluttered Background Traditional template-based tracker slides off target:

22. Experimental Results: Cluttered Background Our algorithm remains locked onto target:

23. Experimental Results: Cluttered Background Tracking error plot: Our algorithm (blue, solid) vs. traditional template-based tracker (red, dashed) Tracking error plot: Our algorithm (blue, solid) vs. traditional template-based tracker (red, dashed) Error in x directionError in y direction

24. Experimental Results: Untextured Background Tracking results of traditional template-based tracker:

25. Experimental Results: Untextured Background Tracking results of our algorithm:

26. Experimental Results: Occlusion

27. Experimental Results: Tracking a vehicle

28. Conclusion Presented an extension to template-based tracking. Achieved robustness to out-of-plane rotation. Effective tracking in both textured and untextured environment. Remaining challenges: Robustness when scale changes. Use motion discontinuities to improve performance. Analysis of parameter sensitivity for untextured backgrounds.

29. Questions?

30. Thank You!