1. Spatial Histograms for Head Tracking Sriram Rangarajan Department of Electrical and Computer Engineering, Clemson University, Clemson, SC 29634

2. Overview of tracker Intensity Gradients (works on the boundary of the ellipse) Modules that are complementary to gradients : 1. Color histograms 2. Spatiograms 3. Co-occurrence matrices 4. Log-Gabor histograms 5. Haar histograms 6. Edge-orientation histograms Complementary module (works inside the ellipse)

3. Gradient module Normal to points on ellipse Gradient score Likelihood score [Stan Birchfield, 1998]

4. Overview of modules used Modelhistogram (from first frame) Targethistogram(from current frame) Similarity measure Likelihood score from module Convert to percentage score, combine with intensity gradient module likelihood and update “state”.

5. Similarity measure between model and target histograms Histogram intersection [Swain & Ballard1991] Likelihood normalization

6. Overview of modules Color histograms Only color information (no spatial information) SpatiogramsColor information + limited spatial information ( global) Edge-orientation histograms Only spatial information Co-occurrence matrices Color information + limited spatial information ( local) Log-Gabor histograms Only spatial information (no color) Haar histogramsOnly spatial information (no color)

7. Color Histograms Ignore spatial information (most cases) Computationally efficient, simple, robust and invariant to any one-to-one spatial transformations

8. Computing color histograms Index for color channel Pixels in a bin Single color channel of image Number of bins for channel C 1

9. Spatiograms Higher-order histograms that capture spatial information globally Captures both values of pixels and a limited amount of their spatial relationship Bins are weighted by mean and covariance of pixels contributing to it [Birchfield and Rangarajan, CVPR 2005]

10. Spatiograms and histograms A histogram (no spatial information) A spatiogram (some spatial Information) Σ µ A histogram (no spatial information) A spatiogram (some spatial Information) Σ µ A histogram (no spatial information) A spatiogram (some spatial Information) Σ µ

11. An illustrative example Three poses of a head Image generated from histogram Image generated from spatiogram

12. Co-occurrence matrices Used for texture analysis Captures the local spatial relationships between colors (or gray levels) Normally used for gray-level images No. of pixel pairs with value (x,y)

13. Co-occurrence matrices 1011 1013 10 131011 131011 10 11 131011 13 Image Co-occurrence matrix Local spatial relationships Color values (C) (C)

14. Texture histograms * Filter bank Image = Histogram (Haar Wavelets or Log-Gabor filters)

15. Haar histograms Histogram of image after convolving with 3-level Haar pyramid: Haar histogram (at scale S and orientation O.) Image obtained by convolving with Haar pyramid at scale S and orientation O

16. Log-Gabor histograms Similar to Haar histograms, but uses a bank of log-Gabor filters. Log-Gabor histogram Image obtained by convolving with filter bank at scale S and orientation O

17. Edge-orientation histograms Obtained from gradient information Complete reliance on spatial information Histogram bin is decided by orientation of a pixel

18. Computing edge-orientation histograms * = Difference of Gaussian kernel (DoG) Image Edge-orientation Histogram

19. Edge-orientation histograms Computed from gradient images obtained by convolving image with Difference of Gaussian (DoG) kernel in x and y Orientation for pixel along vertical direction is 0

20. Results: log-Gabor histograms log-Gabor histogram color histogram Legend:

21. Results: Haar histograms Haar histogram color histogram Legend:

22. Results: Edge-orientation histograms Edge-orientation histogram color histogram Legend:

23. Results: Spatiograms spatiograms color histogram Legend:

24. Results: Co-occurrence matrices Co-occurrence matrices color histogram Legend:

25. Overview of results Color histograms Distracted by skin-colored background SpatiogramsTracks target in skin-colored background and clutter Edge-orientation histograms Fails in a cluttered background Co-occurrence matrices Tracks target in skin-colored background and clutter Log-Gabor histograms Fails in a cluttered background Haar histogramsFails in a cluttered background

26. Mean errors in x and y for Sequence 1

27. Mean errors in x and y for Sequence 2

28. Conclusion Limited amount of spatial information drastically improves tracking results Color information also important:  With only spatial information: tracker is distracted by cluttered background  With only color: tracker is distracted by skin-colored background Global spatial information is the most effective (spatiograms)

29. Thank You!