1. Spatio-temporal Segmentation of Video by Hierarchical Mean Shift Analysis
Daniel DeMenthon
SMVP 2002

2. Motivation Semantic understanding of video Object segmentation
Video compression
Event detection
Video surveillance

3. Related Work Jojic et al. Flexible sprites Layer extraction
Shi and Malik normalized cuts
Irani et al. Event detection

4. Space – Time Volume Segmentation
Frame to Frame
Video Stack segmentation
Patch motion (1,u,v)

5. Feature Space
7 D feature vector, three color features in CIE L*u*v*, 2 motion angles, 2 motion distances.

6. Mapping Pixels in Feature Space

7. Mean Shift Clustering
Introduced by Fukunaga (1990) and applied to image analysis by Yizhong Cheng and Comaniciu and Meer (1997)
Natural borders (Leung et al.)

8. Range Search ATRIA tree O(N log N) for small radii
O(N) for large radii

9. Hierarchical Mean Shift
First standard mean shift is run until competition with very small radius
Weights are assigned to cluster centers equal to the sum of the weights of the member points
Clusters are now treated as the points, and radius is multiplied with factor of 1.25 or 1.50
Repeat until desired radius or the desired number of regions is reached

10. “Flower Garden” Video Sequence
88 x 60, 12 frames

11. Video Strands

12. Color Segmentation

13. Motion Segmentation
Faster lateral motion corresponds to lighter color

14. Comparison of Two Segmentation Algorithms

15. Comments on this Approach
Spatially distant color patches can be clustered together
Experiment was with small number of frames
It is not clear if it can handle the case when video object changes the direction of motion, or when video object stops
All features (color features and motion features) are scaled using heuristics, and that might not work for different video sequences

16. Conclusions and Further Work
Hierarchical mean shift analysis is of lower empirical complexity then standard mean shift analysis
Segmentation can be improved, the bounding areas of moving areas are jagged, usually by post-processing
Leung et al. suggested non parametric segmentation that can be applied here