1. Mean Shift Theory and Applications Reporter: Zhongping Ji

2. Agenda Mean Shift Theory What is Mean Shift ? Density Estimation Methods Deriving the Mean Shift Mean shift properties Applications Clustering Discontinuity Preserving Smoothing Object Contour Detection Segmentation Object Tracking

3. Papers Mean Shift: A Robust Approach Toward Feature Space Analysis Authors: Dorin Comaniciu, Peter Meer (Rutgers University EECS, Member IEEE). IEEE Trans. Pattern analysis and machine intelligence 24(5), 2002 Field: application of modern statistical methods to image understanding problems A Topological Approach to Hierarchical Segmentation using Mean Shift Authors: Sylvain Paris, Fredo Durand. (MIT EECS, computer science and artificial intelligence laboratory) Proceedings of the IEEE conference on Computer Vision and Pattern Recognition (CVPR'07) Field: most aspects of image processing

4. Mean Shift Theory

5. Intuitive Description Distribution of identical billiard balls Region of interest Center of mass Mean Shift vector Objective : Find the densest region

6. Intuitive Description Distribution of identical billiard balls Region of interest Center of mass Mean Shift vector Objective : Find the densest region

7. Intuitive Description Distribution of identical billiard balls Region of interest Center of mass Mean Shift vector Objective : Find the densest region

8. Intuitive Description Distribution of identical billiard balls Region of interest Center of mass Mean Shift vector Objective : Find the densest region

9. Intuitive Description Distribution of identical billiard balls Region of interest Center of mass Mean Shift vector Objective : Find the densest region

10. Intuitive Description Distribution of identical billiard balls Region of interest Center of mass Mean Shift vector Objective : Find the densest region

11. Intuitive Description Distribution of identical billiard balls Region of interest Center of mass Objective : Find the densest region

12. What is Mean Shift ? Non-parametric Density Estimation Non-parametric Density GRADIENT Estimation (Mean Shift) Data Discrete PDF Representation PDF Analysis PDF in feature space Color space Scale space Actually any feature space you can conceive … A tool for: Finding modes in a set of data samples, manifesting an underlying probability density function (PDF) in R N

13. Non-Parametric Density Estimation Assumption : The data points are sampled from an underlying PDF Assumed Underlying PDFReal Data Samples Data point density implies PDF value !

14. Assumed Underlying PDFReal Data Samples Non-Parametric Density Estimation

15. Assumed Underlying PDFReal Data Samples ? Non-Parametric Density Estimation

16. Parametric Density Estimation Assumption : The data points are sampled from an underlying PDF Assumed Underlying PDF Estimate Real Data Samples

17. Kernel Density Estimation Parzen Windows - General Framework Kernel Properties: Normalized Symmetric Exponential weight decay ??? A function of some finite number of data points x 1 …x n Data

18. Kernel Density Estimation Parzen Windows - Function Forms A function of some finite number of data points x 1 …x n Data In practice one uses the forms: or Same function on each dimensionFunction of vector length only

19. Kernel Density Estimation Various Kernels A function of some finite number of data points x 1 …x n Examples: Epanechnikov Kernel Uniform Kernel Normal Kernel Data

20. Kernel Density Estimation Gradient Give up estimating the PDF ! Estimate ONLY the gradient Using the Kernel form: We get : Size of window

21. Kernel Density Estimation Gradient Computing The Mean Shift

22. Yet another Kernel density estimation ! Simple Mean Shift procedure: Compute mean shift vector Translate the Kernel window by m(x)

23. Mean Shift Mode Detection Updated Mean Shift Procedure: Find all modes using the Simple Mean Shift Procedure Prune modes by perturbing them (find saddle points and plateaus) Prune nearby – take highest mode in the window What happens if we reach a saddle point ? Perturb the mode position and check if we return back

24. Convergence

25. Adaptive Gradient Ascent Mean Shift Properties Automatic convergence speed – the mean shift vector size depends on the gradient itself. Near maxima, the steps are small and refined Convergence is guaranteed for infinitesimal steps only  infinitely convergent, For Uniform Kernel ( ), convergence is achieved in a finite number of steps Normal Kernel ( ) exhibits a smooth trajectory, but is slower than Uniform Kernel ( ).

26. Real Modality Analysis Tessellate the space with windows Run the procedure in parallel

27. Real Modality Analysis The blue data points were traversed by the windows towards the mode

28. Real Modality Analysis An example Window tracks signify the steepest ascent directions

29. Mean Shift Strengths & Weaknesses Strengths : Application independent tool Suitable for real data analysis Does not assume any prior shape (e.g. elliptical) on data clusters Can handle arbitrary feature spaces Only ONE parameter to choose h (window size) has a physical meaning, unlike K-Means Weaknesses : The window size (bandwidth selection) is not trivial Inappropriate window size can cause modes to be merged, or generate additional “shallow” modes  Use adaptive window size

30. Mean Shift Applications

31. Clustering Attraction basin : the region for which all trajectories lead to the same mode Cluster : All data points in the attraction basin of a mode Mean Shift : A robust Approach Toward Feature Space Analysis, by Comaniciu, Meer

32. Clustering Synthetic Examples Simple Modal Structures Complex Modal Structures

33. Clustering Real Example Initial window centers Modes foundModes after pruning Final clusters Feature space: L*u*v representation

34. Clustering Real Example L*u*v space representation

35. Clustering Real Example 2D (L*u) space representation Final clusters

36. Discontinuity Preserving Smoothing Feature space : Joint domain = spatial coordinates + color space Meaning : treat the image as data points in the spatial and gray level domain Image Data (slice) Mean Shift vectors Smoothing result Mean Shift : A robust Approach Toward Feature Space Analysis, by Comaniciu, Meer

37. Discontinuity Preserving Smoothing x y z The image gray levels…… can be viewed as data points in the x, y, z space (joined spatial And color space)

38. Discontinuity Preserving Smoothing y z

39. The effect of window size in spatial and range spaces

40. Discontinuity Preserving Smoothing Example

42. Object Contour Detection Ray Propagation Vessel Detection by Mean Shift Based Ray Propagation, by Tek, Comaniciu, Williams Accurately segment various objects (rounded in nature) in medical images

43. Object Contour Detection Ray Propagation Use displacement data to guide ray propagation Discontinuity preserving smoothing Displacement vectors Vessel Detection by Mean Shift Based Ray Propagation, by Tek, Comaniciu, Williams

44. Object Contour Detection Ray Propagation Speed function Normal to the contour Curvature

45. Object Contour Detection Original image Gray levels along red line Gray levels after smoothing Displacement vectors Displacement vectors’ derivative

46. Object Contour Detection Example

47. Importance of smoothing by curvature

48. Segmentation Segment = Cluster, or Cluster of Clusters Algorithm: Run Filtering (discontinuity preserving smoothing) Cluster the clusters which are closer than window size Image Data (slice) Mean Shift vectors Segmentation result Smoothing result Mean Shift : A robust Approach Toward Feature Space Analysis, by Comaniciu, Meer http://www.caip.rutgers.edu/~comanici

49. Segmentation Example …when feature space is only gray levels…

50. Segmentation Example

58. Image editing

59. A Topological Approach to Hierarchical Segmentation using Mean Shift

60. Fast and Hierarchical Efficient numerical schemes to evaluate the density function and extract its modes. A hierarchical segmentation based on a topological analysis of mean shift. Do not aim for better segmentation accuracy and focus on computational efficiency and creating a hierarchy.

62. FAST COMPUTATION OF THE FEATURE POINT DENSITY Coarse grid

63. EXTRACTION OF THE DENSITY MODES 0 label: The processed node is a local maximum. We create a new label. 1 label: All the ascending paths go to the same summit. We copy the label. 2+ labels: Boundary between two modes. We put a special marker.

64. Boundaries refinement

65. Hierarchical Segmentation

67. Boundary Persistence

68. Simplification

69. Examples

74. Monkey saddle

75. Ask & Answer