1. Mean Shift ; Theory and Applications Presented by: Reza Hemati دی 89 December 2010 1 گروه بینایی ماشین و پردازش تصویر Machine Vision and Image Processing Group

2. Mean Shift Theory and Applications Yaron Ukrainitz & Bernard Sarel 2

3. Agenda Mean Shift Theory What is Mean Shift ? Density Estimation Methods Deriving the Mean Shift Mean shift properties Applications Clustering Discontinuity Preserving Smoothing Segmentation Object Tracking Object Contour Detection 3

4. Mean Shift Theory 4

5. Intuitive Description Distribution of identical billiard balls Region of interest Center of mass Mean Shift vector Objective : Find the densest region 5

6. Intuitive Description Distribution of identical billiard balls Region of interest Center of mass Mean Shift vector Objective : Find the densest region 6

7. Intuitive Description Distribution of identical billiard balls Region of interest Center of mass Mean Shift vector Objective : Find the densest region 7

8. Intuitive Description Distribution of identical billiard balls Region of interest Center of mass Mean Shift vector Objective : Find the densest region 8

9. Intuitive Description Distribution of identical billiard balls Region of interest Center of mass Mean Shift vector Objective : Find the densest region 9

10. Intuitive Description Distribution of identical billiard balls Region of interest Center of mass Mean Shift vector Objective : Find the densest region 10

11. Intuitive Description Distribution of identical billiard balls Region of interest Center of mass Objective : Find the densest region 11

12. What is Mean Shift ? Non-parametric Density Estimation Non-parametric Density GRADIENT Estimation (Mean Shift) Data Discrete PDF Representation PDF Analysis PDF in feature space Color space Scale space Actually any feature space you can conceive … A tool for: Finding modes in a set of data samples, manifesting an underlying probability density function (PDF) in R N 12

13. Non-Parametric Density Estimation Assumption : The data points are sampled from an underlying PDF Assumed Underlying PDFReal Data Samples Data point density implies PDF value ! 13

14. Assumed Underlying PDFReal Data Samples Non-Parametric Density Estimation 14

15. Assumed Underlying PDFReal Data Samples ? Non-Parametric Density Estimation 15

16. Parametric Density Estimation Assumption : The data points are sampled from an underlying PDF Assumed Underlying PDF Estimate Real Data Samples 16

17. Kernel Density Estimation Parzen Windows - General Framework Kernel Properties: Normalized Symmetric Exponential weight decay ??? A function of some finite number of data points x 1 …x n Data 17

18. Kernel Density Estimation Parzen Windows - Function Forms A function of some finite number of data points x 1 …x n Data In practice one uses the forms: or Same function on each dimensionFunction of vector length only 18

19. Kernel Density Estimation Various Kernels A function of some finite number of data points x 1 …x n Examples: Epanechnikov Kernel Uniform Kernel Normal Kernel Data 19

20. Kernel Density Estimation Gradient Give up estimating the PDF ! Estimate ONLY the gradient Using the Kernel form: We get : Size of window 20

21. Kernel Density Estimation Gradient Computing The Mean Shift 21

22. Computing The Mean Shift Yet another Kernel density estimation ! Simple Mean Shift procedure: Compute mean shift vector Translate the Kernel window by m(x) 22

23. Mean Shift Mode Detection Updated Mean Shift Procedure: Find all modes using the Simple Mean Shift Procedure Prune modes by perturbing them (find saddle points and plateaus) Prune nearby – take highest mode in the window What happens if we reach a saddle point ? Perturb the mode position and check if we return back 23

24. Adaptive Gradient Ascent Mean Shift Properties Automatic convergence speed – the mean shift vector size depends on the gradient itself. Near maxima, the steps are small and refined For Uniform Kernel ( ), convergence is achieved in a finite number of steps Normal Kernel ( ) exhibits a smooth trajectory, but is slower than Uniform Kernel ( ). 24

25. Real Modality Analysis Tessellate the space with windows Run the procedure in parallel 25

26. Real Modality Analysis The blue data points were traversed by the windows towards the mode 26

27. Real Modality Analysis An example Window tracks signify the steepest ascent directions 27

28. Mean Shift Strengths & Weaknesses Strengths : Application independent tool Suitable for real data analysis Does not assume any prior shape (e.g. elliptical) on data clusters Can handle arbitrary feature spaces Only ONE parameter to choose h (window size) has a physical meaning, unlike K-Means Weaknesses : The window size (bandwidth selection) is not trivial Inappropriate window size can cause modes to be merged, or generate additional “shallow” modes  Use adaptive window size 28

29. Mean Shift Applications 29

30. Clustering Attraction basin : the region for which all trajectories lead to the same mode Cluster : All data points in the attraction basin of a mode Mean Shift : A robust Approach Toward Feature Space Analysis, by Comaniciu, Meer 30

31. Clustering Synthetic Examples Simple Modal Structures Complex Modal Structures 31

32. Clustering Real Example Initial window centers Modes foundModes after pruning Final clusters Feature space: L*u*v representation 32

33. Clustering Real Example L*u*v space representation 33

34. Clustering Real Example 2D (L*u) space representation Final clusters 34

35. Discontinuity Preserving Smoothing Feature space : Joint domain = spatial coordinates + color space Meaning : treat the image as data points in the spatial and gray level domain Image Data (slice) Mean Shift vectors Smoothing result Mean Shift : A robust Approach Toward Feature Space Analysis, by Comaniciu, Meer 35

36. Discontinuity Preserving Smoothing x y z The image gray levels…… can be viewed as data points in the x, y, z space (joined spatial And color space) 36

37. Discontinuity Preserving Smoothing y z Flat regions induce the modes ! 37

38. Discontinuity Preserving Smoothing The effect of window size in spatial and range spaces 38

39. Discontinuity Preserving Smoothing Example 39

40. Discontinuity Preserving Smoothing Example 40

41. Segmentation Segment = Cluster, or Cluster of Clusters Algorithm: Run Filtering (discontinuity preserving smoothing) Cluster the clusters which are closer than window size Image Data (slice) Mean Shift vectors Segmentation result Smoothing result Mean Shift : A robust Approach Toward Feature Space Analysis, by Comaniciu, Meer http://www.caip.rutgers.edu/~comanici 41

42. Segmentation Example …when feature space is only gray levels… 42

43. Segmentation Example 43

44. Segmentation Example 44

45. Segmentation Example 45

46. Segmentation Example 46

47. Segmentation Example 47

48. Segmentation Example 48

49. Non-Rigid Object Tracking … … 49

50. Non-Rigid Object Tracking Real-Time SurveillanceDriver Assistance Object-Based Video Compression 50

51. Current frame …… Mean-Shift Object Tracking General Framework: Target Representation Choose a feature space Represent the model in the chosen feature space Choose a reference model in the current frame 51

52. Mean-Shift Object Tracking General Framework: Target Localization Search in the model’s neighborhood in next frame Start from the position of the model in the current frame Find best candidate by maximizing a similarity func. Repeat the same process in the next pair of frames Current frame …… ModelCandidate 52

53. Mean-Shift Object Tracking Target Representation Choose a reference target model Quantized Color Space Choose a feature space Represent the model by its PDF in the feature space Kernel Based Object Tracking, by Comaniniu, Ramesh, Meer 53

54. Mean-Shift Object Tracking PDF Representation Similarity Function: Target Model (centered at 0) Target Candidate (centered at y) 54

55. Mean-Shift Object Tracking Smoothness of Similarity Function Similarity Function: Problem: Target is represented by color info only Spatial info is lost Solution: Mask the target with an isotropic kernel in the spatial domain f(y) becomes smooth in y f is not smooth Gradient- based optimizations are not robust Large similarity variations for adjacent locations 55

56. Mean-Shift Object Tracking Finding the PDF of the target model Target pixel locations A differentiable, isotropic, convex, monotonically decreasing kernel Peripheral pixels are affected by occlusion and background interference The color bin index (1..m) of pixel x Normalization factor Pixel weight Probability of feature u in model Probability of feature u in candidate Normalization factor Pixel weight 0 model y candidate 56

57. Mean-Shift Object Tracking Similarity Function Target model: Target candidate: Similarity function: 1 1 The Bhattacharyya Coefficient 57

58. Mean-Shift Object Tracking Target Localization Algorithm Start from the position of the model in the current frame Search in the model’s neighborhood in next frame Find best candidate by maximizing a similarity func. 58

59. Linear approx. (around y 0 ) Mean-Shift Object Tracking Approximating the Similarity Function Model location: Candidate location: Independent of y Density estimate! (as a function of y) 59

60. Mean-Shift Object Tracking Maximizing the Similarity Function The mode of = sought maximum Important Assumption: One mode in the searched neighborhood The target representation provides sufficient discrimination 60

61. Mean-Shift Object Tracking Applying Mean-Shift Original Mean-Shift: Find mode ofusing The mode of = sought maximum Extended Mean-Shift: Find mode of using 61

62. Mean-Shift Object Tracking About Kernels and Profiles A special class of radially symmetric kernels: The profile of kernel K Extended Mean-Shift: Find mode of using 62

63. Mean-Shift Object Tracking Choosing the Kernel Epanechnikov kernel: A special class of radially symmetric kernels: Extended Mean-Shift: Uniform kernel: 63

64. 64