Mean Shift ; Theory and Applications Presented by: Reza Hemati دی 89 December گروه بینایی ماشین و پردازش تصویر Machine Vision and Image Processing Group

Mean Shift Theory and Applications Yaron Ukrainitz & Bernard Sarel 2

Agenda Mean Shift Theory What is Mean Shift ? Density Estimation Methods Deriving the Mean Shift Mean shift properties Applications Clustering Discontinuity Preserving Smoothing Segmentation Object Tracking Object Contour Detection 3

Mean Shift Theory 4

Intuitive Description Distribution of identical billiard balls Region of interest Center of mass Mean Shift vector Objective : Find the densest region 5

Intuitive Description Distribution of identical billiard balls Region of interest Center of mass Mean Shift vector Objective : Find the densest region 6

Intuitive Description Distribution of identical billiard balls Region of interest Center of mass Mean Shift vector Objective : Find the densest region 7

Intuitive Description Distribution of identical billiard balls Region of interest Center of mass Mean Shift vector Objective : Find the densest region 8

Intuitive Description Distribution of identical billiard balls Region of interest Center of mass Mean Shift vector Objective : Find the densest region 9

Intuitive Description Distribution of identical billiard balls Region of interest Center of mass Mean Shift vector Objective : Find the densest region 10

Intuitive Description Distribution of identical billiard balls Region of interest Center of mass Objective : Find the densest region 11

What is Mean Shift ? Non-parametric Density Estimation Non-parametric Density GRADIENT Estimation (Mean Shift) Data Discrete PDF Representation PDF Analysis PDF in feature space Color space Scale space Actually any feature space you can conceive … A tool for: Finding modes in a set of data samples, manifesting an underlying probability density function (PDF) in R N 12

Non-Parametric Density Estimation Assumption : The data points are sampled from an underlying PDF Assumed Underlying PDFReal Data Samples Data point density implies PDF value ! 13

Assumed Underlying PDFReal Data Samples Non-Parametric Density Estimation 14

Assumed Underlying PDFReal Data Samples ? Non-Parametric Density Estimation 15

Parametric Density Estimation Assumption : The data points are sampled from an underlying PDF Assumed Underlying PDF Estimate Real Data Samples 16

Kernel Density Estimation Parzen Windows - General Framework Kernel Properties: Normalized Symmetric Exponential weight decay ??? A function of some finite number of data points x 1 …x n Data 17

Kernel Density Estimation Parzen Windows - Function Forms A function of some finite number of data points x 1 …x n Data In practice one uses the forms: or Same function on each dimensionFunction of vector length only 18

Kernel Density Estimation Various Kernels A function of some finite number of data points x 1 …x n Examples: Epanechnikov Kernel Uniform Kernel Normal Kernel Data 19

Kernel Density Estimation Gradient Give up estimating the PDF ! Estimate ONLY the gradient Using the Kernel form: We get : Size of window 20

Kernel Density Estimation Gradient Computing The Mean Shift 21

Computing The Mean Shift Yet another Kernel density estimation ! Simple Mean Shift procedure: Compute mean shift vector Translate the Kernel window by m(x) 22

Mean Shift Mode Detection Updated Mean Shift Procedure: Find all modes using the Simple Mean Shift Procedure Prune modes by perturbing them (find saddle points and plateaus) Prune nearby – take highest mode in the window What happens if we reach a saddle point ? Perturb the mode position and check if we return back 23

Adaptive Gradient Ascent Mean Shift Properties Automatic convergence speed – the mean shift vector size depends on the gradient itself. Near maxima, the steps are small and refined For Uniform Kernel ( ), convergence is achieved in a finite number of steps Normal Kernel ( ) exhibits a smooth trajectory, but is slower than Uniform Kernel ( ). 24

Real Modality Analysis Tessellate the space with windows Run the procedure in parallel 25

Real Modality Analysis The blue data points were traversed by the windows towards the mode 26

Real Modality Analysis An example Window tracks signify the steepest ascent directions 27

Mean Shift Strengths & Weaknesses Strengths : Application independent tool Suitable for real data analysis Does not assume any prior shape (e.g. elliptical) on data clusters Can handle arbitrary feature spaces Only ONE parameter to choose h (window size) has a physical meaning, unlike K-Means Weaknesses : The window size (bandwidth selection) is not trivial Inappropriate window size can cause modes to be merged, or generate additional “shallow” modes  Use adaptive window size 28

Mean Shift Applications 29

Clustering Attraction basin : the region for which all trajectories lead to the same mode Cluster : All data points in the attraction basin of a mode Mean Shift : A robust Approach Toward Feature Space Analysis, by Comaniciu, Meer 30

Clustering Synthetic Examples Simple Modal Structures Complex Modal Structures 31

Clustering Real Example Initial window centers Modes foundModes after pruning Final clusters Feature space: L*u*v representation 32

Clustering Real Example L*u*v space representation 33

Clustering Real Example 2D (L*u) space representation Final clusters 34

Discontinuity Preserving Smoothing Feature space : Joint domain = spatial coordinates + color space Meaning : treat the image as data points in the spatial and gray level domain Image Data (slice) Mean Shift vectors Smoothing result Mean Shift : A robust Approach Toward Feature Space Analysis, by Comaniciu, Meer 35

Discontinuity Preserving Smoothing x y z The image gray levels…… can be viewed as data points in the x, y, z space (joined spatial And color space) 36

Discontinuity Preserving Smoothing y z Flat regions induce the modes ! 37

Discontinuity Preserving Smoothing The effect of window size in spatial and range spaces 38

Discontinuity Preserving Smoothing Example 39

Discontinuity Preserving Smoothing Example 40

Segmentation Segment = Cluster, or Cluster of Clusters Algorithm: Run Filtering (discontinuity preserving smoothing) Cluster the clusters which are closer than window size Image Data (slice) Mean Shift vectors Segmentation result Smoothing result Mean Shift : A robust Approach Toward Feature Space Analysis, by Comaniciu, Meer 41

Segmentation Example …when feature space is only gray levels… 42

Segmentation Example 43

Segmentation Example 44

Segmentation Example 45

Segmentation Example 46

Segmentation Example 47

Segmentation Example 48

Non-Rigid Object Tracking … … 49

Non-Rigid Object Tracking Real-Time SurveillanceDriver Assistance Object-Based Video Compression 50

Current frame …… Mean-Shift Object Tracking General Framework: Target Representation Choose a feature space Represent the model in the chosen feature space Choose a reference model in the current frame 51

Mean-Shift Object Tracking General Framework: Target Localization Search in the model’s neighborhood in next frame Start from the position of the model in the current frame Find best candidate by maximizing a similarity func. Repeat the same process in the next pair of frames Current frame …… ModelCandidate 52

Mean-Shift Object Tracking Target Representation Choose a reference target model Quantized Color Space Choose a feature space Represent the model by its PDF in the feature space Kernel Based Object Tracking, by Comaniniu, Ramesh, Meer 53

Mean-Shift Object Tracking PDF Representation Similarity Function: Target Model (centered at 0) Target Candidate (centered at y) 54

Mean-Shift Object Tracking Smoothness of Similarity Function Similarity Function: Problem: Target is represented by color info only Spatial info is lost Solution: Mask the target with an isotropic kernel in the spatial domain f(y) becomes smooth in y f is not smooth Gradient- based optimizations are not robust Large similarity variations for adjacent locations 55

Mean-Shift Object Tracking Finding the PDF of the target model Target pixel locations A differentiable, isotropic, convex, monotonically decreasing kernel Peripheral pixels are affected by occlusion and background interference The color bin index (1..m) of pixel x Normalization factor Pixel weight Probability of feature u in model Probability of feature u in candidate Normalization factor Pixel weight 0 model y candidate 56

Mean-Shift Object Tracking Similarity Function Target model: Target candidate: Similarity function: 1 1 The Bhattacharyya Coefficient 57

Mean-Shift Object Tracking Target Localization Algorithm Start from the position of the model in the current frame Search in the model’s neighborhood in next frame Find best candidate by maximizing a similarity func. 58

Linear approx. (around y 0 ) Mean-Shift Object Tracking Approximating the Similarity Function Model location: Candidate location: Independent of y Density estimate! (as a function of y) 59

Mean-Shift Object Tracking Maximizing the Similarity Function The mode of = sought maximum Important Assumption: One mode in the searched neighborhood The target representation provides sufficient discrimination 60

Mean-Shift Object Tracking Applying Mean-Shift Original Mean-Shift: Find mode ofusing The mode of = sought maximum Extended Mean-Shift: Find mode of using 61

Mean-Shift Object Tracking About Kernels and Profiles A special class of radially symmetric kernels: The profile of kernel K Extended Mean-Shift: Find mode of using 62

Mean-Shift Object Tracking Choosing the Kernel Epanechnikov kernel: A special class of radially symmetric kernels: Extended Mean-Shift: Uniform kernel: 63

64