Chapter 14: SEGMENTATION BY CLUSTERING 1
2
Outline Introduction Human Vision & Gestalt Properties Applications – Background Subtraction – Shot Boundary Detection Segmentation by Clustering – Simple Clustering Methods – K-means Segmentation by Graph-Theoretic Clustering – The Overall Approach – Affinity Measures – Normalized Cut 3
Introduction 4
General Ideas Tokens – Whatever we need to group (pixels, points, surface elements, etc., etc.) Top down segmentation – Tokens belong together because they lie on the same object Bottom up segmentation – Tokens belong together because they are locally coherent 5
Why & What is segmentation – Emphasize the property – Make main idea interesting – There are too many pixels in a single image – We need some form of compact, summary representation 6
What segmentation can do – Summarizing video – Finding machined parts – Finding people – Finding buildings in satellite images – Searching a collection of images 7
Model Problems – Forming image segments Roughly coherent color and texture – Finding body segments Arms, torso, etc. – Fitting lines to edge points To fit a line to a set of points 8
Segmentation as Clustering Partitioning – Decompose an image into regions – Decompose an image into extended blobs – Take a video sequence and decompose it into shots Grouping 9
Human Vision & Gestalt Properties 10
Figure-ground discrimination – Grouping can be seen in terms of allocating some elements to a figure, some to ground – Can be based on local bottom-up cues or high level recognition 11
Figure-ground discrimination 12
A series of factors affect whether elements should be grouped together 13
A series of factors affect whether elements should be grouped together 14
15
16
Grouping by occlusions 17
Grouping by invisible completions 18
Outline Introduction Human Vision & Gestalt Properties Applications – Background Subtraction – Shot Boundary Detection Segmentation by Clustering – Simple Clustering Methods – K-means Segmentation by Graph-Theoretic Clustering – The Overall Approach – Affinity Measures – Normalized Cut 19
Application Background Subtraction Anything that doesn’t look like a known background is interesting 20
Take a picture? Not good… The background changes slowly overtime… 21
Estimate the value of background pixels using a moving average. Ideally, the moving average should track the changes in the background. 22
23
Shot Boundary Detection Where substantial changes in a video are interesting. 24
25
Computing distance – Frame differencing – Histogram-based – Block comparison – Edge differencing pixel-by-pixel histogram-by-histogram box-by-box edge map-by-edge map 26
Outline Introduction Human Vision & Gestalt Properties Applications – Background Subtraction – Shot Boundary Detection Segmentation by Clustering – Simple Clustering Methods – K-means Segmentation by Graph-Theoretic Clustering – The Overall Approach – Affinity Measures – Normalized Cut 27
Segmentation by Clustering Simple Clustering Methods Divisive clustering 28
Divisive clustering 29 p, q, r, s, t r, s, t p, qqp r s, t st
Agglomerative clustering 30
Agglomerative clustering 31 p, q, r, s, t qp p, q r r, s, t st s, t
Q: what the distance of A: 32
Single-link clustering 33
Complete-link clustering 34
Group average clustering 35
K-means 36
37
38
Outline Introduction Human Vision & Gestalt Properties Applications – Background Subtraction – Shot Boundary Detection Segmentation by Clustering – Simple Clustering Methods – K-means Segmentation by Graph-Theoretic Clustering – The Overall Approach – Affinity Measures – Normalized Cut 39
Segmentation by Graph-theoretic Clustering Terminology for Graph – Vertices – Edges Directed graph Undirected graph Weighted graph Self-loop Connected graph – Connected components
Weighted graph represented by a square matrix The Overall Approach 41 Undirected weighted graph Weighted matrix
42 Set of well separated pointsAffinity matrix
Affinity Measures Affinity by Distance 43
Affinity by Intensity Affinity by Color Affinity by Texture 44
Eigenvectors and segmentation 45 Extracting a Single Good Cluster { association of element I with cluster n } × { affinity between I and j } × { association of element j with cluster n} :Element affinities :The vector of weight linking elements to the th cluster
46 Scaling by scales Normalizes the weights by requiring that Differentiation and dropping :eigenvector of Lagrangian
47 Small
Extracting Weights for a Set of Clusters – quite tight – distinct. 48 The three largest eigenvalues of the affinity matrix for the dataset
49
Outline Introduction Human Vision & Gestalt Properties Applications – Background Subtraction – Shot Boundary Detection Segmentation by Clustering – Simple Clustering Methods – K-means Segmentation by Graph-Theoretic Clustering – The Overall Approach – Affinity Measures – Normalized Cut 50
Normalized Cut 51
Association of set: A B 52 Normalized cut: Cut of set: A B A B Minimize
Minimize Ncut – Transform Ncut equation to a matrical form NP-Hard!
Real Vector & Threshold If y is allowed to take real values then the minimization can be done by solving the generalized eigenvalue system Note that,so the first eigenvector is y=I with eigenvalue 0. The second smallest eigenvector is the real valued solution to this problem! Choosing a threshold to decide that nodes are assigned to one side or the other.
Summarize 1.Define a similarity function between 2 nodes. 2.Compute affinity matrix ( A ) and degree matrix (D). 3.Solve 4.Use the eigenvector with the second smallest eigenvalue to bipartition the graph. 5.Recursively partition the segmented parts if necessary. 55
Example 56
57
Thank you 58