Chapter 14: SEGMENTATION BY CLUSTERING 1

2

Outline Introduction Human Vision & Gestalt Properties Applications – Background Subtraction – Shot Boundary Detection Segmentation by Clustering – Simple Clustering Methods – K-means Segmentation by Graph-Theoretic Clustering – The Overall Approach – Affinity Measures – Normalized Cut 3

Introduction 4

General Ideas Tokens – Whatever we need to group (pixels, points, surface elements, etc., etc.) Top down segmentation – Tokens belong together because they lie on the same object Bottom up segmentation – Tokens belong together because they are locally coherent 5

Why & What is segmentation – Emphasize the property – Make main idea interesting – There are too many pixels in a single image – We need some form of compact, summary representation 6

What segmentation can do – Summarizing video – Finding machined parts – Finding people – Finding buildings in satellite images – Searching a collection of images 7

Model Problems – Forming image segments Roughly coherent color and texture – Finding body segments Arms, torso, etc. – Fitting lines to edge points To fit a line to a set of points 8

Segmentation as Clustering Partitioning – Decompose an image into regions – Decompose an image into extended blobs – Take a video sequence and decompose it into shots Grouping 9

Human Vision & Gestalt Properties 10

Figure-ground discrimination – Grouping can be seen in terms of allocating some elements to a figure, some to ground – Can be based on local bottom-up cues or high level recognition 11

Figure-ground discrimination 12

A series of factors affect whether elements should be grouped together 13

A series of factors affect whether elements should be grouped together 14

15

16

Grouping by occlusions 17

Grouping by invisible completions 18

Outline Introduction Human Vision & Gestalt Properties Applications – Background Subtraction – Shot Boundary Detection Segmentation by Clustering – Simple Clustering Methods – K-means Segmentation by Graph-Theoretic Clustering – The Overall Approach – Affinity Measures – Normalized Cut 19

Application Background Subtraction Anything that doesn’t look like a known background is interesting 20

Take a picture? Not good… The background changes slowly overtime… 21

Estimate the value of background pixels using a moving average. Ideally, the moving average should track the changes in the background. 22

23

Shot Boundary Detection Where substantial changes in a video are interesting. 24

25

Computing distance – Frame differencing – Histogram-based – Block comparison – Edge differencing pixel-by-pixel histogram-by-histogram box-by-box edge map-by-edge map 26

Outline Introduction Human Vision & Gestalt Properties Applications – Background Subtraction – Shot Boundary Detection Segmentation by Clustering – Simple Clustering Methods – K-means Segmentation by Graph-Theoretic Clustering – The Overall Approach – Affinity Measures – Normalized Cut 27

Segmentation by Clustering Simple Clustering Methods Divisive clustering 28

Divisive clustering 29 p, q, r, s, t r, s, t p, qqp r s, t st

Agglomerative clustering 30

Agglomerative clustering 31 p, q, r, s, t qp p, q r r, s, t st s, t

Q: what the distance of A: 32

Single-link clustering 33

Complete-link clustering 34

Group average clustering 35

K-means 36

37

38

Outline Introduction Human Vision & Gestalt Properties Applications – Background Subtraction – Shot Boundary Detection Segmentation by Clustering – Simple Clustering Methods – K-means Segmentation by Graph-Theoretic Clustering – The Overall Approach – Affinity Measures – Normalized Cut 39

Segmentation by Graph-theoretic Clustering Terminology for Graph – Vertices – Edges Directed graph Undirected graph Weighted graph Self-loop Connected graph – Connected components

Weighted graph represented by a square matrix The Overall Approach 41 Undirected weighted graph Weighted matrix

42 Set of well separated pointsAffinity matrix

Affinity Measures Affinity by Distance 43

Affinity by Intensity Affinity by Color Affinity by Texture 44

Eigenvectors and segmentation 45 Extracting a Single Good Cluster { association of element I with cluster n } × { affinity between I and j } × { association of element j with cluster n} :Element affinities :The vector of weight linking elements to the th cluster

46 Scaling by scales Normalizes the weights by requiring that Differentiation and dropping :eigenvector of Lagrangian

47 Small

Extracting Weights for a Set of Clusters – quite tight – distinct. 48 The three largest eigenvalues of the affinity matrix for the dataset

49

Outline Introduction Human Vision & Gestalt Properties Applications – Background Subtraction – Shot Boundary Detection Segmentation by Clustering – Simple Clustering Methods – K-means Segmentation by Graph-Theoretic Clustering – The Overall Approach – Affinity Measures – Normalized Cut 50

Normalized Cut 51

Association of set: A B 52 Normalized cut: Cut of set: A B A B Minimize

Minimize Ncut – Transform Ncut equation to a matrical form NP-Hard!

Real Vector & Threshold If y is allowed to take real values then the minimization can be done by solving the generalized eigenvalue system Note that,so the first eigenvector is y=I with eigenvalue 0. The second smallest eigenvector is the real valued solution to this problem! Choosing a threshold to decide that nodes are assigned to one side or the other.

Summarize 1.Define a similarity function between 2 nodes. 2.Compute affinity matrix ( A ) and degree matrix (D). 3.Solve 4.Use the eigenvector with the second smallest eigenvalue to bipartition the graph. 5.Recursively partition the segmented parts if necessary. 55

Example 56

57

Thank you 58