1. Chapter 14: SEGMENTATION BY CLUSTERING 1

2. 2

3. Outline Introduction Human Vision & Gestalt Properties Applications – Background Subtraction – Shot Boundary Detection Segmentation by Clustering – Simple Clustering Methods – K-means Segmentation by Graph-Theoretic Clustering – The Overall Approach – Affinity Measures – Normalized Cut 3

4. Introduction 4

5. General Ideas Tokens – Whatever we need to group (pixels, points, surface elements, etc., etc.) Top down segmentation – Tokens belong together because they lie on the same object Bottom up segmentation – Tokens belong together because they are locally coherent 5

6. Why & What is segmentation – Emphasize the property – Make main idea interesting – There are too many pixels in a single image – We need some form of compact, summary representation 6

7. What segmentation can do – Summarizing video – Finding machined parts – Finding people – Finding buildings in satellite images – Searching a collection of images 7

8. Model Problems – Forming image segments Roughly coherent color and texture – Finding body segments Arms, torso, etc. – Fitting lines to edge points To fit a line to a set of points 8

9. Segmentation as Clustering Partitioning – Decompose an image into regions – Decompose an image into extended blobs – Take a video sequence and decompose it into shots Grouping 9

10. Human Vision & Gestalt Properties 10

11. Figure-ground discrimination – Grouping can be seen in terms of allocating some elements to a figure, some to ground – Can be based on local bottom-up cues or high level recognition 11

12. Figure-ground discrimination 12

13. A series of factors affect whether elements should be grouped together 13

14. A series of factors affect whether elements should be grouped together 14

15. 15

16. 16

17. Grouping by occlusions 17

18. Grouping by invisible completions 18

19. Outline Introduction Human Vision & Gestalt Properties Applications – Background Subtraction – Shot Boundary Detection Segmentation by Clustering – Simple Clustering Methods – K-means Segmentation by Graph-Theoretic Clustering – The Overall Approach – Affinity Measures – Normalized Cut 19

20. Application Background Subtraction Anything that doesn’t look like a known background is interesting 20

21. Take a picture? Not good… The background changes slowly overtime… 21

22. Estimate the value of background pixels using a moving average. Ideally, the moving average should track the changes in the background. 22

23. 23

24. Shot Boundary Detection Where substantial changes in a video are interesting. 24

25. 25

26. Computing distance – Frame differencing – Histogram-based – Block comparison – Edge differencing pixel-by-pixel histogram-by-histogram box-by-box edge map-by-edge map 26

27. Outline Introduction Human Vision & Gestalt Properties Applications – Background Subtraction – Shot Boundary Detection Segmentation by Clustering – Simple Clustering Methods – K-means Segmentation by Graph-Theoretic Clustering – The Overall Approach – Affinity Measures – Normalized Cut 27

28. Segmentation by Clustering Simple Clustering Methods Divisive clustering 28

29. Divisive clustering 29 p, q, r, s, t r, s, t p, qqp r s, t st

30. Agglomerative clustering 30

31. Agglomerative clustering 31 p, q, r, s, t qp p, q r r, s, t st s, t

32. Q: what the distance of A: 32

33. Single-link clustering 33

34. Complete-link clustering 34

35. Group average clustering 35

36. K-means 36

37. 37

38. 38

39. Outline Introduction Human Vision & Gestalt Properties Applications – Background Subtraction – Shot Boundary Detection Segmentation by Clustering – Simple Clustering Methods – K-means Segmentation by Graph-Theoretic Clustering – The Overall Approach – Affinity Measures – Normalized Cut 39

40. Segmentation by Graph-theoretic Clustering Terminology for Graph – Vertices – Edges Directed graph Undirected graph Weighted graph Self-loop Connected graph – Connected components 40 4 2 3

41. Weighted graph represented by a square matrix The Overall Approach 41 Undirected weighted graph Weighted matrix

42. 42 Set of well separated pointsAffinity matrix

43. Affinity Measures Affinity by Distance 43

44. Affinity by Intensity Affinity by Color Affinity by Texture 44

45. Eigenvectors and segmentation 45 Extracting a Single Good Cluster { association of element I with cluster n } × { affinity between I and j } × { association of element j with cluster n} :Element affinities :The vector of weight linking elements to the th cluster

46. 46 Scaling by scales Normalizes the weights by requiring that Differentiation and dropping :eigenvector of Lagrangian

47. 47 Small

48. Extracting Weights for a Set of Clusters – quite tight – distinct. 48 The three largest eigenvalues of the affinity matrix for the dataset

49. 49

50. Outline Introduction Human Vision & Gestalt Properties Applications – Background Subtraction – Shot Boundary Detection Segmentation by Clustering – Simple Clustering Methods – K-means Segmentation by Graph-Theoretic Clustering – The Overall Approach – Affinity Measures – Normalized Cut 50

51. Normalized Cut 51

52. Association of set: A B 52 Normalized cut: Cut of set: A B A B Minimize

53. Minimize Ncut – Transform Ncut equation to a matrical form. 53 052000000 501200000 210200.100 0220 0000 000 03010 000030147 00 001001 000014002 000007120 700000000 080000000 004.000000 000 00000 0000 0000 00000 15 000 0000005.00 000000070 00000000 10 NP-Hard!

54. Real Vector & Threshold 54 052000000 501200000 210200.100 0220 0000 000 03010 000030147 00 001001 000014002 000007120 700000000 080000000 004.000000 000 00000 0000 0000 000005000 0000005.00 000000070 000000001 If y is allowed to take real values then the minimization can be done by solving the generalized eigenvalue system Note that,so the first eigenvector is y=I with eigenvalue 0. The second smallest eigenvector is the real valued solution to this problem! Choosing a threshold to decide that nodes are assigned to one side or the other.

55. Summarize 1.Define a similarity function between 2 nodes. 2.Compute affinity matrix ( A ) and degree matrix (D). 3.Solve 4.Use the eigenvector with the second smallest eigenvalue to bipartition the graph. 5.Recursively partition the segmented parts if necessary. 55

56. Example 56

57. 57

58. Thank you 58