1. Emel Doğrusöz Esra Ataer Muhammet Baştan Tolga Can
Image Clustering
Emel Doğrusöz
Esra Ataer
Muhammet Baştan
Tolga Can
Information Retrieval Systems

2. Information Retrieval Systems
Outline
What is image clustering/segmentation?
Why is it used?
Segmentation algorithms.
Effectiveness
How C3M can be applied to image segmentation?
Summary
Information Retrieval Systems

3. Information Retrieval Systems
What defines an object?
"I stand at the window and see a house, trees, sky. Theoretically I
might say there were 327 brightnesses and nuances of colour. Do
I have "327"? No. I have sky, house, and trees." --Max Wertheimer
Information Retrieval Systems

4. Segmentation and Grouping
•To recognize objects – rather than dealing with too many pixels – we need a compact/summary representation
•Obtain this representation from image/motion sequence/set of tokens
•“What is interesting and what is not” depends on the application
Information Retrieval Systems

5. Information Retrieval Systems
Image segmentation
Segmentation = splitting an image into regions based on some criteria (intensity, color, texture, orientation energy, …).
Information Retrieval Systems

6. Segmentation and Grouping
• Tokens
– whatever we need to group (pixels, points, surface elements, etc.)
• Grouping (or clustering)
– collect together tokens that “belong together”
– top down segmentation
• tokens belong together because they lie on the same object
– bottom up segmentation
• tokens belong together because they are locally coherent
• Fitting
– associate a model with tokens
– Issues: which model? which token goes to which element? how many elements in the model?
Information Retrieval Systems

7. Information Retrieval Systems
Image segmentation
Haralick and Shapiro
”The regions should be uniform and homogenous with respect to some characteristic such as intensity value or texture. Region interiors should be simple and without many small holes. Adjacent regions should have significantly different values with respect to the characteristic on which they are uniform. Boundaries of each segment should be simple, not ragged, and must be spatially accurate.”
Information Retrieval Systems

8. Information Retrieval Systems
Applications
Object recognition : recognize objects from image
Optical character recognition (characters, words from document image)
Face recognition
Fingerprint recognition
Medical image processing - diagnosis
e.g., cancerous/healthy cell
Industrial automation
Content based image retrieval
e.g., searching for an object class from a video database
Information Retrieval Systems

9. Segmentation Algorithms
Simple Segmentation Algorithms
Thresholding
Background Subtraction
Segmentation by Clustering
Simple Clustering Methods
K-means
Segmentation by Graph-Theoretic Clustering
Normalized Cuts
Information Retrieval Systems

10. Simple Clustering Methods
Two natural Algorithms:
Agglomerative clustering
attach closest to cluster it is closest to
repeat
Divisive clustering
split cluster along best boundary
Information Retrieval Systems

11. Simple Clustering Methods
Point-Cluster distance
single-link clustering
complete-link clustering
group-average clustering
Dendrograms
yield a picture of output as clustering process continues
Information Retrieval Systems

12. Information Retrieval Systems
Divisive Methods
Construct a single cluster containing all points
Until the clustering is satisfactory
- Split the cluster that yields the two components with the largest intercluster distance
Information Retrieval Systems

13. Divisive Methods, an Example
Information Retrieval Systems

14. Agglomerative Methods
Make each point a separate cluster
Until the clustering is satisfactory
Merge the two clusters with the smallest inter-cluster distance
Information Retrieval Systems

15. Information Retrieval Systems
Thresholding
Gray level thresholding is the simplest segmentation process.
Multilevel thresholding
(object)
(background)
Information Retrieval Systems

16. Information Retrieval Systems
Thresholding
Thresholding is computationally inexpensive and fast
Correct threshold selection is crucial for successful threshold segmentation
Information Retrieval Systems

17. Thresholding-example
Information Retrieval Systems

18. Information Retrieval Systems
K-means Clustering
Choose a fixed number of clusters
Choose cluster centers and point-cluster allocations to minimize error
Information Retrieval Systems

19. Information Retrieval Systems
K-means Algorithm
Choose k data points to act as cluster centers
Until the clustering is satisfactory
Assign each data point to the cluster that has the nearest cluster center
Ensure each cluster has at least one data point splitting, etc
Replace the cluster centers with the means of the elements in the clusters
Information Retrieval Systems

20. Information Retrieval Systems
Image
Clusters on intensity
Clusters on color
I gave each pixel the mean intensity or mean color of its cluster --- this is basically just vector quantizing the image intensities/colors. Notice that there is no requirement that clusters be spatially localized and they’re not.
K-means clustering using intensity alone and color alone
Information Retrieval Systems

21. Segmentation By Graph-Theoretic Clustering
Form the graph
Cut the graph
Consider each connected part as a cluster
Information Retrieval Systems

22. Information Retrieval Systems
Forming Graph
Each pixel is treated as a node in a graph
Each node (pixel) is linked to its neighboring nodes (pixels), the strength of each link (in graph theory, a link is also called an edge) is determined by the affinity measure of the two associated nodes (pixels). For example, if two pixels have similar color, or texture, or motion, then the link between them is strong
Information Retrieval Systems

23. Information Retrieval Systems
Criterias
Affinity by Distance
Affinity by Intensity
Affinity by Color
Affinity by Texture
Information Retrieval Systems

24. Information Retrieval Systems
Example Graph
Information Retrieval Systems

25. Information Retrieval Systems
Example Graph
Information Retrieval Systems

26. Information Retrieval Systems
Cut the Graph
To separate pixels into groups, we need to “cut” some edges. Graph cut methods find the cut that minimizes the total edge weights in the cut while maximizing the edge weights in the clusters
Information Retrieval Systems

27. Information Retrieval Systems
Cutting the Graph
Information Retrieval Systems

28. Information Retrieval Systems
Information Retrieval Systems

29. Information Retrieval Systems
Information Retrieval Systems

30. Information Retrieval Systems
Eigenvectors and Cut
Aim : to assign each pixel to a cluster while maximizing the affinity between pixels in each cluster
We could maximize
But we have to satisfy
This is an eigenvalue problem – choose eigenvector of A with largest eigenvalue
Information Retrieval Systems

31. Information Retrieval Systems
Algorithm
Information Retrieval Systems

32. Min Cut is not always the best
Information Retrieval Systems

33. Information Retrieval Systems
Normalized Cuts
A cut penalizes large segments
Fix by normalizing for size of segments
Volume( A ) = sum of costs of all edges that touch A
Information Retrieval Systems

34. Information Retrieval Systems
Example:
Information Retrieval Systems

35. Background subtraction
If we know the background
Use moving average etc. to estimage backgroud
Subtract the background from the current frame
Large absolute values are interesting pixels
Applications
Traffic monitoring
Surveillance/security
User interaction
Information Retrieval Systems

36. Background subtraction
Segment moving foreground from static bacground
Current image
Background image
Foreground pixels
Information Retrieval Systems

37. Information Retrieval Systems
Some Other Methods
Watershed Segmentation
Fitting & Hough Transform
Fit line, circle, etc.
Segmentation with Expectation Maximization (EM)
New algorithms are emerging...
Information Retrieval Systems

38. Quality of Segmentation
How to judge the effectiveness of the segmentation algorithm?
The Berkeley Segmentation Dataset and Benchmark
12000 hand labeled segmentation of images from Corel data set
Information Retrieval Systems

39. Quality of Segmentation
Information Retrieval Systems

40. Quality of Segmentation
In scientific articles, the output of the segmentation algorithm is given for some example images.
 judging by visual inspection
Information Retrieval Systems
From Trecvid 2003 News Videos
K-means, K = 5

41. Information Retrieval Systems
C3M
What should correspond to the document vectors?
Individual pixels
A group of pixels
Entire image
Information Retrieval Systems

42. Information Retrieval Systems
C3M
Individual pixel  document vector
If single band (e.g., grayscale image)
One value per pixel (if no position info is kept)
Histogram will be too sparse, only one nonzero Ex: [ ] (1 x #bins), some columns will be zero, not desired.
Multiple bands (e.g., RGB, HSV images)
3 or more values per pixel
D-matrix : (#pixels) x (#columns)
Ex: 264 x 352 HSV image  x 3 D-matrix
Information Retrieval Systems

43. Information Retrieval Systems
C3M
Group of pixels (e.g., 4x4, 8x8 grids)
Vectors can contain more data
Mean, std of pixel values, texture, etc.
Entire images (to group similar images)
Several color, texture features can be extracted from each image to form a feature vector corresponding to a document vector
Information Retrieval Systems

44. Information Retrieval Systems
C3M - Effectiveness
Will the resulting segmentation be meaningfull?
Advantage over K-means
Number of clusters can be estimated beforehand
Single pass (may be more efficient, faster)
Is it possible to apply C3M in a graph theoretic clustering algorithm (e.g., Normalized Cuts)?  better segmentation
Information Retrieval Systems

45. Information Retrieval Systems
Summary
Segmentation is still an unsolved problem in image processing & computer vision.
Some algorithms perform well on some specific domains and poorly on others.
“Normalized Cuts” is nowadays popular, usually giving the best segmentation.
Variations of the existing algorithms are emerging continuously.
C3M can be adapted to image segmentation
Information Retrieval Systems