Emel Doğrusöz Esra Ataer Muhammet Baştan Tolga Can Image Clustering Emel Doğrusöz Esra Ataer Muhammet Baştan Tolga Can 11.01.2019 Information Retrieval Systems

Information Retrieval Systems Outline What is image clustering/segmentation? Why is it used? Segmentation algorithms. Effectiveness How C3M can be applied to image segmentation? Summary 11.01.2019 Information Retrieval Systems

Information Retrieval Systems What defines an object? "I stand at the window and see a house, trees, sky. Theoretically I might say there were 327 brightnesses and nuances of colour. Do I have "327"? No. I have sky, house, and trees." --Max Wertheimer 11.01.2019 Information Retrieval Systems

Segmentation and Grouping •To recognize objects – rather than dealing with too many pixels – we need a compact/summary representation •Obtain this representation from image/motion sequence/set of tokens •“What is interesting and what is not” depends on the application 11.01.2019 Information Retrieval Systems

Information Retrieval Systems Image segmentation Segmentation = splitting an image into regions based on some criteria (intensity, color, texture, orientation energy, …). 11.01.2019 Information Retrieval Systems

Segmentation and Grouping • Tokens – whatever we need to group (pixels, points, surface elements, etc.) • Grouping (or clustering) – collect together tokens that “belong together” – top down segmentation • tokens belong together because they lie on the same object – bottom up segmentation • tokens belong together because they are locally coherent • Fitting – associate a model with tokens – Issues: which model? which token goes to which element? how many elements in the model? 11.01.2019 Information Retrieval Systems

Information Retrieval Systems Image segmentation Haralick and Shapiro ”The regions should be uniform and homogenous with respect to some characteristic such as intensity value or texture. Region interiors should be simple and without many small holes. Adjacent regions should have significantly different values with respect to the characteristic on which they are uniform. Boundaries of each segment should be simple, not ragged, and must be spatially accurate.” 11.01.2019 Information Retrieval Systems

Information Retrieval Systems Applications Object recognition : recognize objects from image Optical character recognition (characters, words from document image) Face recognition Fingerprint recognition Medical image processing - diagnosis e.g., cancerous/healthy cell Industrial automation Content based image retrieval e.g., searching for an object class from a video database 11.01.2019 Information Retrieval Systems

Segmentation Algorithms Simple Segmentation Algorithms Thresholding Background Subtraction Segmentation by Clustering Simple Clustering Methods K-means Segmentation by Graph-Theoretic Clustering Normalized Cuts 11.01.2019 Information Retrieval Systems

Simple Clustering Methods Two natural Algorithms: Agglomerative clustering attach closest to cluster it is closest to repeat Divisive clustering split cluster along best boundary 11.01.2019 Information Retrieval Systems

Simple Clustering Methods Point-Cluster distance single-link clustering complete-link clustering group-average clustering Dendrograms yield a picture of output as clustering process continues 11.01.2019 Information Retrieval Systems

Information Retrieval Systems Divisive Methods Construct a single cluster containing all points Until the clustering is satisfactory - Split the cluster that yields the two components with the largest intercluster distance 11.01.2019 Information Retrieval Systems

Divisive Methods, an Example 11.01.2019 Information Retrieval Systems

Agglomerative Methods Make each point a separate cluster Until the clustering is satisfactory Merge the two clusters with the smallest inter-cluster distance 11.01.2019 Information Retrieval Systems

Information Retrieval Systems Thresholding Gray level thresholding is the simplest segmentation process. Multilevel thresholding (object) (background) 11.01.2019 Information Retrieval Systems

Information Retrieval Systems Thresholding Thresholding is computationally inexpensive and fast Correct threshold selection is crucial for successful threshold segmentation 11.01.2019 Information Retrieval Systems

Thresholding-example 11.01.2019 Information Retrieval Systems

Information Retrieval Systems K-means Clustering Choose a fixed number of clusters Choose cluster centers and point-cluster allocations to minimize error 11.01.2019 Information Retrieval Systems

Information Retrieval Systems K-means Algorithm Choose k data points to act as cluster centers Until the clustering is satisfactory Assign each data point to the cluster that has the nearest cluster center Ensure each cluster has at least one data point splitting, etc Replace the cluster centers with the means of the elements in the clusters 11.01.2019 Information Retrieval Systems

Information Retrieval Systems Image Clusters on intensity Clusters on color I gave each pixel the mean intensity or mean color of its cluster --- this is basically just vector quantizing the image intensities/colors. Notice that there is no requirement that clusters be spatially localized and they’re not. K-means clustering using intensity alone and color alone 11.01.2019 Information Retrieval Systems

Segmentation By Graph-Theoretic Clustering Form the graph Cut the graph Consider each connected part as a cluster 11.01.2019 Information Retrieval Systems

Information Retrieval Systems Forming Graph Each pixel is treated as a node in a graph Each node (pixel) is linked to its neighboring nodes (pixels), the strength of each link (in graph theory, a link is also called an edge) is determined by the affinity measure of the two associated nodes (pixels). For example, if two pixels have similar color, or texture, or motion, then the link between them is strong 11.01.2019 Information Retrieval Systems

Information Retrieval Systems Criterias Affinity by Distance Affinity by Intensity Affinity by Color Affinity by Texture 11.01.2019 Information Retrieval Systems

Information Retrieval Systems Example Graph 11.01.2019 Information Retrieval Systems

Information Retrieval Systems Example Graph 11.01.2019 Information Retrieval Systems

Information Retrieval Systems Cut the Graph To separate pixels into groups, we need to “cut” some edges. Graph cut methods find the cut that minimizes the total edge weights in the cut while maximizing the edge weights in the clusters 11.01.2019 Information Retrieval Systems

Information Retrieval Systems Cutting the Graph 11.01.2019 Information Retrieval Systems

Information Retrieval Systems 11.01.2019 Information Retrieval Systems

Information Retrieval Systems 11.01.2019 Information Retrieval Systems

Information Retrieval Systems Eigenvectors and Cut Aim : to assign each pixel to a cluster while maximizing the affinity between pixels in each cluster We could maximize But we have to satisfy This is an eigenvalue problem – choose eigenvector of A with largest eigenvalue 11.01.2019 Information Retrieval Systems

Information Retrieval Systems Algorithm 11.01.2019 Information Retrieval Systems

Min Cut is not always the best 11.01.2019 Information Retrieval Systems

Information Retrieval Systems Normalized Cuts A cut penalizes large segments Fix by normalizing for size of segments Volume( A ) = sum of costs of all edges that touch A 11.01.2019 Information Retrieval Systems

Information Retrieval Systems Example: 11.01.2019 Information Retrieval Systems

Background subtraction If we know the background Use moving average etc. to estimage backgroud Subtract the background from the current frame Large absolute values are interesting pixels Applications Traffic monitoring Surveillance/security User interaction 11.01.2019 Information Retrieval Systems

Background subtraction Segment moving foreground from static bacground Current image Background image Foreground pixels 11.01.2019 Information Retrieval Systems

Information Retrieval Systems Some Other Methods Watershed Segmentation Fitting & Hough Transform Fit line, circle, etc. Segmentation with Expectation Maximization (EM) New algorithms are emerging... 11.01.2019 Information Retrieval Systems

Quality of Segmentation How to judge the effectiveness of the segmentation algorithm? The Berkeley Segmentation Dataset and Benchmark 12000 hand labeled segmentation of images from Corel data set 11.01.2019 Information Retrieval Systems

Quality of Segmentation 11.01.2019 Information Retrieval Systems http://www.eecs.berkeley.edu/Research/Projects/CS/vision/grouping/segbench/BSDS300/html/dataset/images.html

Quality of Segmentation In scientific articles, the output of the segmentation algorithm is given for some example images.  judging by visual inspection 11.01.2019 Information Retrieval Systems From Trecvid 2003 News Videos K-means, K = 5

Information Retrieval Systems C3M What should correspond to the document vectors? Individual pixels A group of pixels Entire image 11.01.2019 Information Retrieval Systems

Information Retrieval Systems C3M Individual pixel  document vector If single band (e.g., grayscale image) One value per pixel (if no position info is kept) Histogram will be too sparse, only one nonzero. Ex: [ 0 0 0 ... 1 ... 0 0 0] (1 x #bins), some columns will be zero, not desired. Multiple bands (e.g., RGB, HSV images) 3 or more values per pixel D-matrix : (#pixels) x (#columns) Ex: 264 x 352 HSV image  92928 x 3 D-matrix 11.01.2019 Information Retrieval Systems

Information Retrieval Systems C3M Group of pixels (e.g., 4x4, 8x8 grids) Vectors can contain more data Mean, std of pixel values, texture, etc. Entire images (to group similar images) Several color, texture features can be extracted from each image to form a feature vector corresponding to a document vector 11.01.2019 Information Retrieval Systems

Information Retrieval Systems C3M - Effectiveness Will the resulting segmentation be meaningfull? Advantage over K-means Number of clusters can be estimated beforehand Single pass (may be more efficient, faster) Is it possible to apply C3M in a graph theoretic clustering algorithm (e.g., Normalized Cuts)?  better segmentation 11.01.2019 Information Retrieval Systems

Information Retrieval Systems Summary Segmentation is still an unsolved problem in image processing & computer vision. Some algorithms perform well on some specific domains and poorly on others. “Normalized Cuts” is nowadays popular, usually giving the best segmentation. Variations of the existing algorithms are emerging continuously. C3M can be adapted to image segmentation 11.01.2019 Information Retrieval Systems