Machine Learning on Images Janez Brank

Introduction  Collections of images (a.k.a. pictorial databases)  Image retrieval  Image classification –How do we represent images? –How do we classify them based on this representation?

Global image descriptors (1)  Describe entire image with one vector  Histograms –Split (or “quantize”) the color space into C disjoint regions. –For each color region, remember what percentage of the image is covered by pixels whose colors are from that region.  Color coherence vectors (Pass et al., 1996, 1999) –Distinguish pixels belonging to a larger patch of pixels of the same color from the “lonelier” ones.

Global image descriptors (2)  Autocorrelograns (Huang et al., 1997) –Given a randomly chosen pixel of color c and a randomly chosen pixel at distance d from it, what is the probability that this second pixel also has color c? –Store this for all c and for some d.  Banded autocorrelograms (Huang et al., 1998) –For each c, sum the corresponding acg entries over all d.

Segmentation  Divide the image into several “regions”. Each should be uniform in terms of color and/or texture. –Describe each region separately. –How do we define similarity between images based on the similarity between regions?  WALRUS (Natsev et al., 1998) –Slice the image into small windows or tiles (e.g. 4  4 pixels) –Describe a window using the principal coefficients of the wavelet transform (  a 12-dimensional vector) –Cluster the descriptions (and hence windows).

Similarity measures for segmented images  WALRUS: –Look for pairs of regions (one region from each image) whose descriptions are close enough. –Calculate the percentage of the images covered by these regions.  IRM (integrated region matching; Li et al., 2000) –Distance between images is expressed as a weighted sum of distances between regions –To assign weights, similarity between regions, as well as their size, is taken into account.

Machine learning techniques  The k nearest neighbor method: –Classify a new instance by finding the k training instances closest to it and choosing the class prevailing among them.  Support vector machines: –Instances assumed to be vectors; separate two classes by placing a hyperplane between them. –By pretending to map the vectors into a different space (  kernel functions), we can achieve non-linear (e.g. polynomial) discrimination surfaces.

The database  1172 images, chosen from a larger collection, classified manually into 14 disjoint classes

The experiments  Global image descriptors: –histograms, coherence vectors, autocorrelograms, banded autocorrelograms –Several color space quantizations: RGB (64, 216, 256 regions), HSV (256 regions) –Nearest neighbors (using L 1 - ali L 2 -norm as the distance measure) –Support vector machines (three different kernels)  Segmentation: –Various segmentation parameters –Two similarity measures: WALRUS in IRM –Nearest neighbors

Poskusi z globalnimi opisi slik

Poskusi s segmentacijo

Conclusions  Autocorrelograms better than histograms, banded acg as good as full ones  The quantization of the color space should not be too rough

Conclusions  Autocorrelograms better than histograms, banded acg as good as full ones  The quantization of the color space should not be too rough  Support vector machines produce better classifiers than the nearest-neighbor method

Conclusions  Autocorrelograms better than histograms, banded acg as good as full ones  The quantization of the color space should not be too rough  Support vector machines produce better classifiers than the nearest-neighbor method  Cubic and radial kernels better than linear

Conclusions  Autocorrelograms better than histograms, banded acg as good as full ones  The quantization of the color space should not be too rough  Support vector machines produce better classifiers than the nearest-neighbor method  Cubic, radial kernels better than linear  Segmentation parameters are important  IRM is better than the WALRUS similarity measure  Segmentation is not better than global descriptors

Conclusions  Autocorrelograms better than histograms, banded acg as good as full ones  The quantization of the color space should not be too rough  Support vector machines produce better classifiers than the nearest-neighbor method  Cubic, radial kernels better than linear  Segmentation parameters are important  IRM is better than the WALRUS similarity measure  Segmentation is not better than global descriptors  Looking at more nearest neigbors yields worse results

Zaključki  Avtokorelogrami boljši od histogramov, zgoščeni AKG enako dobri kot navadni  Metoda podpornih vektorjev daje boljše klasifikatorje od metode najbližjih sosedov  Metoda k najbližjih sosedov daje najboljše rezultate pri k = 1  Kvantizacija barvnega prostora ne sme biti pregroba  IRM daje boljše rezultate od WALRUS ove mere podobnosti  Klasifikatorji, dobljeni na podlagi segmentacije, niso nič boljši od tistih na podlagi globalnih opisov  S parametri segmentacije se da na točnost precej vplivati

Comparison with related work  Huang et al. (1998) built a sort of decision tree over a set of images, and also compared it with NN –Banded autocorrelograms, SVD (for each node of the tree) –Eleven classes, 90 images per class –Classification accuracies: around 80 %

Possibilities of future work  Understanding the unexpected results –The poor performance of segmentation-based classifiers –Why is k-NN best when k = 1?  Compare with other techniques –Other segmentation methods –Other color spaces  Looking for new image representation methods and new similarity measures