Visual Querying By Color Perceptive Regions Alberto del Bimbo, M. Mugnaini, P. Pala, and F. Turco University of Florence, Italy Pattern Recognition, 1998

Introduction Relevance of visual elements depends on: 1. User’s subjectivity (not known in advance) 2. Context of application (known in advance) Efficient system for retrieval by visual content: I. Provide a query paradigm that allow users to naturally specify both selective and imprecise queries II. Define retrieval facilities that are meaningful for the context of application III. Define similarity metrics which are satisfactory for user perception

Introduction PICASSO system allows image indexing and retrieval based on shapes, colors, and spatial relationships This paper concentrates on color facilities System supports both retrieval by global color similarity and retrieval by similarity of local color regions Shape, size, and position of color regions are considered as optional features that the user can select in the query

Hierarchical Color Image Segmentation Image segmentation: partition of an image into a set of non-overlapped regions - homogeneous with respect to some criteria - whose union covers the entire image PICASSO: Multiple descriptions of each image, each covers a different level of precision Each database image is segmented into uniform color regions at different degrees of resolution, so as to obtain a pyramidal multi-resolution representation

Hierarchical Color Image Segmentation Minimize the associated energy: By a heuristic approach, starting from the finest level of resolution: Every pair of adjacent regions is checked to verify if their merge decreases the image energy; the two regions that provide the maximum decrease of image energy are merged Continue the above process until a minimum of image energy is reached Parameters are then changed so that a new minimum is reached with a lower number of regions

Hierarchical Color Image Segmentation After the segmentation process, for an image, N segmented images are obtained and represented through a multi-layered graph Region V n.k is connected through intra-level links to neighboring regions, and through inter-level links to its son regions at layer n-1 The graph is a multi-resolution index of the chromatic and positional content of the image

Color Region Representation Use CIE L*u*v* color space close distances in color space correspond to close distances in user’s perception Computation of color distance between two generic points in L*u*v* space requires to evaluate the length of the shortest path linking the two points Not possible in real time because of its complexity

Color Region Representation Uniform tessellation of L*u*v* color space Number of colors reduced to a small set of reference colors Distance between two generic colors belonging to the same reference color is well approximated by Euclidean distance Experiments showed 128 colors suffice to achieve a reasonable compromise between accuracy and computational effort Distances between reference colors are pre- computed

Region Description At coarsest resolution: image is represented by a single region and color vector retains global color characteristics for the entire image As resolution increases: regions correspond to smaller areas in image and therefore have a smaller number of reference colors For a generic region R n (at level n of resolution), with k-child regions (at level n-1), the color vector is computed as union of color vectors associated with child regions, hence

Region Description Color regions are modeled through: Area Where #R = region and #I = image pixels Spatial location Absolute position of its centroid Shape Using the first 13 central moments defined as: Binary 128-dimensional color vector

Color Image Retrieval PICASSO supports retrieval by visual example of images with one or more colored regions Queries are formulated through visual examples Regions can be either sketched and then filled with appropriate colors or extracted from images Similarity of color regions takes into account either chromatic qualities of sketched regions or combination of chromatic and spatial attributes Highest node of each pyramid includes both the binary color vector associated with the whole image and the full image histogram

Color Image Retrieval Color index file: stores the color vectors associated with the highest node of each image pyramid For pruning unqualified images Matching score (M) between a query region R Q and a database image region R I : Similarity coefficient for the whole image:

Results PICASSO allows: Querying by color regions (positions are unimportant) Querying by global color similarity (histogram) Test for querying by color regions: Features: color/area, color/position, color/position/area Retrieved images were shown to (30) users who evaluate if retrieved images are relevant/ not relevant Use data to compute precision and recall Best values for color/position queries

Conclusion I. System is used by Alinari Archives in Florence II. Multi-resolution segmentation III. Algorithms and index structure seem computationally complex