Outline Content-Based Image Retrieval Query-by-Example Query-by-Feature Feature Vector

CBIR and CBR Content-based Image Retrieval (CBIR) como exemplo de Content-based Retrieval (CBR) concentra em low-level features. Principais idéias de CBIR: Representar uma imagem como um conjunto de feature descriptors. Definir medidas de similaridade dos descritores Quando um usuário especificar uma query, o sistema retorna imagens, que são ordenadas por similaridade.

CBIR Architecture

Image Retrieval Feature Extraction Feature Extraction Select Database Images Query Image Image Database Feature Extraction Feature Extraction Select Compare Metadatabase Feature Vectors Query Result

CBIR de Butterflies Permitir non-expert users encontrar algumas espécies de butterflies usando informações de aparência de butterflies Aparência: Color, Texture, Shape

Problemas Como podemos descrever uma butterfly? Como podemos comunicar nossa descrição para uma máquina?

Problemas Usuários diferentes têm percepções diferentes. Usuários podem não se lembrar claramente a aparêcia de uma butterfly. Usuários normalmente não têm expertise para descrever butterflies. Usuários normalmente não têm paciência para fazer o browse num grande conjunto resultado.

Soluções Usar um processo de consulta interativo e direcionado ao usuário: QBF/QBE query process Query By Features e Query By Example Fuzzy feature description para cada butterfly Uma “What You See Is What You Get” query interface Um conjunto representativo de coleção butterflies

QBF/QBE query process (1) QBF query: A QBF query is to choose some features of butterflies and expect that the system returns all butterflies with those features. Features of butterflies: Dominant color, texture pattern, shape. QBE query: A QBE query is to point an image and expect that the system returns all butterflies similar to that.

QBF/QBE query process (2) Properties of QBF: Rough search When to use: The first query and when users want to enlarge the view in the search space Properties of QBE: Fine search Usually the last query and when users want to see the neighbors of the query one in the search space.

QBF/QBE query process (3) Result page: (Each result page should contain two parts) Result Images: These are the butterfly images satisfy the query conditions. Users can invoke QBE queries from these images. Related Features: These are the features related to the previous query conditions. Users can invoke QBF queries from these features.

Feature Description (1) Feature Description for a butterfly: Like metadata which describe the appearance of this butterfly. This makes QBF queries possible. Feature Description consists of some feature descriptors. Feature descriptor: A ( “feature value” , “match level” ) pair.

Feature Description (2)

Feature Description (3) Color

Feature Description (4) Texture

Feature Description (5) Shape

Feature Description (6) QBF query: Single feature query: Result images: images with its corresponding degree of match > 0. Ranked by: degree of match. We call this ranked sequence “Feature sequence.” Multiple features query: Merge the corresponding feature sequences.

Result Presentation For QBF query: For QBE query: Property: rough search Presentation: representative butterflies only For QBE query: Property: fine search Presentation: For very similar images: present them all For less similar images: representative ones

Feature Vector Indexing Goal: To make search efficiently. Problems of Indexing in CBIR: Dimension of feature space is very high. Index structure should support Euclidean and non-Euclidean similarity measures. Solution: Dimension reduction: KLT, DCT, DWT. Similarity indexing: R*-tree, SS-tree, SR-tree.

Semi-Automatic Feature Extraction Segmentation: Background segmentation Butterfly object segmentation Feature extraction: Color: color histogram Texture: manual annotation Shape: manual annotation

Classic CBIR with Color Feature Most of the CBR systems rely on the notion of color, this may differ: Dominant color Scalable color based on color histograms (local for one region, global for the whole image) Color Structure Descriptor (incoporates the spatial structure)

What color is the apple ? We are so visual !!!! 16.04.2017 What color is the apple ? We are so visual !!!! I’d say it is Bright Red I really couldn’t tell you (I am color blind) I think it is “Crimson” It is Red!

Color Histogram: Representation A list of Color-Percentage pairs: Describe the colors and its percentages in an image.

Color Quantization Indexed Colors A jpg Image with 256-color components in each RGB channel 256 x 256 x 256 colors in total → n groups, e.g, in 256 groups, that makes a reduction 256x256, I.e., that each group takes 256 colors to count.

Similarity Measures - Overview Minkowski Similarity Distance L1 : r = 1 Distance L2 : r = 2 Quadratic Similarity Intersection Similarity (Swain et Ballard 1991)

Example (cont.) Minkowski Similarity Is a L-1 metric where Ik and Jk is the number of pixels in bin k for image I and J Distance between above three images D(H1, H2) = 1 + 1 + 1 + 1 + 1 + 1 + 1 + 1 = 8 D(H1, H3) = 6 + 6 + 2 + 2 + 2 + 2 + 2 + 2 = 24 D(H2, H3) = 5 + 5 + 3 + 3 + 1 + 1 + 1 + 1 = 23

Example (cont.) Minkowski Similarity Is a L-2 metric Distance between above three images D(H1, H2) = (1 + 1 + 1 + 1 + 1 + 1 + 1 + 1)1/2 = 2.8 D(H1, H3) = (36 + 36 + 4 + 4 + 4 + 4 + 4 + 4)1/2 = 9.8 D(H2, H3) = (25 + 25 + 9 + 9 + 1 + 1 + 1 + 1) 1/2 = 8.5

QBIC distance Weighted Euclidean distance (QBIC) Is a L-2 metric(?) distance between histogram H1 and H2: D = (H1 - H2)T A (H1 - H2) where A is a symmetric color similarity matrix A (i, j) = 1 – d (ci, cj) / dmax where ci and cj are the i-th and j-th color bins, d (ci , cj) is the color distance in the color space, and dmax is the maximum distance between any two colors in the color space

Limitation Ignore similarity between colors Example Two color bins Bin-1 color range: 1 – 10 Bin-2 color range: 11 – 20

Three color pixels Pixel 1 is Color 10  Bin-1 Pixel 2 is Color 11  Bin-2 Pixel 3 is Color 20  Bin-2 Pixel 2 is similar to Pixel 3 than Pixel 1  unreasonable !

Different image with same histogram Limitation (cont.) Ignore spatial relationships among pixels Different image with same histogram

Noise-Free Queries (NFQ’s) NFQ is more precise. User can specify semantic constraints: Spatial constraints (relative distances) Scaling constraints (relative sizes) Rectangular query Noise-free query Similar Less relevant

Challenges How do we extract features if we do not know the matching areas beforehand ? How do we index the images ? Noise-free query

One Solution – Local Color Histogram (LCH) Each subimage has a color histogram. Any combination of the histograms can be selected for comparison with the corresponding color histograms of the query image.

Limitations of LCH Dilemma: Limitation: Using large partitions is not precise Using small partitions is too expensive Limitation: difficult to handle scaling

Resultados esperados de uma boa CBIR com segmentação Query 2 3 4 5 12 18 Query 2 3 216 396

DEMOS Hermitage Museum Web Site (QBIC) http://hermitagemuseum.org/ http://hermitagemuseum.org/fcgi-bin/db2www/qbicColor.mac/qbic?selLang=English http://www.aa-lab.cs.uu.nl/cbirsurvey/cbir-survey/