1. Outline Content-Based Image Retrieval Query-by-Example
Query-by-Feature
Feature Vector

2. CBIR and CBR
Content-based Image Retrieval (CBIR) como exemplo de Content-based Retrieval (CBR)
concentra em low-level features.
Principais idéias de CBIR:
Representar uma imagem como um conjunto de feature descriptors.
Definir medidas de similaridade dos descritores
Quando um usuário especificar uma query, o sistema retorna imagens, que são ordenadas por similaridade.

3. CBIR Architecture

4. Image Retrieval Feature Extraction Feature Extraction Select
Database Images
Query Image
Image Database
Feature
Extraction
Feature
Extraction
Select
Compare
Metadatabase
Feature Vectors
Query Result

5. CBIR de Butterflies
Permitir non-expert users encontrar algumas espécies de butterflies usando informações de aparência de butterflies
Aparência:
Color, Texture, Shape

6. Problemas Como podemos descrever uma butterfly?
Como podemos comunicar nossa descrição para uma máquina?

7. Problemas Usuários diferentes têm percepções diferentes.
Usuários podem não se lembrar claramente a aparêcia de uma butterfly.
Usuários normalmente não têm expertise para descrever butterflies.
Usuários normalmente não têm paciência para fazer o browse num grande conjunto resultado.

8. Soluções
Usar um processo de consulta interativo e direcionado ao usuário: QBF/QBE query process
Query By Features e Query By Example
Fuzzy feature description para cada butterfly
Uma “What You See Is What You Get” query interface
Um conjunto representativo de coleção butterflies

9. QBF/QBE query process (1)
QBF query:
A QBF query is to choose some features of butterflies and expect that the system returns all butterflies with those features.
Features of butterflies:
Dominant color, texture pattern, shape.
QBE query:
A QBE query is to point an image and expect that the system returns all butterflies similar to that.

10. QBF/QBE query process (2)
Properties of QBF:
Rough search
When to use:
The first query and when users want to enlarge the view in the search space
Properties of QBE:
Fine search
Usually the last query and when users want to see the neighbors of the query one in the search space.

11. QBF/QBE query process (3)
Result page: (Each result page should contain two parts)
Result Images:
These are the butterfly images satisfy the query conditions.
Users can invoke QBE queries from these images.
Related Features:
These are the features related to the previous query conditions.
Users can invoke QBF queries from these features.

12. Feature Description (1)
Feature Description for a butterfly:
Like metadata which describe the appearance of this butterfly.
This makes QBF queries possible.
Feature Description consists of some feature descriptors.
Feature descriptor:
A ( “feature value” , “match level” ) pair.

13. Feature Description (2)

14. Feature Description (3)
Color

15. Feature Description (4)
Texture

16. Feature Description (5)
Shape

17. Feature Description (6)
QBF query:
Single feature query:
Result images: images with its corresponding degree of match > 0.
Ranked by: degree of match.
We call this ranked sequence “Feature sequence.”
Multiple features query:
Merge the corresponding feature sequences.

18. Result Presentation For QBF query: For QBE query:
Property: rough search
Presentation: representative butterflies only
For QBE query:
Property: fine search
Presentation:
For very similar images: present them all
For less similar images: representative ones

19. Feature Vector Indexing
Goal:
To make search efficiently.
Problems of Indexing in CBIR:
Dimension of feature space is very high.
Index structure should support Euclidean and non-Euclidean similarity measures.
Solution:
Dimension reduction: KLT, DCT, DWT.
Similarity indexing: R*-tree, SS-tree, SR-tree.

20. Semi-Automatic Feature Extraction
Segmentation:
Background segmentation
Butterfly object segmentation
Feature extraction:
Color: color histogram
Texture: manual annotation
Shape: manual annotation

21. Classic CBIR with Color Feature
Most of the CBR systems rely on the notion of color, this may differ:
Dominant color
Scalable color based on color histograms (local for one region, global for the whole image)
Color Structure Descriptor (incoporates the spatial structure)

22. What color is the apple ? We are so visual !!!!
What color is the apple ? We are so visual !!!!
I’d say it is
Bright Red
I really
couldn’t tell you
(I am color blind)
I think it is
“Crimson”
It is
Red!

23. Color Histogram: Representation
A list of Color-Percentage pairs:
Describe the colors and its percentages in an image.

24. Color Quantization Indexed Colors
A jpg Image with 256-color components in each RGB channel
256 x 256 x 256 colors in total → n groups, e.g, in 256 groups, that makes a reduction 256x256, I.e., that each group takes 256 colors to count.

25. Similarity Measures - Overview
Minkowski Similarity
Distance L1 : r = 1
Distance L2 : r = 2
Quadratic Similarity
Intersection Similarity
(Swain et Ballard 1991)

26. Example (cont.) Minkowski Similarity Is a L-1 metric
where Ik and Jk is the number of pixels in bin k for image I and J
Distance between above three images
D(H1, H2) = = 8
D(H1, H3) = = 24
D(H2, H3) = = 23

27. Example (cont.) Minkowski Similarity Is a L-2 metric
Distance between above three images
D(H1, H2) = ( )1/2 = 2.8
D(H1, H3) = ( )1/2 = 9.8
D(H2, H3) = ( ) 1/2 = 8.5

28. QBIC distance Weighted Euclidean distance (QBIC) Is a L-2 metric(?)
distance between histogram H1 and H2:
D = (H1 - H2)T A (H1 - H2)
where A is a symmetric color similarity matrix
A (i, j) = 1 – d (ci, cj) / dmax
where ci and cj are the i-th and j-th color bins,
d (ci , cj) is the color distance in the color space, and dmax is the maximum distance between any two colors in the color space

29. Limitation Ignore similarity between colors Example Two color bins
Bin-1 color range: 1 – 10
Bin-2 color range: 11 – 20

30. Three color pixels
Pixel 1 is Color 10  Bin-1
Pixel 2 is Color 11  Bin-2
Pixel 3 is Color 20  Bin-2
Pixel 2 is similar to Pixel 3 than Pixel 1  unreasonable !

31. Different image with same histogram
Limitation (cont.)
Ignore spatial relationships among pixels
Different image with same histogram

32. Noise-Free Queries (NFQ’s)
NFQ is more precise.
User can specify semantic constraints:
Spatial constraints (relative distances)
Scaling constraints (relative sizes)
Rectangular query
Noise-free
query
Similar
Less relevant

33. Challenges
How do we extract features if we do not know the matching areas beforehand ?
How do we index the images ?
Noise-free
query

34. One Solution – Local Color Histogram (LCH)
Each subimage has a color histogram.
Any combination of the histograms can be selected for comparison with the corresponding color histograms of the query image.

35. Limitations of LCH Dilemma: Limitation:
Using large partitions is not precise
Using small partitions is too expensive
Limitation:
difficult to handle scaling

36. Resultados esperados de uma boa CBIR com segmentação
Query 2 3 4 5 12 18
Query 2 3
216
396

37. DEMOS Hermitage Museum Web Site (QBIC)