1. Content-Based Image Retrieval

2. What is Content-based Image Retrieval (CBIR)?
Image Search Systems that search for images by image content
<-> Keyword-based Image/Video Retrieval (ex. Google Image Search, YouTube)

3. How does CBIR work ? Extract Features from Images
Let the user do Query
Query by Sketch
Query by Keywords
Query by Example
Refine the result by Relevance Feedback
Give feedback to the previous result

4. Query by Example CBIR “Get similar images” Query Sample Results
Pick example images, then ask the system to retrieve “similar” images.
Results
CBIR
“Get similar images”
Query Sample
What does “similar” mean?

5. Relevance Feedback User gives a feedback to the query results
System recalculates feature weights
2nd Result
Feedback
1st Result
Query
Feedback
Initial
sample

6. Two Classes of CBIR Narrow vs. Broad Domain
Medical Imagery Retrieval
Finger Print Retrieval
Satellite Imagery Retrieval
Broad
Photo Collections
Internet

7. The Architecture of a typical CBIR System
Server
Image
Manager
Image
Database
Retrieval
Module
Multi-dimensional Indexing
Feature Extraction
Feature
Database
Client
User Interface
This figure shows the main components of a typical CBIR system.
For each image in an image database, visual features such as color, texture and
shape are first extracted and stored. For the purpose of efficiency,
multi-dimensional indexing techniques are often used to index the
feature vectors. When a visual query submitted by the user through the
query interface and processing module, the corresponding feature
vector of the query will be generated and compared with the feature
vectors of the images in the database by some similarity
measurements, and similar images will be returned to the user. The
user can refine the query by providing feedback information.

8. The Retrieval Process of a typical CBIR System
Feature
Database
Feature Extraction for the Query Image
Feature
Extraction 1
Extraction 2
Extraction n
( )
Feature Vector 1
( )
Feature Vector 2
( )
Feature Vector n
Image
Database
Similarity
Comparison
Image
Manager
Sorting
Query Image
(Image ID, similarity)
This figure further shows the retrieval process of a typical CBIR system, using a Rose image as the query.
Different feature vectors of the query image will be generated by different feature extraction methods, and be compared with feature vectors of the images in the database, using some similarity comparison metrics. The images will be sorted based on the calculated similarities, and the top hits will be returned to the user.
Images
Interface
Manager
Results

9. Basic Components of CBIR
Feature Extraction
Data indexing
Query and feedback processing

10. How Images are represented

11. Image Features Representing the Images Segmentation Low Level Features
Color
Texture
Shape

12. Image Features
Information about color or texture or shape which are extracted from an image are known as image features
Also a low-level features
Red, sandy
As opposed to high level features or concepts
Beaches, mountains, happy

13. Global features Averages across whole image
Tends to loose distinction between foreground and background
Poorly reflects human understanding of images
Computationally simple
A number of successful systems have been built using global image features

14. Local Features Segment images into parts Two sorts: Tile Based
Region based

15. Regioning and Tiling Schemes
Tiles
Regions

16. Tiling Break image down into simple geometric shapes
Similar Problems to Global
Plus dangers of breaking up significant objects
Computational Simple
Some Schemes seem to work well in practice

17. Regioning Break Image down into visually coherent areas
Can identify meaningful areas and objects
Computationally intensive
Unreliable

18. Color Produce a color signature for region/whole image
Typically done using color correllograms or color histograms

19. Color Features Color Histograms Color Layout Other Color Features
Color Space Selection
Color Space Quantization
Color Histogram Calculation
Feature Indexing
Similarity Measures
Color Layout
Histograms based on spatial distribution of single color
Histograms based on spatial distribution of color pair
Histograms based on spatial distribution of color triple
Other Color Features
Color Moments
Color Sets
Color Features
Color Histograms
Color Space Selection
Color Space Quantization
Color Histogram Calculation
Feature Indexing
Similarity Measures
Color Layout
Histograms based on spatial distribution of single color
Histograms based on spatial distribution of color pair
Histograms based on spatial distribution of color triple
Other Color Features
Color Moments
Color Sets

20. Color Space Selection Which Color Space? HSV?
RGB, CMY, YCrCb, CIE, YIQ, HLS, …
HSV?
Designed to be similar to human perception

21. HSV Color Space H (Hue) S (Saturation) V (Value) How to Use This?
Dominant color (spectral)
S (Saturation)
Amount of white
V (Value)
Brightness
How to Use This?

22. Content Based Image Retrieval
CBIR
utilizes unique features (shape, color, texture) of images
Users prefer
To retrieve relevant image by semantic categories
But, CBIR can not capture high-level semantics in user’s mind
Despite of advances in multimedia and Web technology,
image understanding by machines for search engines on the Web
is still a difficult problem.
CBIR systems use the visual content of images such as color histogram,
color layout, texture and shape features to represent and index images
as points in a feature space.
However, the performance of retrieval system is usually very low
because the low level feature representation cannot capture
the user's high level concept in a query image.

23. Relevance Feedback Relevance Feedback Relevance Feedback Phase
Learns the associations between high-level semantics and low-level features
Relevance Feedback Phase
User identifies relevant images within the returned set
System utilizes user feedback in the next round
To modify the query (to retrieve better results)
This process repeats  until user is satisfied
Relevance feedback is an approach that learns association
between high-level semantics and low-level features.
In RF process, user identifies relevant images
within the retrieved image set and the system utilizes user feedback
in the next round to modify the query.
This process is repeated until user is satisfied.

24. 1st iteration 2nd iteration User Feedback User Feedback Display
Estimation &
Display selection
Feedback to system
2nd iteration
Display
This animation illustrate how a typical relevance feedback approach works. At first, the retrieval system presents to user a set of images. The user evaluates which one is close to what he is looking for and marked down which one is relevant and give feedback to system. In turn, the system learns from user’s feedback and select the next set of images to display, you can see that with relevance feedback, the second iteration shows more relevant result assuming the user is looking for image of tree. This process continues and user can get the result refine to what he really wants.
User Feedback

25. Now, We have many features (too many?)
How to express visual “similarity” with these features?

26. Visual Similarity ? “Similarity” is Subjective and Context-dependent.
“Similarity” is High-level Concept.
Cars, Flowers, …
But, our features are Low-level features.
Semantic Gap!

27. Which features are most important?
Not all features are always important.
“Similarity” measure is always changing
The system has to weight features on the fly.
How ?

28. Q & A