Content-Based Image Retrieval

What is Content-based Image Retrieval (CBIR)? Image Search Systems that search for images by image content <-> Keyword-based Image/Video Retrieval (ex. Google Image Search, YouTube)

How does CBIR work ? Extract Features from Images Let the user do Query Query by Sketch Query by Keywords Query by Example Refine the result by Relevance Feedback Give feedback to the previous result

Query by Example CBIR “Get similar images” Query Sample Results Pick example images, then ask the system to retrieve “similar” images. Results CBIR “Get similar images” Query Sample What does “similar” mean?

Relevance Feedback User gives a feedback to the query results System recalculates feature weights 2nd Result Feedback 1st Result Query Feedback Initial sample

Two Classes of CBIR Narrow vs. Broad Domain Medical Imagery Retrieval Finger Print Retrieval Satellite Imagery Retrieval Broad Photo Collections Internet

The Architecture of a typical CBIR System Server Image Manager Image Database Retrieval Module Multi-dimensional Indexing Feature Extraction Feature Database Client User Interface This figure shows the main components of a typical CBIR system. For each image in an image database, visual features such as color, texture and shape are first extracted and stored. For the purpose of efficiency, multi-dimensional indexing techniques are often used to index the feature vectors. When a visual query submitted by the user through the query interface and processing module, the corresponding feature vector of the query will be generated and compared with the feature vectors of the images in the database by some similarity measurements, and similar images will be returned to the user. The user can refine the query by providing feedback information.

The Retrieval Process of a typical CBIR System Feature Database Feature Extraction for the Query Image Feature Extraction 1 Extraction 2 Extraction n (2 3 5 0 7 9) Feature Vector 1 (1 1 0 0 1 0) Feature Vector 2 (33.5 6.7 28.6 11.8 5.5) Feature Vector n Image Database Similarity Comparison Image Manager Sorting Query Image (Image ID, similarity) This figure further shows the retrieval process of a typical CBIR system, using a Rose image as the query. Different feature vectors of the query image will be generated by different feature extraction methods, and be compared with feature vectors of the images in the database, using some similarity comparison metrics. The images will be sorted based on the calculated similarities, and the top hits will be returned to the user. Images Interface Manager Results

Basic Components of CBIR Feature Extraction Data indexing Query and feedback processing

How Images are represented

Image Features Representing the Images Segmentation Low Level Features Color Texture Shape

Image Features Information about color or texture or shape which are extracted from an image are known as image features Also a low-level features Red, sandy As opposed to high level features or concepts Beaches, mountains, happy

Global features Averages across whole image Tends to loose distinction between foreground and background Poorly reflects human understanding of images Computationally simple A number of successful systems have been built using global image features

Local Features Segment images into parts Two sorts: Tile Based Region based

Regioning and Tiling Schemes Tiles Regions

Tiling Break image down into simple geometric shapes Similar Problems to Global Plus dangers of breaking up significant objects Computational Simple Some Schemes seem to work well in practice

Regioning Break Image down into visually coherent areas Can identify meaningful areas and objects Computationally intensive Unreliable

Color Produce a color signature for region/whole image Typically done using color correllograms or color histograms

Color Features Color Histograms Color Layout Other Color Features Color Space Selection Color Space Quantization Color Histogram Calculation Feature Indexing Similarity Measures Color Layout Histograms based on spatial distribution of single color Histograms based on spatial distribution of color pair Histograms based on spatial distribution of color triple Other Color Features Color Moments Color Sets Color Features Color Histograms Color Space Selection Color Space Quantization Color Histogram Calculation Feature Indexing Similarity Measures Color Layout Histograms based on spatial distribution of single color Histograms based on spatial distribution of color pair Histograms based on spatial distribution of color triple Other Color Features Color Moments Color Sets

Color Space Selection Which Color Space? HSV? RGB, CMY, YCrCb, CIE, YIQ, HLS, … HSV? Designed to be similar to human perception

HSV Color Space H (Hue) S (Saturation) V (Value) How to Use This? Dominant color (spectral) S (Saturation) Amount of white V (Value) Brightness How to Use This?

Content Based Image Retrieval CBIR utilizes unique features (shape, color, texture) of images Users prefer To retrieve relevant image by semantic categories But, CBIR can not capture high-level semantics in user’s mind Despite of advances in multimedia and Web technology, image understanding by machines for search engines on the Web is still a difficult problem. CBIR systems use the visual content of images such as color histogram, color layout, texture and shape features to represent and index images as points in a feature space. However, the performance of retrieval system is usually very low because the low level feature representation cannot capture the user's high level concept in a query image.

Relevance Feedback Relevance Feedback Relevance Feedback Phase Learns the associations between high-level semantics and low-level features Relevance Feedback Phase User identifies relevant images within the returned set System utilizes user feedback in the next round To modify the query (to retrieve better results) This process repeats  until user is satisfied Relevance feedback is an approach that learns association between high-level semantics and low-level features. In RF process, user identifies relevant images within the retrieved image set and the system utilizes user feedback in the next round to modify the query. This process is repeated until user is satisfied.

1st iteration 2nd iteration User Feedback User Feedback Display Estimation & Display selection Feedback to system 2nd iteration Display This animation illustrate how a typical relevance feedback approach works. At first, the retrieval system presents to user a set of images. The user evaluates which one is close to what he is looking for and marked down which one is relevant and give feedback to system. In turn, the system learns from user’s feedback and select the next set of images to display, you can see that with relevance feedback, the second iteration shows more relevant result assuming the user is looking for image of tree. This process continues and user can get the result refine to what he really wants. User Feedback

Now, We have many features (too many?) How to express visual “similarity” with these features?

Visual Similarity ? “Similarity” is Subjective and Context-dependent. “Similarity” is High-level Concept. Cars, Flowers, … But, our features are Low-level features. Semantic Gap!

Which features are most important? Not all features are always important. “Similarity” measure is always changing The system has to weight features on the fly. How ?

Q & A