Computer Vision, Part 1

Topics for Vision Lectures 1.Content-Based Image Retrieval (CBIR) 2.Object recognition and scene understanding

Content-Based Image Retrieval

Example: Google Search by ImageGoogle Search by Image

Query Image Extract Features (Primitives) Image Database Features Database Similarity Measure Matched Results Relevance Feedback Algorithm From Basic technique Each image in database is represented by a feature vector: x 1, x 2,...x N, where x i = (x i 1, x i 2, …, x i m ) Query is represented in terms of same features: Q =(Q 1, Q 2, …, Q m ) Goal: Find stored image with vector x i most similar to query vector Q

Distance measure: Possible distance measures d(Q, x i ): – Inner (dot) product – Histogram distance (for histogram features) – Graph matching (for shape features).

Some issues in designing a CBIR system Query format, ease of querying Speed Crawling, preprocessing Interactivity, user relevance feedback Visual features which to use? How to combine? Curse of dimensionality Indexing Evaluation of performance

Types of Features Typically Used Intensities Color Texture Shape Layout

Intensity histograms artSpy/intensity.html

Color Features Hue, saturation, value

Color Histograms (8 colors)

Color auto-correlogram Pick any pixel p 1 of color C i in the image I. At distance k away from p 1 pick another pixel p 2. What is the probability that p 2 is also of color C i ? p1p1 p2p2 k Red ? Image: I From:

The auto-correlogram of image I for color C i, distance k: Integrates both color information and space information. From:

Two images with their autocorrelograms. Note that the change in spatial layout would be ignored by color histograms, but causes a significant difference in the autocorrelograms. From

Color histogram rank: 411; Auto-correlogram rank: 1 Color histogram rank: 310; Auto-correlogram rank: 5 Color histogram rank: 367; Auto-correlogram rank: 1 From: u/rdz/Papers/ecdl2/spati al.htm

Texture representations Gray-level co-occurrence Entropy Contrast Fourier and wavelet transforms Gabor filters

Texture Representations Each image has the same intensity distribution, but different textures Can use auto-correlogram based on intensity (gray-level co-occurrence)

Texture from entropy Images filtered by entropy: Each output pixel contains entropy value of 9x9 neighborhood around original pixel From:

Texture from fractal dimension From: cv07.pdf

Texture from Contrast Example : ess.html ess.html

Texture from Wavelets artSpy/dwt.html

Shape representations Some of these need segmentation (another whole story!) Area, eccentricity, major axis orientation Skeletons, shock graphs Fourier transformation of boundary Histograms of edge orientations

From:

From: Histogram of edge orientations

Visual abilities largely missing from current CBIR systems Object recognition Perceptual organization Similarity between semantic concepts

Image 1Image 2 Examples of semantic similarity

Image 1Image 2 Examples of semantic similarity

Image 1 Image 2 Examples of semantic similarity

In general, current systems have not yet had significant impact on society due to an inability to bridge the semantic gap between computers and humans.

Image Understanding and Analogy- Making

Bongard problems as an idealized domain for exploring the semantic gap