1. Alexander Gelbukh www.Gelbukh.com
Special Topics in Computer Science Advanced Topics in Information Retrieval Lecture 6 (book chapter 12): Multimedia IR: Indexing and Searching
Alexander Gelbukh

2. Previous Chapter: Conclusions
Basically, images are handled as text described them
Namely, feature vectors (or feature hierarchies)
Context can be used when available to determine features
Also, queries by example are common
From the point of view of DBMS, integration with IR and multimedia-specific techniques is needed
Object-oriented technology is adequate

3. Previous Chapter: Research topics
How similarity function can be defined?
What features of images (video, sound) there are?
How to better specify the importance of individual features? (Give me similar houses: similar = size? color? strructure? Architectural style?)
How to determine the objects in an image?
Integration with DBMSs and SQL for fast access and rich semantics
Integration with XML
Ranking: by similarity, taking into account history, profile

4. The problem Data examples: Query examples:
2D/3D color/grayscale images: e.g., brain scans, scientific databases of vector fields
(2D) video,
(1D) voice/music; (1D) time series: e.g., financial/marketing time series; DNA/genomic databases
Query examples:
find photographs with the same color distribution as this
find companies whose stock prices move as this one
find brain scans with a texture of a tumor
Applications: search; data mining

5. Solution
Reduce the problem to search for multi-dimensional points (feature vectors, but vector space is not used)
Define a distance measure
for time series: e.g., Euclidean distance between vectors
for images: e.g., color distribution (Euclidean distance); another approach: mathematical morphology
Other features as vectors
For search within distance, the vectors are organized in R-trees
Clustering plays important role

6. Types of queries All within given distance Nearest-neighbor
Find all images that are within 0.05 distance from this one
Nearest-neighbor
Find 5 stocks most similar to IBM
All pairs within given distance
Further: clustering
Whole object vs. sub-pattern match
Find parts of image that are...
E.g., in 512  512 brain scans, find pieces similar to the given 16  16 typical X-ray of a tumor
Like passage retrieval for text documents

7. Neighbor and pairs types of queries
The objects are organized in R-trees
For neighbor queries: branch-and-bound algorithm
For pairs: recently discovered algorithms
These types of queries are not discussed here

8. Desiderata for a method
Fast
No sequential search with all objects
Correct
100% recall
Precision is less important, though kept low. False alarms are easy to discard manually
Little space overhead
Dynamic
easy to insert, delete, update

9. Types of methods Linear quadtrees grid-files
Complexity = hypersurface of the query region
Grows exponentially with dimensionality
grid-files
Complexity grows exponentially with dimensionality
R-trees methods, such as R*-trees
Most used due to lower complexity

10. R-tree
Objects and parts of images represented as Minimal Bounding Rectangle (MBR)
Can overlap for different objects
Larger objects contain smaller objects
MBRs are nested
MBRs are arranged into a tree
In storage, an index of disk blocks is maintained
Disk blocks are fetched at once at hardware level
For better insertion/deletion, tight MBRs are needed
Good clustering is needed

11. File structure of R-tree
Corresponds to disk blocks
Fanout = 3: number of parts to group

12. R-tree
R-tree

13. Search in R-tree Range queries:
find objects within distance  from query object
= Find MBRs that intersect with query’s MBR
Determine MBR of the query
Descend the tree
Discarding all MBRs that do not intersect with the query’s MBR
Many variations of R-tree method have been proposed

14. Indexing Only consider here whole match queries Problems:
Given collection of objects and distance function
Find objects within given distance  from given object Q
Problems:
Slow comparison of two objects
Huge database
GEMINI approach
GEneric Multimedia object INdexIng
Attempts to solve both problems

15. GEMINI indexing Quick-and-dirty test to quickly discard bad objects
Uses clusters to avoid sequential search
Quick test
Single-valued feature, e.g., average for series. Averages differ much  objects differ much
Not vice-versa. False alarms are OK
Several features, but fewer than all data. E.g., deviation for series

16. Algorithm Map the actual objects into f-dimensional feature space
Use clusters (e.g., R-trees) to search
Retrieve objects, compute the actual distances, and discard false alarms

18. Feature selection Features should reflect distances
Allow no misses (100% recall)
features should make things look closer
Lower Bound lemma:
If distance in feature space  actual distance
then 100% recall
(we speak about whole-match queries)
Holds for distance search, nearest-neighbor, pair search

19. Algorithm (more detail)
Determine distance
Choose features
Prove that distance in feature space  for actual objects
Use quick method (R-tree) to search in feature space
For found objects, compute the actual distances (this can be expensive)
Discard false alarms
objects with greater actual distances, even if in feature space the distance is OK
Example: similar averages, but different series

20. Discussion The method does NOT improve quality
Provides SAME quality as sequential search, but faster
Distance definition requires domain/application expert
How much do the two images differ?
What is important/unimportant for the specific application?
Feature selection requires a good knowledge engineer
Choose the most characteristic feature: discriminative
If needed, choose the second best, etc.
Good features should be orthogonal: combination adds info

21. Example: Time series
In yearly stock movements, find ones similar to IBM
Distance: Euclidean (365-D vectors); others exist
Features:
First feature is average.
If needed, Discrete Fourier Transform (DFT) coefficients
Or, Discrete Cosine Transform, waivelet Transform, etc.
Lower-bound lemma:
Parseval theorem: DFT preserves distances (DCT, WT too)
First several coefficients give  distance
Transforms “concentrate energy” in the first coefficients
Thus, the more realistic prediction of distance

22. Time series: Applications
Such feature selection is effective for many skewed spectrum distributions
Colored noises: the energy decreases as F–b
b = 0: white spectrum: unpredictable. Method useless.
b = 1: pink noise: works of art
b = 2: brown noise: stock movements
b > 2: black noise: river levels, rainfall patterns
The greater b the better the first coefficients of the transform predict the actual distance
Some other n-D signals show similar properties
JPEG compression ignores higher coefficients

23. Time series: Performance
Fewer features  more false alarms  time lost
More features  more complex computation
Optimal number of features proves to be about 1..3
for skewed enough distributions
JPEG compression shows that photographs have it

24. Time series: Sub-pattern search
Use sliding window
Encode each window with few features

25. Example: Color images
Give me images with a texture of tumor like this one
Give me images with blue at top and red at bottom
Handles color, texture, shape, position, dominant edges

26. Color images: Color representation
Compute color histogram
Distance: use color similarity matrix
Very expensive computationally: cross-talk between features (compare all to all features)

28. Color images: Feature mapping
The GEMINI question again: What single feature is the most representative?
Take average R, G, B
Lower-bound?
Yes: Quadratic Distance Bounding theorem

29. Automatic feature selection
Features can be selected automatically
In texts: Latent semantic indexing (LSI)
Many methods
Principle components analysis (= LSI), ...
In fact, they can reduce features, but not define them
Of colors, one can select characteristic combinations
But not classify into faces and flowers
So description of the objects is still on human researchers

30. Research topics Object detection (pattern and image recognition)
Automatic feature selection
Spatial indexing data structures (more than 1D)
New types of data.
What features to select? How to determine them?
Mixed-type data (e.g., webpages, or images with sound and description)
What clustering/IR methods are better suited for what features? (What features for what methods?)
Similar methods in data mining, ...

31. Conclusions How to accelerate search? Same results as sequential
Ideas:
Quick-and-dirty rejection of bad objects, 100% recall
Fast data structure for search (based on clustering)
Careful check of all found candidates
Solution: mapping into fewer-D feature space
Condition: lower-bounding of the distance
Assumption: skewed spectrum distribution
Few coefficients concentrate energy, rest are less important

32. Thank you!
Till Tuesday 11, 6 pm