1. Spatial and Temporal Data Mining
V. Megalooikonomou
Generic Multimedia Indexing
(slides are based on notes by C. Faloutsos)

2. General Overview Multimedia Indexing Spatial Access Methods (SAMs)
k-d trees
Point Quadtrees
MX-Quadtree
z-ordering
R-trees
Generic Multimedia Indexing

3. Mutlimedia Indexing – Detailed outline
Generic Multimedia Indexing
problem dfn
Distance function
Similarity queries – Types
Requirements (ideal method)
Basic idea, Lower-bounding
Gemini approach
Applications
1-D Time sequences
2-D Color images

4. Generic Multimedia Indexing - problem
Given a database of multimedia objects
Design fast search algorithms that locate objects that match a query object, exactly or approximately
Objects:
1-d time sequences
Digitized voice or music
2-d color images
2-d or 3-d gray scale medical images
Video clips
E.g.: “Find companies whose stock prices move similarly”

5. Mutlimedia Indexing – Detailed outline
Generic Multimedia Indexing
problem dfn
Distance function
Similarity queries – Types
Requirements (ideal method)
Basic idea, Lower-bounding
Gemini approach
Applications
1-D Time sequences
2-D Color images

6. Generic Multimedia Indexing- problem
1st step: provide a measure for the distance between two objects
Distance function D():
Given two objects OA, OB the distance (=dis-similarity) of the two objects is denoted by
D(OA, OB)
E.g., Euclidean distance (sum of squared differences) of two equal-length time series

7. Mutlimedia Indexing – Detailed outline
Generic Multimedia Indexing
problem dfn
Distance function
Similarity queries
Requirements (ideal method)
Basic idea, Lower-bounding
Gemini approach
Applications
1-D Time sequences
2-D Color images

8. Types of Similarity QueriesSn
avg 1
365
day
F(S1)
F(Sn)
std
Similarity queries are classified into:
Whole match queries:
Given a collection of N objects O1,…, ON and a query object Q find data objects that are within distance  from Q
Sub-pattern Match:
Given a collection of N objects O1,…, ON and a query (sub-) object Q and a tolerance  identify the parts of the data objects that match the query Q

9. Types of Similarity Queries
std S1
F(S1) 1
365
day
F(Sn) Sn
avg
day 1
365
Similarity queries are classified into:
Whole match queries:
Given a collection of N objects O1,…, ON and a query object Q find data objects that are within distance  from Q
Sub-pattern Match:
Given a collection of N objects O1,…, ON and a query (sub-) object Q and a tolerance  identify the parts of the data objects that match the query Q

10. Types of Similarity Queries
std S1
F(S1) 1
365
day
F(Sn) Sn
avg
day 1
365
Similarity queries are classified into:
Whole match queries:
Given a collection of N objects O1,…, ON and a query object Q find data objects that are within distance  from Q
Sub-pattern Match:
Given a collection of N objects O1,…, ON and a query (sub-) object Q and a tolerance  identify the parts of the data objects that match the query Q

11. Types of Similarity Queries
Similarity queries are classified into:
Whole match queries:
Given a collection of N objects O1,…, ON and a query object Q find data objects that are within distance  from Q
Sub-pattern Match:
Given a collection of N objects O1,…, ON and a query (sub-) object Q and a tolerance  identify the parts of the data objects that match the query Q

12. Types of Similarity QueriesSn
avg 1
365
day
F(S1)
F(Sn)
std
Additional types of queries:
K- Nearest Neighbor queries:
Given a collection of N objects O1,…, ON and a query object Q find the K most similar data objects to Q
All pairs queries (or ‘spatial joins’):
Given a collection of N objects O1,…, ON find all objects that are within distance  from each other

13. Types of Similarity QueriesSn
avg 1
365
day
F(S1)
F(Sn)
std
Additional types of queries:
K- Nearest Neighbor queries:
Given a collection of N objects O1,…, ON and a query object Q find the K most similar data objects to Q
All pairs queries (or ‘spatial joins’):
Given a collection of N objects O1,…, ON find all objects that are within distance  from each other

14. Mutlimedia Indexing – Detailed outline
Generic Multimedia Indexing
problem dfn
Distance function
Similarity queries – Types
Requirements (ideal method)
Basic idea, Lower-bounding
Gemini approach
Applications
1-D Time sequences
2-D Color images

15. Idea method – requirements
Fast: sequential scanning and distance calculation with each and every object too slow for large databases
“Correct”: No false dismissals. False alarms are acceptable. Why?
Small space overhead
Dynamic: easy to insert, delete, and update objects

16. Approach Outline
Use k feature extraction functions to map objects into k-dimensional space (applying a mapping F () )
Use highly fine-tuned database SAMs (Spatial Access Methods) like R-trees to accelerate the search (by pruning out large portions of the database that are not promising)…

17. Mutlimedia Indexing – Detailed outline
Generic Multimedia Indexing
problem dfn
Distance function
Similarity queries – Types
Requirements (ideal method)
Basic idea, Lower-bounding
Gemini approach
Applications
1-D Time sequences
2-D Color images

18. Basic idea Focus on ‘whole match’ queries Sequential scanning?
Given a collection of N objects O1,…, ON, a distance/dis-similarity function D(Oi, Oj), and a query object Q find data objects that are within distance  from Q
Sequential scanning?

19. Basic idea Focus on ‘whole match’ queries Sequential scanning?
Given a collection of N objects O1,…, ON, a distance/dis-similarity function D(Oi, Oj), and a query object Q find data objects that are within distance  from Q
Sequential scanning?
May be too slow.. Why?

20. Basic idea Focus on ‘whole match’ queries Sequential scanning?
Given a collection of N objects O1,…, ON, a distance/dis-similarity function D(Oi, Oj), and a query object Q find data objects that are within distance  from Q
Sequential scanning?
May be too slow.. for the following reasons:
Distance computation is expensive (e.g., editing distance in DNA strings)
The Database size N may be huge
Faster alternative?

21. Basic idea Faster alternative: Example:
Step 1: a ‘quick and dirty’ test to discard quickly the vast majority of non-qualifying objects
Step 2: use of SAMs to achieve faster than sequential searching
Example:
Database of yearly stock price movements
Euclidean distance function
Characterize with a single number (‘feature’)
Or use two or more features

22. Basic idea - illustration
day 1
365 S1 Sn
F(S1)
F(Sn)
Feature1
Feature2
A query with tolerance  becomes a sphere with radius 

23. Basic idea – caution!
The mapping F() from objects to k-d points should not distort the distances
D(): distance of two objects
Df(): distance of their corresponding feature vectors
Ideally, perfect preservation of distances
In practice, a guarantee of no false dismissals
How?

24. Basic idea – caution!
The mapping F() from objects to k-d points should not distort the distances
D(): distance of two objects
Df(): distance of the corresponding feature vectors
Ideally, perfect preservation of distances
In practice, a guarantee of no false dismissals
How? If the distance in f-space matches or underestimates the distance between two objects in the original space

25. Basic idea – Lower bounding
Let O1, O2 be two objects with distance function D() and F(O1), F(O2), be their feature vectors with distance function Df(), then:
To guarantee no false dismissals for whole match queries, the feature extraction function F() should satisfy:
Df(F(O1), F(O2))  D(O1, O2)
for every pair of objects O1, O2

26. Lower bounding - Proof
Let Q be the query object and O be the qualifying object and  be the tolerance.
Prove: If object O qualifies it will be retrieved by a range query in the f-space
Or, D(Q, O)    Df(F(Q), F(O))  
However, Df(F(Q), F(O))  D(Q, O)   
What about ‘all-pairs’?
What about ‘nearest-neighbor’ queries?

27. Lower bounding - Proof
Let Q be the query object and O be the qualifying object and  be the tolerance.
Prove: If object O qualifies it will be retrieved by a range query in the f-space
Or, D(Q, O)    Df(F(Q), F(O))  
However, Df(F(Q), F(O))  D(Q, O)   
What about ‘all-pairs’? (‘spatial join’ on f-space)
What about ‘nearest-neighbor’ queries?

28. Lower bounding - Proof
Let Q be the query object and O be the qualifying object and  be the tolerance.
Prove: If object O qualifies it will be retrieved by a range query in the f-space
Or, D(Q, O)    Df(F(Q), F(O))  
However, Df(F(Q), F(O))  D(Q, O)   
What about ‘all-pairs’? (‘spatial join’ on f-space)
What about ‘nearest-neighbor’ queries? ??

29. Mutlimedia Indexing – Detailed outline
Generic Multimedia Indexing
problem dfn
Distance function
Similarity queries – Types
Requirements (ideal method)
Basic idea, Lower-bounding
Gemini approach
Applications
1-D Time sequences
2-D Color images

30. GEneric Multimedia object INdexIng
GEMINI approach:
Determine distance function D()
Find one or more numerical feature-extraction functions (to provide a ‘quick and dirty’ test)
Prove that Df() lower-bounds D() to guarantee no false dismissals
Use a SAM (e.g., R-tree) to store and retrieve k-d feature vectors
!!! The methodology focuses on the speed of search only; not on the quality of the results which relies on the distance function

31. Generic Multimedia Object Indexing
Applications:
1-d time sequences
2-d color images
Problems to solve:
How to apply the lower-bounding lemma
‘Curse of Dimensionality’ (time sequences)
‘Cross-talk’ of features (color images)

32. Mutlimedia Indexing – Detailed outline
Generic Multimedia Indexing
problem dfn
Distance function
Similarity queries – Types
Requirements (ideal method)
Basic idea, Lower-bounding
Gemini approach
Applications
1-D Time sequences
2-D Color images

33. 1-D Time Sequences Distance function: Euclidean distance
Find features that:
Preserve/lower-bound the distance
Carry as much information as possible(reduce false alarms)
If we are allowed to use only one feature what would this be?

34. 1-D Time Sequences Distance function: Euclidean distance
Find features that:
Preserve/lower-bound the distance
Carry as much information as possible(reduce false alarms)
If we are allowed to use only one feature what would this be? The average.
… extending it…

35. 1-D Time Sequences Distance function: Euclidean distance
Find features that:
Preserve/lower-bound the distance
Carry as much information as possible(reduce false alarms)
If we are allowed to use only one feature what would this be? The average.
… extending it…
The average of 1st half, of the 2nd half, of the 1st quarter, etc.
Coefficients of the Fourier transform (DFT), wavelet transform, etc.

36. 1-D Time Sequences
Show that the distance in feature space lower-bounds the actual distance
What about DFT?

37. 1-D Time Sequences
Show that the distance in feature space lower-bounds the actual distance
What about DFT?
Parseval’s Theorem: DFT preserves the energy of the signal as well as the distances between two signals.
D(x,y) = D(X,Y)
where X and Y are the Fourier transforms of x and y
If we keep the first k  n coefficients of DFT we lower-bound the actual distance

38. 1-D Time Sequences
Response time improves as the transform concentrates more the energy of the signal
DFT concentrates the energy for a large class of signals, the colored noises
Colored noises: skewed energy spectrum that drops as O(f -b)
Energy spectrum or power spectrum of a signal is the square of the amplitude |Xf| as a function of the frequency f
b = 2: random walks or brown noise (very predictable)
b  2: black noises
b = 1: pink noise
b = 0: white noise (completely unpredictable)
Colored noises even in images (photographs)

39. Mutlimedia Indexing – Detailed outline
Generic Multimedia Indexing
problem dfn
Distance function
Similarity queries – Types
Requirements (ideal method)
Basic idea, Lower-bounding
Gemini approach
Applications
1-D Time sequences
2-D Color images

40. 2-D color images
Image features for Content Based Image Retrieval (CBIR):
Low Level:
Color – color histograms
Texture – directionality, granularity, contrast
Shape – turning angle, moments of inertia, pattern spectrum
Position – 2D strings method
…etc
Object Level:
Regions

41. 2-D color images – Color histograms
Each color image – a 2-d array of pixels
Each pixel – 3 color components (R,G,B)
h colors – each color denoting a point in 3-d color space (as high as 224 colors)
For each image compute the h-element color histogram – each component is the percentage of pixels that are most similar to that color
The histogram of image I is defined as:
For a color Ci , Hci(I) represents the number of pixels of color Ci in image I
OR:
For any pixel in image I, Hci(I) represents the possibility of that pixel having color Ci.

42. 2-D color images – Color histograms
Usually cluster similar colors together and choose one representative color for each ‘color bin’
Most commercial CBIR systems include color histogram as one of the features (e.g., QBIC of IBM)
No space information

43. Color histograms - distance
One method to measure the distance between two histograms x and y is:
where the color-to-color similarity matrix A has entries aij that describe the similarity between color i and color j

44. Color histograms – lower bounding
Two obstacles for using color-histograms as feature vectors in GEMINI:
‘Dimensionality curse’ (h is large 64, 128)
Distance function is quadratic
It involves all cross terms (‘cross-talk’ among features)
- expensive to compute
- precludes the use of SAMs
bright red
pink
orange x q
e.g.,64 colors

45. Color histograms – lower bounding
1st step: define the distance function between two color images D()=dh()
2nd step: find numerical features (one or more) whose Euclidean distance lower-bounds dh()
If we allowed to use one numerical feature to describe the color image what should it be?
Avg. amount for each color component (R,G,B)
Where … , similarly for G and B
Where P is the number of pixels in the image, R(p) is the red component (intensity) of the p-th pixel

46. Color histograms – lower bounding
Given the average color vectors and of two images we define davg() as the Euclidean distance between the 3-d average color vectors
3rd step: to prove that the feature distance davg() lower-bounds the actual distance dh()
Main idea of approach:
First a filtering using the average (R,G,B) color,
then a more accurate matching using the full h-element histogram

47. Color auto-correlogram
pick any pixel p1 of color Ci in the image I
at distance k away from p1 pick another pixel p2
what is the probability that p2 is also of color Ci ?
Red ? k P2 P1
Image: I

48. Color auto-correlogram
The auto-correlogram of image I for color Ci , distance k:
Integrate both color information and space information.

49. Color auto-correlogram

50. Implementations Pixel Distance Measures
Use D8 distance (also called chessboard distance):
Choose distance k=1,3,5,7
Computation complexity:
Histogram:
Correlogram:

51. Implementations Features Distance Measures:
D( f(I1) - f(I2) ) is small  I1 and I2 are similar.
Example: f(a)=1000, f(a’)=1050; f(b)=100, f(b’)=150
For histogram:
For correlogram:

52. Color Histogram vs Correlogram
If there is no difference between the query and the target images, both methods have good performance.
Correlogram method
Query Image
(512 colors)
1st
2nd
3rd
4th
5th
Histogram method
1st
2nd
3rd
4th
5th

53. Color Histogram vs Correlogram
The correlogram method is more stable to color change than the histogram method.
Query
Correlogram method: 1st
Histogram method: 48th
Target

54. Color Histogram vs Correlogram
The correlogram method is more stable to large appearance change than the histogram method
Query
Correlogram method: 1st
Histogram method: 31th
Target

55. Color Histogram vs Correlogram
The correlogram method is more stable to contrast & brightness change than the histogram method.
Query 1
Query 2
Query 3
Query 4
C: 178th
H: 230th
C: 1st
H: 1st
C: 1st
H: 3rd
C: 5th
H: 18th
Target

56. Color Histogram vs Correlogram
The color correlogram describes the global distribution of local spatial correlations of colors.
It’s easy to compute
It’s more stable than the color histogram method

57. Mutlimedia Indexing – Conclusions
GEMINI is a popular method
Whole matching problem
Should pay attention to:
Distance functions
Feature Extraction functions
Lower Bounding
Particular application
Sub-pattern matching?