1. Multimedia and Time-series Data
Today we need to query and analyze vast amounts of multimedia data (i.e., images, sound tracks, video tracks).
Joint Research from Database Management, Computer Vision and Signal Processing aims to solve problems related to multimedia data management.
Today we will see an overview of some techniques used for this problems.
We will also briefly discuss how time-series data are managed in database systems.
Dr. N. Mamoulis
Advanced Database Technologies

2. Advanced Database Technologies
Multimedia Data
There are four major types of multimedia data: images, video tracks, sound tracks, and text.
From the above, the easiest type to manage is text, since we can order, index, and search text using string management techniques.
Management of simple sounds is also relatively easy, since they can be represented by signal sequences over channels.
We will describe techniques for image retrieval. These can are extended and applied also for video retrieval.
Dr. N. Mamoulis
Advanced Database Technologies

3. Content-based Image Retrieval
Images were traditionally managed by first annotating their contents and then using text-retrieval techniques to index them.
However, with the increase of information in digital image format some drawbacks of this technique were revealed:
Manual annotation requires vast amount of labor
Different people may perceive differently the contents of an image; thus no objective keywords for search are defined
A new research field was born in the 90’s: Content-based Image Retrieval aims at indexing and retrieving images based on their visual contents.
Dr. N. Mamoulis
Advanced Database Technologies

4. Advanced Database Technologies
Feature Extraction
The basis of Content-based Image Retrieval is to extract and index some visual features of the images.
There are general features (e.g., color, texture, shape, etc.) and domain-specific features (e.g., objects contained in the image).
Domain-specific feature extraction can vary with the application domain and is based on pattern recognition
On the other hand, general features can be used independently from the image domain.
Next, we will see how the general features of images are indexed and queried in a database.
Dr. N. Mamoulis
Advanced Database Technologies

5. Advanced Database Technologies
Color Features
The color is the dominant general visual feature of an image. Humans primarily distinguish images based on the color proportions in them.
There are three types of receptors in our eyes. They receive red, green and blue colors. This is the motivation of the RGB model. Adding these three primary colors together produces the rest of known colors. An image can be represented by a two-dimensional array of color values (pixels).
pixel representation
RGB color space
Dr. N. Mamoulis
Advanced Database Technologies

6. Color Features (cont’d)
To represent the color of an image compactly, a color histogram is used. Colors are partitioned to k groups according to their similarity and the percentage of each group in the image is measured.
Images are transformed to k-dimensional points and a distance metric (e.g., Euclidean distance) is used to measure the similarity between them.
k-dimensional space
k-bins
Dr. N. Mamoulis
Advanced Database Technologies

7. Color Features (cont’d)
A color space which facilitates quantization of color distribution in an image is HSV
Hue: what the color is. This tells where along the spectrum your color lies.
Saturation: how pure the color is (high values -> stronger color small values -> faded color).
Value: how bright the color is.
HSV
RGB
Dr. N. Mamoulis
Advanced Database Technologies

8. Advanced Database Technologies
Texture and Shape
Texture refers to the visual patterns that are homogeneous and do not result from the presence of one color or intensity (e.g., clouds, bricks, hair, fabric, etc.).
Several representations of texture have been considered, among them a co-occurrence matrix for spatial color dependencies, visual texture properties (coarseness, regularity, etc.), and wavelet transforms of texture images.
The boundaries of shapes are also indexed using transforms (e.g., Fourier Descriptors, Moment Invariants). We will see later various transforms used for approximation and indexing.
Dr. N. Mamoulis
Advanced Database Technologies

9. Advanced Database Technologies
Color Layout
Querying by global color may give too many false results in large image collections.
100% similar
by global color
Images are usually divided into blocks and color features are extracted from each of the blocks. Similarity is then aggregated from all blocks.
More sophisticated techniques for image partitioning have been proposed (e.g., space partitioning using quadtrees)
Dr. N. Mamoulis
Advanced Database Technologies

10. Advanced Database Technologies
Time-series Data
A time-series is a sequential collection of events over time.
Time series data are found in various fields, e.g., stock market values, sensor indications, cardiograms.
There are several search problems over time-series data, e.g.:
find (approximate/exact) occurrences of a query subsequence q in a long sequence T.
find the most similar sequence to a query sequence q
classify/cluster sequences based on their similarity
find frequent event patterns in long sequences
...
Dr. N. Mamoulis
Advanced Database Technologies

11. Common Search Problems in Multimedia Data and Time-series Data
Comparing long sequences, or long feature vectors is time consuming.
Indexing them as points in a high dimensional space does not work due to the ‘curse of dimensionality’; the efficiency of spatial access methods degrades fast with dimensionality and the entropy of high dimensional space makes NN search less meaningful (more in presentation 1...)
The problem can be alleviated by:
data approximation and dimensionality reduction
data compression and linear scan
Dr. N. Mamoulis
Advanced Database Technologies

12. Using Transformations to Reduce Dimensionality
In many cases the embedded dimensionality of a search problem is much lower than the actual dimensionality
Some methods apply transformations on the data and approximate them with low-dimensional vectors
The aim is to reduce dimensionality and at the same time maintain the data characteristics
If d(a,b) is the distance between two objects a, b in real (high-dimensional) and d’(a’,b’) is their distance in the transformed low-dimensional space, we want d’(a’,b’)d(a,b).
d’(a’,b’)
d(a,b)
Dr. N. Mamoulis
Advanced Database Technologies

13. Example: Discrete Fourier Transform (DFT)20 40 60 80
100
120
140 1 2 3 X X' 4 5 6 7 8 9
How to represent a time-series (or a color histogram) using only n numbers (in a n-dimensional space)?
Basic Idea: Represent the time series as a linear combination of sines and cosines, but keep only the first n/2 coefficients.
Why n/2 coefficients? Because each wave requires 2 numbers, for the phase (w) and amplitude (A,B).
Dr. N. Mamoulis
Advanced Database Technologies

14. Using Transformations to Reduce Dimensionality (cont’d)
Other popular transformations include:
Discrete Wavelet Transform. The sequence is transformed to a linear combination of Wavelet basis functions, but only the first n coefficients are used.
Singular Value Decomposition. Similarly, the sequence is transformed to a linear combination of eigenwaves, and only the first n coefficients are used.
Piecewise approximations are also used for time-series data.
In presentation 2 we will see one of the most commonly used methods for dimensionality reduction (FastMap). In presentation 3 we will see how the problem is solved for another similarity metric, the dynamic time warping.
Dr. N. Mamoulis
Advanced Database Technologies

15. Using Transformations to Reduce Dimensionality (cont’d)
After the transformation, the Euclidean distance of the vectors in the low-dimensional space lower bounds the actual distance in the high-dimensional space. If d(a,b) is the distance between two objects a, b in real (high-dimensional) and d’(a’,b’) is their distance in the transformed low-dimensional space, then d’(a’,b’)d(a,b).
A spatial access method is used (e.g., R-trees) to index the low-dimensional vectors and a NN-search algorithm (e.g., INN) to perform similarity search.
d’(a’,b’)
d(a,b)
Dr. N. Mamoulis
Advanced Database Technologies

16. Using Transformations to Reduce Dimensionality (cont’d)
Assume that we want to find most similar image (or time-series) to a query q.
q is transformed to q’ in the low dimensional space.
INN is performed using the R-tree and the nearest neighbor o’ is retrieved.
The distance between the (real) high-dimensional vectors is computed d(q,o).
We know that no object p with d’(q’,p’)>d(q,o) can be the NN because d(q,p)d’(q’,p’)>d(q,o)
A range query is applied on the R-tree using d(q,o) in the low dimensional space, to get all objects p with d’(q’,p’)<d(q,o).
The real distance d(q,p) for each p is computed and the closest p is the actual NN. q o
d(q,o)
d’(q’,o’)
d(q,o) o’ q’
candidates
Dr. N. Mamoulis
Advanced Database Technologies

17. Searching for Similar Subsequences in a Long SequenceT:
|T|- s windows
We index the long sequence T as follows.
We slide a window of size s at each position of T
We apply a transformation (e.g., DFT) to approximate the subsequence at each of the |T|-s positions.
We index the transformations using a SAM
(e.g., R-tree)
Use the tree to search for subsequences
which are close to the query subsequence.
The results are approximate, so they have to
be validated by accessing the exact data s
Dr. N. Mamoulis
Advanced Database Technologies

18. Advanced Database Technologies
Summary
Multimedia data can be represented by a set of feature values in the high-dimensional space.
Time-series data are event sequences in time.
Similarity search is expensive in high dimensionality; the data are usually approximated by low dimensional points, so that their distances ‘lower bound’ their actual high dimensional distances.
The low dimensional vectors are indexed by spatial access methods and NN search facilitates fast approximate similarity search.
We will now see why in high dimensional spaces search is often meaningless, a dimensionality reduction method and a time-series similarity technique.
Dr. N. Mamoulis
Advanced Database Technologies