1. Using Relevance Feedback in Multimedia Databases
Chotirat “Ann” Ratanamahatana
Eamonn Keogh
7th International Conference on VISual Information Systems
at 10th International Conference on Distributed Multimedia Systems
September 9, 2004

2. Roadmap
Time series in multimedia databases and their similarity measures
Euclidean distance and its limitation
Dynamic time warping (DTW)
Global constraints and R-K Band
Relevance Feedback and Query Refinement
Experimental Evaluation
Conclusions and future work

3. What are Time Series
A collection of observations made sequentially in time.
People measure things…
and things…change over time…
Their blood pressure
George Bush's popularity rating
The annual rainfall in San Francisco
The value of their Google stock

4. Time Series in Multimedia Databases
Image data may best be thought of as time series…

5. Image to Time Series

6. Video to Time Series Hand moving down to grasp gun
Steady pointing
Hand moving to
shoulder level
Hand moving down
to grasp gun
Hand moving above holster
Hand at rest

7. Time Series in Multimedia Databases
Video
George Washington’s
Manuscript

8. Classification in Time Series
Class B
Class A
Which class does
belong to?
Pattern Recognition is a type of supervised classification where an input pattern is classified into one of the classes based on its similarity to these predefined classes.

9. Euclidean Distance Metric
Given 2 time series
Q = q1, …, qn and
C = c1, …, cn
their Euclidean distance is
defined as Q C

10. Limitations of Euclidean Metric
Very sensitive to some distortion in the data
Training data consists
of 10 instances from
each of the 3 classes
Perform a 1-nearest neighbor algorithm, with “leaving-one-out” evaluation, averaged over 100 runs.
Euclidean distance Error rate:
29.77%
DTW Error rate:
3.33 %

11. Dynamic Time Warping (DTW)
Euclidean Distance
One-to-one alignments
Time Warping Distance
Non-linear alignments are allowed

12. How Is DTW Calculated? (I)Q C
Warping path w

13. How Is DTW Calculated? (II)
Each warping path w can be found using dynamic programming to evaluate
the following recurrence:
where γ(i, j) is the cumulative distance of the distance d(i, j) and its minimum
cumulative distance among the adjacent cells.
(i-1, j)
(i, j-1)
(i, j)
(i-1, j-1)

14. Global Constraints (I)
Prevent any
unreasonable
warping
Sakoe-Chiba Band
Itakura Parallelogram

15. Global Constraints (II)
A Global Constraint for a sequence of size m is defined by R, where
Ri = d  d  m, 1  i  m.
Ri defines a freedom of warping above and to the right of the diagonal
at any given point i in the sequence. Ri
Sakoe-Chiba Band
Itakura Parallelogram

16. Ratanamahatana-Keogh Band (R-K Band)
Solution: we create an arbitrary shape and size of the band that is
appropriate for the data we want to classify.

17. How Do We Create an R-K Band?
First Attempt: We could look at the data and manually create the shape of the bands.
(then we need to adjust the width of each band as well until we get a good result)
100 % Accuracy!

18. Learning an R-K Band Automatically
Our heuristic search algorithm automatically learns the bands from the data.
(sometimes, we can even get an unintuitive shape that give a good result.)
100 % Accuracy as well!

19. R-K Band Learning With Heuristic Search

20. R-K Band Learning in Action!

21. Classification Examples with R-K Bands
Error rate
Euclidean
32.13%
DTW 10%
4.52%
R-K Bands
0.9%

22. Face Classification

23. Relevance Feedback
A well-known and effective method in improving the query performance, especially in text-mining domains.
Refining the query based on user’s reaction
Only relatively little research has been done on relevance feedback in images or multimedia data.

24. Query Refinement
Averaging a collection of time series using DTW, according to their weights and warping (DTW) alignments.

25. Experiment: Datasets Gun Problem Leaf Dataset
Handwritten Word Spotting data

26. Experimental Design
Given an initial query, we measure the precision and recall for each round of the relevance feedback retrieval.
Show the 10 best matches (k-nearest neighbors).
User ranks each result.
Accumulatively build the training set.
Learn an R-K band according to the current training data.
Generate a new query (query refinement), and repeat.

28. Results: Gun

29. Results: Leaf

30. Results: Wordspotting

31. Conclusions
Different shapes and widths of the band contributes to the classification accuracy / precision.
We have shown that incorporating R-K Band into relevance feedback can reduce the error rate in classification, and improve the precision at all recall levels in video and image retrieval.

32. Future Work
Investigate other choices that may make envelope learning more accurate.
Heuristic functions
Search algorithm (refining the search)
Is there a way to always guarantee an optimal solution?
Examine the best way to deal with multi-variate time series for more complex data.
Explore other utilities of R-K Band and relevance feedback, specifically on real-world problems: music, bioinformatics, biomedical data, etc.

33. Thank You Questions? Contact: ratana@cs.ucr.edu eamonn@cs.ucr.edu
Homepage:
All datasets are publicly available at:
UCR Time Series Data Mining Archive: