Using Relevance Feedback in Multimedia Databases Chotirat “Ann” Ratanamahatana Eamonn Keogh 7th International Conference on VISual Information Systems at 10th International Conference on Distributed Multimedia Systems September 9, 2004

Roadmap Time series in multimedia databases and their similarity measures Euclidean distance and its limitation Dynamic time warping (DTW) Global constraints and R-K Band Relevance Feedback and Query Refinement Experimental Evaluation Conclusions and future work

What are Time Series A collection of observations made sequentially in time. People measure things… and things…change over time… Their blood pressure George Bush's popularity rating The annual rainfall in San Francisco The value of their Google stock

Time Series in Multimedia Databases Image data may best be thought of as time series…

Image to Time Series

Video to Time Series Hand moving down to grasp gun Steady pointing Hand moving to shoulder level Hand moving down to grasp gun Hand moving above holster Hand at rest

Time Series in Multimedia Databases Video George Washington’s Manuscript

Classification in Time Series Class B Class A Which class does belong to? Pattern Recognition is a type of supervised classification where an input pattern is classified into one of the classes based on its similarity to these predefined classes.

Euclidean Distance Metric Given 2 time series Q = q1, …, qn and C = c1, …, cn their Euclidean distance is defined as Q C

Limitations of Euclidean Metric Very sensitive to some distortion in the data Training data consists of 10 instances from each of the 3 classes Perform a 1-nearest neighbor algorithm, with “leaving-one-out” evaluation, averaged over 100 runs. Euclidean distance Error rate: 29.77% DTW Error rate: 3.33 %

Dynamic Time Warping (DTW) Euclidean Distance One-to-one alignments Time Warping Distance Non-linear alignments are allowed

How Is DTW Calculated? (I) Q C Warping path w

How Is DTW Calculated? (II) Each warping path w can be found using dynamic programming to evaluate the following recurrence: where γ(i, j) is the cumulative distance of the distance d(i, j) and its minimum cumulative distance among the adjacent cells. (i-1, j) (i, j-1) (i, j) (i-1, j-1)

Global Constraints (I) Prevent any unreasonable warping Sakoe-Chiba Band Itakura Parallelogram

Global Constraints (II) A Global Constraint for a sequence of size m is defined by R, where Ri = d 0  d  m, 1  i  m. Ri defines a freedom of warping above and to the right of the diagonal at any given point i in the sequence. Ri Sakoe-Chiba Band Itakura Parallelogram

Ratanamahatana-Keogh Band (R-K Band) Solution: we create an arbitrary shape and size of the band that is appropriate for the data we want to classify.

How Do We Create an R-K Band? First Attempt: We could look at the data and manually create the shape of the bands. (then we need to adjust the width of each band as well until we get a good result) 100 % Accuracy!

Learning an R-K Band Automatically Our heuristic search algorithm automatically learns the bands from the data. (sometimes, we can even get an unintuitive shape that give a good result.) 100 % Accuracy as well!

R-K Band Learning With Heuristic Search

R-K Band Learning in Action!

Classification Examples with R-K Bands Error rate Euclidean 32.13% DTW 10% 4.52% R-K Bands 0.9%

Face Classification

Relevance Feedback A well-known and effective method in improving the query performance, especially in text-mining domains. Refining the query based on user’s reaction Only relatively little research has been done on relevance feedback in images or multimedia data.

Query Refinement Averaging a collection of time series using DTW, according to their weights and warping (DTW) alignments.

Experiment: Datasets Gun Problem Leaf Dataset Handwritten Word Spotting data

Experimental Design Given an initial query, we measure the precision and recall for each round of the relevance feedback retrieval. Show the 10 best matches (k-nearest neighbors). User ranks each result. Accumulatively build the training set. Learn an R-K band according to the current training data. Generate a new query (query refinement), and repeat.

Results: Gun

Results: Leaf

Results: Wordspotting

Conclusions Different shapes and widths of the band contributes to the classification accuracy / precision. We have shown that incorporating R-K Band into relevance feedback can reduce the error rate in classification, and improve the precision at all recall levels in video and image retrieval.

Future Work Investigate other choices that may make envelope learning more accurate. Heuristic functions Search algorithm (refining the search) Is there a way to always guarantee an optimal solution? Examine the best way to deal with multi-variate time series for more complex data. Explore other utilities of R-K Band and relevance feedback, specifically on real-world problems: music, bioinformatics, biomedical data, etc.

Thank You Questions? Contact: ratana@cs.ucr.edu eamonn@cs.ucr.edu Homepage: http://www.cs.ucr.edu/~ratana All datasets are publicly available at: UCR Time Series Data Mining Archive: http://www.cs.ucr.edu/~eamonn/TSDMA