A Novel Scheme for Video Similarity Detection Chu-Hong Hoi, Steven March 5, 2003

Outline  Introduction  Overview  Phase I: Coarse Similarity Measure  Pyramid Partitioning & Density Histogram  Naïve Pyramid Density Histogram (NPDH)  Fuzzy Pyramid Density Histogram (FPDH)  Phase II: Fine Similarity Measure  Near Feature Trajectory (NFT)  Simplification Algorithm  Similarity Measure Based on NFT  Experiments and Results  Conclusion & Future Work

Introduction  Motivation Huge volume of video data are distributed over the Web. How to fast detect the similar video effectively?  Applications Copyright issues / watermarking Content-based video retrieval

Overview  Challenging Issues Efficiency Effectiveness  We propose a Two-Phase Similarity Detection Framework based on two kinds of signatures with different granularity.  Solutions by two kinds of signatures Coarse Signature  Pyramid Density Histogram Fine Signature  Nearest Feature Trajectory

Overview A two-phase framework for video similarity detection

Phase I: Coarse Similarity Measure  Video data indexing A frame is considered as a feature point. A video sequence is formed by a series of feature points. It is hard to index and search the video dat in original data space.  Two partitions of data space Regular partitioning (Fig.2 (a)) Pyramid partitioning (Fig.2 (b)) (S. Berchtold-SIGMOD 98) Center Point at (0.5,0.5,…,0.5)

Phase I: Coarse Similarity Measure  Pyramid Density Histogram (PDH) Map the feature points to the pyramid data space, and statistically calculate the distribution of the feature points Obtain a density histogram of feature points as the coarse signature  Two kinds of PDHs Na ï ve Pyramid Density Histogram Fuzzy Pyramid Density Histogram

Phase I: Coarse Similarity Measure  Na ï ve Pyramid Density Histogram 2d-dimension NPDH vector u=(u1,u2, …,u2d)  How to calculate the density histogram? For a d-dimension feature point v=(v1,v2, …,vd) Center Point at (0.5,0.5,…,0.5)

Phase I: Coarse Similarity Measure  Fuzzy Pyramid Density Histogram In NPDH, a given feature point is allocated to only 1 pyramid. It would loss the information of other dimensions. In FPDH, we fuzzyly allocate a feature point v to d pyramids based on the value of its d dimensions. Center Point at (0.5,0.5,…,0.5) j=1,2,…,d

Phase I: Coarse Similarity Measure  Similarity Filtering Based on PDH Given a query example q and a compared sample s from the video database. Set a filtering threshold δ, then video s is filtered out if it satisfies the following condition:

Phase II: Fine Similarity Measure  Conventional Similarity Measure Nearest Neighbor (NN) or (k-NN) Nearest Center (NC) Disadvantage: ignore the temporal information of video sequences  Nearest Feature Trajectory (NFT) A video sequence is considered as a series of feature trajectories rather than isolated key- frames.

Phase II: Fine Similarity Measure  Nearest Feature Trajectory A frame in a video sequence is considered as a feature point. Two feature points form a feature line. A series of feature lines form a feature trajectory in a video shot. A video sequence consists of a series of feature trajectories. Each trajectory corresponds to a individual shot or a gradual transition of shots. Similarity measure is based on the nearest average distance of feature trajectories in two video sequences.

Phase II: Fine Similarity Measure  Generation of Simplified Feature Trajectory Formulate the procedure by Minimum Square Error approach The minimum procedure of MSE is time-consuming!

Phase II: Fine Similarity Measure  We propose an algorithm for efficient generate the feature trajectories. Define a local similarity measure function to approximate the deviation degree. The larger the value of LR(v k ) is, the larger the deviation degree at v k is. Based on the LR(v k ), we remove the point with the minimum value each time until there remains only feature points.

Phase II: Fine Similarity Measure  Similarity Measure Based NFT

Phase II: Fine Similarity Measure Distance Measure of Two Feature Trajectories Similarity Measure of Two Video Sequences Considering the boundary problem, if 0 ≦ λ ≦ 1, falls in the line segment; otherwise, it falls out of the line

Experiments and Results  Ground Truth Data About 300 video clips with different coding formats, resolutions and slight color modifications  Feature Extraction RGB Color Histogram 64 dimensions  Performance Evaluation Metric Average Precision Rate Average Recall Rate

Experiments and Results  Coarse Similarity Measure FPDH vs. NPDH

Experiments and Results  Fine Similarity Measure NFT vs. NN

Conclusions We propose an effective two-phase framework to achieve the video similarity detection. Different from the conventional way, our similarity measurement scheme is based on different granular similarity measure. In the coarse measurement phase, we suggest Fuzzy Pyramid Density Histogram. In the fine measurement phase, we present the Nearest Feature Trajectory technique. Experimental results show that our scheme is better than the conventional approach.

Future Work  Engaging more effective features in our scheme to improve the performance  Enlarging our database and testing more versatile data  Cost performance evaluation

Q & A Thank you!