Similarity Measurement and Detection of Video Sequences Chu-Hong HOI Supervisor: Prof. Michael R. LYU Marker: Prof. Yiu Sang MOON 25 April, 2003 Dept. of Computer Science & Engineering, CUHK The 2nd Term Presentation of M.Phil.

2 Outline Introduction Overview Coarse Similarity Measure  Pyramid Partitioning & Density Histogram  Naïve Pyramid Density Histogram (NPDH)  Fuzzy Pyramid Density Histogram (FPDH)  General Pyramid Density Histogram (GPDH) Fine Similarity Measure  Near Feature Trajectory (NFT)  Simplification Algorithm  Similarity Measure Based on NFT Experiments and Results Conclusions and Future Work

3 Introduction Motivation  Huge volume of video data are distributed over the Web.  To fast and effective detect the similar video becomes an important work. Applications  Copyright issues, complement of watermarking  Content-based video retrieval  Fault tolerant services on the web

4 Overview Challenging Issues  Efficiency  Effective Measurement Solution  We propose a novel t wo-phase similarity detection framework based on two kinds of signatures with different granularities. Two kinds of signatures  Coarse Signature Pyramid Density Histogram  Fine Signature Nearest Feature Trajectory

5 Overview A two-phase framework for video similarity detection

6 Coarse Similarity Measure Difficulties of fast searching  The number of frames in video sequences is too large for efficient indexing and searching.  It is difficult to perform fast measurement in the original high dimension data space. Solution  Use partitioning technique solve the problem. Partitions of data space  Regular partitioning (a)  Pyramid partitioning (b) (S. Berchtold-SIGMOD 98) Center Point at (0.5,0.5,…,0.5)

7 Coarse Similarity Measure Pyramid Density Histogram (PDH)  Map the feature points to the pyramid data space, and then statistically calculate the distribution of the feature points in each pyramid  Obtain a density histogram of feature points as a coarse signature  Perform fast filtering based on the PDH signature Three kinds of PDHs  Naïve Pyramid Density Histogram  Fuzzy Pyramid Density Histogram  General Pyramid Density Histogram

8 Coarse Similarity Measure Definition 1. (Naïve Pyramid Density Histogram) Given a video sequence S formed by n d-dimension feature points, the original data space can be mapped to a 2d-dimension feature vector u by the PDH technique. The NPDH feature vector denoted by is calculated as: Center Point at (0.5,0.5,…,0.5)

9 Coarse Similarity Measure Definition 2. (Fuzzy Pyramid Density Histogram) Given a video sequence S formed by n d-dimension feature points with d-dimension, S can be mapped to a 2d- dimension feature vector u by the Fuzzy Pyramid Density Histogram technique. The FPDH vector u denoted as, which is calculated as follow: Center Point at (0.5,0.5,…,0.5) For each j=1,2,…,d

10 Coarse Similarity Measure NPDH vs. FPDH  In NPDH, a given feature point is allocated to only one pyramid. It would loss the information of other dimensions.  In FPDH, we fuzzily allocate a feature point to d pyramids based on the value in each dimension.  FPDH can fully exploit the information in each dimension compared with NPDH. Extension of FPDH  In order to obtain a more general form of FPDH, we present the General Pyramid Density Histogram (GPDH).  FPDH is the special case of GPDH with general factor n=2.

11 Coarse Similarity Measure Definition 3. (General Pyramid Density Histogram) Given a video sequence S formed by n d-dimension feature points with d-dimension, S can be mapped to a (nd)- dimension feature vector u by the PDH technique. The GPDH vector u denoted as, which is calculated as follow: For each j=1,2,…,d

12 Coarse Similarity Measure Similarity Filtering Based on PDH  Given a query example Q and a compared sample C from the video database.  Set a filtering threshold, then the video C is filtered out if it satisfies the following condition:

13 Fine Similarity Measure Conventional Similarity Measure  Nearest Neighbor (NN) or (k-NN)  Hausdorff Measure  Disadvantage: ignore the temporal information of video sequences, not effective enough Nearest Feature Trajectory (NFT)  A video sequence is considered as a series of feature trajectories rather than isolated key- frames.

14 Fine Similarity Measure Nearest Feature Trajectory  A frame in a video sequence is considered as a feature point.  Two feature points form a feature line.  A series of feature lines form a feature trajectory in a video shot.  A video sequence consists of a series of feature trajectories.  Similarity measure is based on the nearest distance of feature trajectories in two video sequences.

15 Fine Similarity Measure Generation of Simplified Feature Trajectory  Formulate the procedure by least Mean Square Error approach The minimum procedure of MSE is time-consuming! original simplified target line segment reconstruction N: frame num. of S : frame num. of S’

16 Fine Similarity Measure We propose an algorithm for efficient generating the feature trajectories.  Define a local similarity measure function to approximate the deviation degree.  The larger the value of is, the larger the deviation degree at is.  Based on the value of, we remove the point with the minimum value each time until there remains only feature points in the trajectory.

17 Fine Similarity Measure Similarity Measure Based NFT

18 Fine Similarity Measure  distance of two feature trajectories is measured as  dissimilarity of two video sequences is given as  Considering the boundary problem, if 0 ≦ λ ≦ 1, falls in the line segment; otherwise, it falls out of the line

19 Experiments and Results Ground Truth Data  We collect 300 video clips with different coding formats, resolutions and slight color modifications Low Level Feature Extraction  RGB Color Histogram  64-dimension Performance Evaluation Metric  Average Precision Rate  Average Recall Rate

20 Experiments and Results Coarse Similarity Measure  FPDH vs. NPDH

21 Experiments and Results Coarse Similarity Measure  GPDH varies with different factor n

22 Experiments and Results Fine Similarity Measure  NFT vs. NN & Hausdorff

23 Conclusions  We propose an effective two-phase framework to achieve the video similarity detection.  Different from the conventional way, our similarity measurement scheme is based on different granular similarity measure.  In the coarse measurement phase, we suggest Pyramid Density Histogram technique.  In the fine measurement phase, we present the Nearest Feature Trajectory algorithm.  Experimental results show that our scheme is better than the conventional approaches.

24 Future Work Engaging more effective features in our scheme to improve the performance Enlarging our database and testing more versatile data Conducting the cost performance evaluations and comparisons of our scheme

25 Q & A Thank you!