Video Fingerprinting: Features for Duplicate and Similar Video Detection and Query- based Video Retrieval Anindya Sarkar, Pratim Ghosh, Emily Moxley and B. S. Manjunath Presented by: Anindya Sarkar Vision Research Lab, Department of Electrical & Computer Engg, University of California, Santa Barbara Januray 30, 2008

June 27, 2015 Problem Definition: Duplicate video and similar video detection –we represent a video compactly (fingerprint), for efficient storage and faster search without compromising the retrieval accuracy Query-based video retrieval –Input: short length (1-2% of big video length) query video –Output: actual “big” video from which the query is taken

June 27, 2015 Generation of Duplicate Videos Dataset: BBC rushes dataset, provided for the TRECVID task of video summarization Operations performed: –Image processing (per frame) based: Blurring using 3x3 and 5x5 window Gamma correction by 20% and -20% Gaussian noise addition at SNR of -20,0,10,20,30 and 40 dB JPEG compression at QF=10,30,50,70 and 90 –Frame drop based errors: frame drops of 20%, 40% and 60% of the original video for both random and bursty case.

June 27, 2015 Interpretation of Similar videos Different takes of the same scene are considered as “similar” videos These videos are similar in content –However, due to human variability at both the cameraman and actor level, (camera angles, cuts, and actor performance), videos may look similar but are still different BBC rushes dataset has unedited footage of the different retakes – hence, ideally suited for generation of similar videos

June 27, 2015 Keyframe based Video Fingerprint Video Summarization and key-frame extraction N frames in the actual video K key-framesK x d Video Fingerprint d-dimensional signature computed per key-frame Features used for fingerprint creation: 1. Compact Fourier Mellin Transform 2. Scale Invariant Feature Transform

June 27, 2015 R R is the maximum radius of in-circle m,n=0M-1 N-1 ∆r∆r ∆θ∆θ ∆r= log(R)/M, ∆θ=2π/N M is the no of concentric circles. N is the no. of diverging radial lines. x=e m∆r cos(n∆θ) y=e m∆r sin(n∆θ) (x,y) (m,n) Log-Polar Transformation First fix the value of M,N origin Any 2-D Matrix

June 27, 2015 CFMT FEATURE EXTRACTION m, n=0M-1 N-1 -(K-1) K-1 V-1 |FFT| -(V-1) Normalization & vectorization 50% A.C. Energy PCA Quantization

June 27, 2015 SIFT Feature Generally used for object recognition – hence, can be used as an image similarity measure Distance between SIFT features – number of descriptor comparisons makes it computationally prohibitive Speed up – quantize descriptors to a finite vocabulary (consisting of words) –Each image is a weighted vector of the word frequencies

June 27, 2015 most specific words M=1 M=3 more general words words image descriptors Straight vocabulary – created by clustering – e.g. 12 dimensional feature needs 12 clusters Vocabulary tree: created using hierarchical k-means on SIFT features; final vocabulary size=3+9=12 Each feature belongs to one “word” at each level M=9

June 27, 2015 Straight Vocabulary vs Vocabulary Tree Straight vocabulary: –Does not consider relationship between words That is, ignores that certain words are closer to each other than other words. –At very coarse level (dictionary size ~10-20), additional words are more descriptive than the relationship among words. Therefore, outperforms vocabulary tree. In our experiments, low-dimensional SIFT features, obtained using straight vocabulary, perform much better as “fingerprints” than tree-based features

June 27, 2015 Non-keyframe based Video Fingerprint N frames P=N/K frames, where each window has P frames P frames Video Fingerprint K x 125 Video Fingerprint Extraction for each of K windows Computing the 125-dim YCbCr Histogram in YCbCr Space using P Consecutive Frames and thus avoiding Key Frames Extraction. Whole color space is quantized into 125 bins (5 bins for each of Y, C b and C r ). Features used for fingerprint creation: YCbCr histogram based feature

June 27, 2015 Signature Distance Computation For two (K x d) fingerprints, X and Y, where X(i) = i th feature vector of X Properties of this distance function: Such a distance relation is called a “quasi-distance” d ( X ; Y ) = K X i = 1 ½ m i n 1 · j ·K jj X ( i ) ¡ Y ( j )jj 1 ¾ d ( X ; Y ) = 0 ; i sposs i bl eeven i f X 6 = Y d ( X ; Y ) 6 = d ( Y ; X )

June 27, 2015 Motivation Behind Distance Function This closest-overlap based distance is robust to: Frame reordering: For 2 signatures, temporal sequence may not be maintained between them – e.g. a video consisting of a reordering of scenes from the same video is still regarded as a duplicate Frame drops: If frame drops occur or some video frames are corrupted by noise, distance between duplicate videos should still be small

June 27, 2015 Experiments and Results We present precision-recall plots for both similarity and duplicate detection, over 3888 videos –CFMT for dimensions 36/24/20/12/4 –SIFT for dimensions 781/341/33/21/12 –CFMT vs best performing SIFT for duplicate detection –SIFT vs best performing CFMT for similarity detection CFMT performs better for duplicate detection SIFT performs better for similarity detection

June 27, 2015 Precision-recall curves for different dimensional CFMT for duplicate detection Precision-recall curves for different dimensional CFMT for similarity detection

June 27, 2015 Precision-recall curves for different dimensional SIFT for duplicate detection Precision-recall curves for different dimensional SIFT for similarity detection

June 27, 2015 Precision-recall curves, comparing different descriptors for similarity detection Precision-recall curves comparing different descriptors for duplicate detection

June 27, 2015 Full-length Video Retrieval with Clip Querying Generation of the small-length query: –We put together 4 different scenes from a full length video to create our input query: –Each individual scene is represented by 8 keyframes –For a single query, we have 4x8=32 keyframes –We experiment with different features for query representation Repository is of full-length video signature (65 videos): –Number of keyframes used to create the signature size for “large video” is varied from 1%-4% of video length

June 27, 2015 Algorithm Step 1: Input query signature X query is a (32 x d) matrix Step 2: Its distance from all the stored “large video” signatures (X large ) is computed, as shown below: Step 3: The best matched video is returned ¢ ( i ) = m i n j jj X query ( i ) ¡ X l arge ( j )jj 1 ; 1 · i · 32 ( 1 ) D ( X query ; X l arge ) = 32 X i = 1 ¢ ( i )= 32 ( 2 )

June 27, 2015 Video name CFMT- 36 CFMT- 20 CFMT- 12 YCbCr- 125 SIFT -781 SIFT -31 SIFT- 21 Query Query Query Query Video name CFMT- 36 CFMT- 20 CFMT- 12 YCbCr- 125 SIFT -781 SIFT -31 SIFT- 21 Query Query Query Query Retrieval results for 1% summary lengths for “large” videos Retrieval results for 4% summary lengths for “large” videos

June 27, 2015 Conclusions CFMT features provide quick/accurate retrieval for duplicate videos SIFT features perform better for similar video detection Future work –expanding the domain of “similar” videos (non-retakes yet still similar ?) –Importance of an efficient summary to create video signature (strategic keyframes vs random keyframes ?)

June 27, 2015 Thanks for your patience. Questions?