1. Problem Statement A pair of images or videos in which one is close to the exact duplicate of the other, but different in conditions related to capture, edits, and rendering. Problem: Spatial shift and scale variations

2. Tasks and Applications Near Duplicate Retrieval (NDR): –Copyright infringement detection –query-by-example application Near Duplicate Detection (NDD) –Link news stories and group them into threads –Filter out the redundant near duplicate images or videos in the top results from text keywords based web search

3. Prior Work Attributed Relational Graph (ARG) matching: ACM Multimedia 2004 Point set matching: ACM Multimedia 2004 One-to-one symmetric matching algorithm: T-MM 2007 and ACM Multimedia 2007 Large-scale near duplicate detection: CIVR 2007

4. Prior Work: Spatial pyramid match kernel  First quantize descriptors (SIFT) into words, then do one pyramid match per word in image coordinate space. Lazebnik, Schmid & Ponce, CVPR 2006

5. Fusion of information from different levels. Alignment of different subclips (Level-1 as an example) EMD Distance Matrix between Sub-clips Integer-value Alignment Smoke Fire Smoke Level-0 Level-1 Temporally Constrained Hierarchical Agglomerative Clustering Fire Temporal Pyramid Matching for Event Recognition in News Video Level-2 D. Xu & S.-F. Chang, CVPR 2007 and T-PAMI 2008

6. Earth Mover’s Distance (EMD) d ij Supplier P is with a given amount of goods Receiver Q is with a given limited capacity Weights: Solved by linear programming 1/m 1/2m 1/2m

7. Spatially Aligned Pyramid Matching Non-overlapped and overlapped partition at multiple-levels: Divide images into non-overlapped blocks Divide images into overlapped blocks with size equaling of the original image (in width and height) sampled at a fixed interval, say 1/8 of the image width and height.

8. First Stage Matching Objective: Compute the pairwise distances between any two blocks and. Solution: We represent each block as a bag of orderless SIFT descriptors and use EDM distance to measure the similarity between two sets of descriptors of unequal cardinality. Jianguo Zhang et al. Local Features and Kernels for Classification of Texture and Object Categories: A Comprehensive Study, IJCV, 2007

9. Second Stage Matching (1) Objective: Align the blocks from one query image x to corresponding blocks in its near duplicate image y. SAPM (our work): One block may be matched to another block at a different position and/or scale level to robustly handle piecewise spatial translations and scale variations. SPM: fixed block-to-block matching

10. Second Stage Matching (2) Eq (3) can be reestablished from Eq (4): (Assume R<C) 1) adding C − R virtual blocks in image x, 2) setting, for all r satisfying R < r ≤ C. Solution: Integer-flow EMD

11. Comparison of SAPM, SPM and TPM Three blocks in the query images (i.e.,(a)) and their matched counterparts in near duplicate images (i.e.,(b), (c), (d)) are highlighted and associated by the same color outlines.

12. Third Stage Matching Extension to Near Duplicate Video Identification: One video clip V 1 comprises {x(1), x(2), …, x(M)}, where x(i) is the i-th frame and M is the total number of frames of V 1 ; Another video clip V 2 comprises {y(1), y(2),…, y(N)}, where N and y(j) are similarly defined. Solution: temporal matching with EMD again.

13. Discussion If the query image was divided into non- overlapped blocks (e.g., L2-N) and the corresponding database images were divided into overlapped blocks (e.g. L2-O) at the same level, spatial shifts and some degree of scale change are addressed (e.g., ) a broad range of scale variations is considered by matching the query image and the database images at different levels (e.g., ) Ideally, SAPM can deal with any variations from spatial shift and scale variation by using more denser scales and spatial spacings.

14. Near Duplicate Retrieval and Detection NDR: We directly fuse the distances from different levels : NDD: Generalized Neighborhood Component Analysis (GNCA) We use p = {0, 1, 2, 3, 4} to indicate partitions designated as level-0 non- Overlapped (L0-N), level-1 non-overlapped (L1-N), level-1 overlapped (L1-O), level-2 non- overlapped (L2-N), and level-2 overlapped (L2-O).

15. Experiments: Three Datasets Columbia Near Duplicate Image Database: TRECVID 2003 corpus New Image Dataset: TRECVID 2005 and 2006 corpus –150 near duplicate pairs (300 images) and 300 non-duplicate images New Video Dataset: TRECVID 2005 and 2006 corpus –50 near duplicate pairs (100 videos) and 200 non-duplicate videos The images are collected from real broadcast news (rather than edits of the same image by the authors).

16. Comparison of SAPM with SPM and TPM for Image NDR Columbia Database New Image Dataset

17. SAPM and GNCA for Image NDD Performance Measure: Equal Error Rate (EER) SAPM+NCA and SAPM+GNCA: 20 positive and 80 negative samples to train the projection matrices in NCA and GNCA, another 40 positive and 160 negative samples were used for SVM training. SPM, TPM and SAPM: all training samples (60 positive and 240 negative) were used for SVM training. Test samples: 90 (positive) and 4840 (negative).

18. Comparison of SAPM+TM, SPM+TM and TPM+TM for Video NDR 1: Single-level L0-N->L0-N; 2: Single-level L1-N ->L1-N (or L1-O); 3: Multi-level. Two weighting schemes in temporal matching: normalized weight (NW) and unit weight (UW)

19. Conclusion A multi-level spatial matching framework for image and video near duplicate identification. GNCA outperforms NCA for near duplicate detection.