1. WISE: Large Scale Content-Based Web Image Search Michael Isard Joint with: Qifa Ke, Jian Sun, Zhong Wu Microsoft Research Silicon Valley 1

2. 2 What leaf ? Artist ? Higher resolution? Bank web-site? Email ? Ad. in e-bay ? …… © : Who else using this? Query by Images “A picture is worth a thousand words.”

3. Partial-Duplicate Image Search Given a query image, find its partial duplicates from a database of web-images

4. Two Major Challenges How to represent images – No text annotations or labels – Noise and modification How to efficiently index and query images – Large number of images (millions) 4 Index ?

5. Image Representation: Bag-of-Words [CVPR’09, ICCV’09] Descriptor ID 1 3506 999,999 1,000,000 Detection [Lowe 2004, Matas et al 2002, Winder et al 2007] Bag-of-words [Sivic&Zisserman’2003] …… Code-book NormalizationDescription [Lowe 2004, Winder et al 2007] 3506 1 999,999 206 … 2: Quantization1: Feature extraction: Bundle Features3: Representation 5

6. 6 …

7. Matching query to database Use an index – Each visual word has a ‘posting list’ – Lists every image containing the word At query time – Look up the posting list for each query word – Merge lists to find candidate images Partial match: don’t need every word to be present 7

8. How much work to query? Disk-based index, bottleneck is random reads – One seek per posting list Also one seek per matching image – To fetch thumbnail etc Keep as little information as possible in posting lists, to keep index size small 8

9. Index Pipeline Implemented in a large computer cluster – 256-nodes, using Dryad/DryadLINQ 9 Image Crawler Image Crawler Content Chunk Visual Words Feature Quantizer Feature Quantizer Crawler Bundled Features Feature Extractor Feature Extractor Media DB (Thumbnail, URL) Local Features Indexer Inverted Index Visual Word

10. Query Pipeline 10 Bundled Features Feature Extractor Feature Extractor Visual Words Feature Quantizer Feature Quantizer Query Image Query Image Inverted Index Server Index Server Search Results (chunkID, imgID) Search Results (chunkID, imgID) Media DB (Thumbnail, URL) GUI Results …

11. Bag-of-Words: Limitations Quantization – Lost discriminative power – Sensitive to image variations and noises – Soft quantization [Philbin et al, CVPR 2008] – Hamming embedding [Jegou et al, ECCV 2008] Descriptor ID 1 3506 999,999 1000,000 …… Quantization

12. Geometric verification In practice, bag of words is too weak Does not exploit any geometry Post-process to check spatial layout of matching features Requires a disk seek per image – Only used as a re-ranking step to shortlist of matched images 12

13. Geometric Re-ranking Re-rank top 300 images

14. Geometry in the index Previous works: – Jegou et al ECCV 2008 Try to match similar orientations and scales – Perdoch et al CVPR 2009 Match oriented features more effectively Still feature-by-feature – Global geometric consitency applied at the end 14

15. Single Feature is Weak + +

16. Neighboring Features ?

17. Define Neighboring Features Previous works – kNN voting [Sivic&Zisserman 2003] – Higher-order spatial features [Liu et al][Yuan et al][Tirilly et al][Quack et al] – Post geometric spatial verification [Lowe’2004][Chum et al 2007][Nister 2006][Philbin et al 2007]…… – Geometric Min-Hash [Chum et al 2009] Challenges – Repeatable – Partial matching – Scalable: simple enough to build into index

18. Define Neighboring Features DoG Features [Lowe 2004] MSER Features [Matas et al 2002] - point features - repeatable - region features - repeatable

19. Define Neighboring Features DoG Features [Lowe 2004] MSER Features [Matas et al 2002] - point features - repeatable - region features - repeatable region groups points?

20. Bundled Feature: Definition Bundled Feature = A set of DOG features bundled by a MSER region DoG interest points MSER region

21. Bundled Feature: Definition Bundled Features

22. Matched bundle p = { p i } Query bundle q = { q j } = { } Matching Bundles: Membership Membership score: Voting weight:

23. Matched bundles p 1, p 2, p 3 Query bundle q = { q j } = { } Membership score: Matching Bundles: Membership p1p1 p2p2 p3p3

24. Matching Bundles: Geometric Constraint 1 3 2 4 4 1 3 2 5 order in target img: 1 < 3 < 4 < 5 order inconsistency: 0 + 0 + 0 = 0 Penalize inconsistent relative orders: y 1 < 2 < 3 < 4 order in query img: 1 < 2 < 3 < 4 1 3 2 4 2 1 3 4 5 5 > 2 > 1 < 3 1 + 1 + 0 = 2 matching order: inconsistency: order in query img: querycandidatequerycandidate

25. Matching Bundles: Formulation Bundle matching score: Image matching score: geometric constraintmembership - Repeatable - Partial matching - Scalable?

26. Inverted Index (without Bundles) Posting …… Image ID Visual word Visual word …… … … … … … … 27 … … …… Image ID = 27

27. Inverted Index with Bundles Posting …… Image ID Visual word Visual word …… … … … … … … 27, [3,1,1] … … 27, [1,2,1] …… Image ID = 27 Bundle Bits 9 bits 5 bits Bundle IDX-OrderY-Order p1p1 p2p2 p3p3 1 1 3 3 2 2 27, [2,2,2]

28. Retrieval Query Image I q Inverted index with bundle bits 1,1, [2,5,9] 1,1, [2,5,9] 3,1, [3,4,5] 9,2, [3,2,5] 9,2, [3,2,5] 10, 1, [1,1,2] 10, 1, [1,1,2] 12,1, [1.1.2] … … 10, 1, [2,2,1] 10, 1, [2,2,1] … … … … … … …… p1p1 p2p2 p3p3 Top candidate images

29. Experimental Settings Image database: – 1M web images from query-click log Ground truth partial duplicates – 780 known partial duplicate images in 19 groups Baseline bag-of-words – Visual word vocabulary size = 1 M – Soft quantization factor = 4 – 500 features per image

30. Partial Duplicate Example ……

31. Partial Duplicate Example ……

32. Example Query Results Challenging cases Query

33. Evaluation: Precision-Recall A query returns N images – T : correct matches – A : expected matches Recall Precision

34. Comparison: Precision-Recall Baseline bag-of-words (started from 13 th ) Bundled features (started from 13 th ) Query image:

35. More Precision-Recall Comparisons

36. Average Precision (AP) for one query: – Area under Precision-Recall curve mAP: mean of AP’s from all testing queries Evaluation: mAP AP

37. mAP: Baseline Bag-of-Words

38. mAP: Hamming Embedding (HE)

39. mAP: Bundle (Membership)

40. mAP: Bundle (both terms) 26%40%

41. mAP: Bundle + HE 49%

42. Bundle VS. Geometric Re-ranking Re-rank top 300 images

43. Bundle + Geometric Re-ranking Re-rank top 300 images 24%77%

44. More Results

45. Failure Case

46. Demo Setup Document Server Web Server Index Servers Client 46 6 million images

47. Demo Query image Results 47

48. Conclusion Bundle feature – More discriminative – Enforce spatial constraints while traversing index – Partial match – Scalable: built into index 9 bits 5 bits Bundle IDX-OrderY-Order

49. Thanks!