1. January, 7th 2011Simon Giraudot & Ugo Martin - MoSIG - Machine learning1/1/ Large-Scale Image Retrieval with Compressed Fisher Vectors Presentation of the article by Florent Perronnin, Yan Liu, Jorge Sanchez and Hervé Poirier from Xerox Research center Europe

2. January, 7th 2011Simon Giraudot & Ugo Martin - MoSIG - Machine learning2/2/ 1 – About Fisher Vectors a. Building the Fisher Vector of an image b. Measuring the similarity 2 – Compression techniques a. α = 0 binarization b. Local Sensitive Hashing c. Spectral hashing 3 – Experimental results a. Influence of the vocabulary size b. Comparative analyse of the compression techniques c. Comparison with compressed BOV Outline

3. January, 7th 2011Simon Giraudot & Ugo Martin - MoSIG - Machine learning3/3/ Large-Scale Image Retrieval with Compressed Fisher Vectors 0 – Recall : Bag of Visual Words (BOV) Offline Image → Interest points → Local Descriptors Clustering builds a visual vocabulary Query Image → Interest points → Local Descriptors → Visual words Tf-idf scoring (using inverted lists)

4. January, 7th 2011Simon Giraudot & Ugo Martin - MoSIG - Machine learning4/4/ Large-Scale Image Retrieval with Compressed Fisher Vectors 1 – About Fishers Vectors a. Building the Fisher Vector of an image As with BOV, build a set of visual words : Image X → Set of descriptors Clustering with Gaussian Mixture Model (GMM) : : Gaussian = visual word. : weight = global frequency N: number of visual words

5. January, 7th 2011Simon Giraudot & Ugo Martin - MoSIG - Machine learning5/5/ Large-Scale Image Retrieval with Compressed Fisher Vectors 1 – About Fishers Vectors a. Building the Fisher Vector of an image i-th Gaussian part : Soft assignment of a descriptor to Gaussian i : D dimensional vector

6. January, 7th 2011Simon Giraudot & Ugo Martin - MoSIG - Machine learning6/6/ Large-Scale Image Retrieval with Compressed Fisher Vectors 1 – About Fishers Vectors a. Building the Fisher Vector of an image Fisher Vector of X : concatenation N.D dimensional vector

7. January, 7th 2011Simon Giraudot & Ugo Martin - MoSIG - Machine learning7/7/ Large-Scale Image Retrieval with Compressed Fisher Vectors 1 – About Fishers Vectors b. Measuring the similarity Fisher vectorsTf-idf generated by Gaussian i → Influence of frequent (i.e. background) descriptors discounted

8. January, 7th 2011Simon Giraudot & Ugo Martin - MoSIG - Machine learning8/8/ Large-Scale Image Retrieval with Compressed Fisher Vectors 1 – About Fishers Vectors b. Measuring the similarity Dot product of Fisher vectors measures similarity ~ tf-idf scoring

9. January, 7th 2011Simon Giraudot & Ugo Martin - MoSIG - Machine learning9/9/ Large-Scale Image Retrieval with Compressed Fisher Vectors 2 – Compression techniques Descriptors of size D = 64 Size of Fisher Vectors = major problem Visual vocabulary of size N = 100 D x N = 6400 dimensions ! Using Floating points 25kB signature

10. January, 7th 2011Simon Giraudot & Ugo Martin - MoSIG - Machine learning10/ Large-Scale Image Retrieval with Compressed Fisher Vectors 2 – Compression techniques Examples of Hamming distances: Ha (010011 ; 011001) = 2 Ha (hello ; yemmo) = 3 Ha (23445 ; 89415) = 3 Principle : binarization → from Euclidian space to Hamming space

11. January, 7th 2011Simon Giraudot & Ugo Martin - MoSIG - Machine learning11/ Large-Scale Image Retrieval with Compressed Fisher Vectors 2 – Compression techniques a. α = 0 binarization Normalization → discounts the influence of large values A {x 1, x 2, …, x N } B {x 1 α, x 2 α, …, x N α } Y = X α with 0 < α < 1 X Y

12. January, 7th 2011Simon Giraudot & Ugo Martin - MoSIG - Machine learning12/ Large-Scale Image Retrieval with Compressed Fisher Vectors 2 – Compression techniques a. α = 0 binarization N(D+1) bits 0,8kB signature Dot product :

13. January, 7th 2011Simon Giraudot & Ugo Martin - MoSIG - Machine learning13/ Large-Scale Image Retrieval with Compressed Fisher Vectors 2 – Compression techniques b. Local Sensitive Hashing Binarization: ● Using a set of random vectors (r b, b = 1...B) ● Computing the sign of r b 'x

14. January, 7th 2011Simon Giraudot & Ugo Martin - MoSIG - Machine learning14/ Large-Scale Image Retrieval with Compressed Fisher Vectors 2 – Compression techniques b. Local Sensitive Hashing 1 1 1 0 0 r1r1 r5r5 r3r3 r4r4 r2r2 Fisher Vector {x 1, x 2,..., x N } with N huge (~6400) Compressed Fisher Vector {1, 1, 1, 0, 0} B bits Scalar product with B vectors (example with B = 5) X

15. January, 7th 2011Simon Giraudot & Ugo Martin - MoSIG - Machine learning15/ Large-Scale Image Retrieval with Compressed Fisher Vectors 2 – Compression techniques c. Spectral hashing Binarization such that: Vectors far apart in original Euclidian space = Binarized vectors far apart in the Hamming space

16. January, 7th 2011Simon Giraudot & Ugo Martin - MoSIG - Machine learning16/ Large-Scale Image Retrieval with Compressed Fisher Vectors 2 – Compression techniques c. Spectral hashing ● LSH good but can we do better than random ? ● Minimize the sum of Hamming distances ● Eigenvector basis of Laplacian of a similarity graph → Fourier tranform → Spectral hashing ● Data uncorrelated = uniformly distributed in each direction

17. January, 7th 2011Simon Giraudot & Ugo Martin - MoSIG - Machine learning17/ Large-Scale Image Retrieval with Compressed Fisher Vectors 3 – Experimental results a. Influence of the vocabulary size

18. January, 7th 2011Simon Giraudot & Ugo Martin - MoSIG - Machine learning18/ Large-Scale Image Retrieval with Compressed Fisher Vectors 3 - b. Comparative analyse of the compression techniques

19. January, 7th 2011Simon Giraudot & Ugo Martin - MoSIG - Machine learning19/ Large-Scale Image Retrieval with Compressed Fisher Vectors 3 – Experimental results c. Comparison with compressed BOV

20. January, 7th 2011Simon Giraudot & Ugo Martin - MoSIG - Machine learning20/ Large-Scale Image Retrieval with Compressed Fisher Vectors Conclusion ● Strong compression ● Decreased memory usage ● Queries simpler and faster ● More efficient than other techniques using compression