1. 1 Unsupervised Modeling and Recognition of Object Categories with Combination of Visual Contents and Geometric Similarity Links Gunhee Kim Christos Faloutsos Martial Hebert Computer Science Carnegie Mellon University October 31, 2008, Vancouver, Canada ACM MIR 2008

2. 2 Outline Problem Statement & Our Approach Word Histogram & Network Construction pLSA and LDA based Models Unsupervised Modeling & Recognition Experiments Discussion

3. 3 Unsupervised Modeling Category discovery + Ranking

4. 4 Recognition Novel Images Bicycle SheepSign Classification + Localization

5. 5 Intuition Combination of Topic contents and Link Analysis Latent Topic: Bicycles Word distributions Same latent Topic Different latent Topic (Sparse and irregular links) link distributions (Dense and consistent links) link distributions [1] Sivic, ICCV 2005 [2] Fei Fei, ICCV 2005

6. 6 Intuition Combination of Topic contents and link analysis Samples of visual words based on Bag-of-Words Samples of links generated by image matching Two types of evidence into a single generative model – Ex. Hierarchical Bayesian Models (pLSA, LDA)

7. 7 Our Previous Work Unsupervised Modeling using Link Analysis Techniques [Kim, CVPR08] Large Scale Network Link Analysis Techniques (ex. PageRank) - Only links - Only modeling → Visual content + Links → Modeling + Recognition

8. 8 Pros over Conventional Models (1/2) Easy Plug-in of geometric information Indirect Formulation: Link generation with geometric consistency + Independent of number of parts [Liu, ICCV 2008] [Lazebnik, CVPR 2006] [Sudderth, ICCV 2005] [Niebles, CVPR 2007] [FeiFei CVPR07 Tutorials]

9. 9 Pros over Conventional Models (2/2) Ambiguity in definition of visual words Word A Word B Word C Semantically similar Different + Relaxed by similarity links between words

10. 10 Outline Problem Statement & Our Approach Word Histogram & Network Construction pLSA and LDA based Models Unsupervised Modeling & Recognition Experiments Discussion

11. 11 Visual Words Histogram Follow Standard Bag-of-Words Approach – Harris Affine + SIFT – Dictionary Formation: K-mean clustering Word ID Freq Word ID Freq Word ID Freq : Freq. of word w in the image j (Weighted by Links)

12. 12 Network Generation Pairwise Image Matching – Spectral Matching [Leordeanu, ICCV 2005]   : Sum of weights of links from image a to image b

13. 13 Outline Problem Statement & Our Approach Word Histogram & Network Construction pLSA and LDA based Models Unsupervised Modeling & Recognition Experiments Discussion

14. 14 pLSA Based Model Standard pLSA [Hofmann NIPS 1999] [Cohn NIPS 2001]

15. 15 LDA Based Model Standard LDA [Blei JMLR 2003] w N  z M   c  L z Linked LDA [Erosheva PNAS 2004]

16. 16 Outline Problem Statement & Our Approach Word Histogram & Network Construction pLSA and LDA based Models Unsupervised Modeling & Recognition Experiments Discussion

17. 17 Unsupervised Modeling (1/2) 1. Category Discovery – Find out class memberships of all training images – pLSA based model – LDA based model

18. 18 Unsupervised Modeling (2/2) 2. Ranking – 1. For recognition, only fixed number of example images to be matched. O(m) → O(K) – 2. Highly probable mis-clustering in low ranked images – pLSA based Model – LDA based Model : Most cited documents in topic i Most matched image in class i

19. 19 New image Recognition Word ID Freq 30* K high ranked images

20. 20 Recognition 3. Classification – Similar formula to unsupervised clustering. – pLSA based Model – LDA based Model 4. Localization – pLSA based Model – LDA based Model [Sivic ICCV 2005]

21. 21 Outline Problem Statement & Our Approach Word Histogram & Network Construction pLSA and LDA based Models Unsupervised Modeling & Recognition Experiments Discussion

22. 22 Learned category accuracy 1.0 0.9 0.8 0.7 0.6 0.5 0.4 0.3 StandardWeighted Linked + Weighted Comparison Tests 5 Objects in Caltech-101 – Similar experimental setup to [Kim CVPR 2008] 55.8% 62.1% 97.8%

23. 23 Unsupervised Category Discovery Experiment Setup – MSRC: 6 Objects (75 training / testing) – PASCAL/ETHZ: 4 objects (40 training / testing) 85.4 % 90.3 % PGPM PC MBMC MD MS MG PP

24. 24 Classification of Unseen Images Experiment Setup – MSRC: 6 Objects (75 training / testing) – PASCAL/ETHZ: 4 objects (40 training / testing) 85.4 % 90.3 % PGPM PC MBMC MD MS MG PP 80.5 % 82.16 %

25. 25 Ranking 1.0 0.8 0.6 0.4 0.2 Learned category accuracy Number of selected examples per object 51015202530 77.5 % < 85.4 % Only Link < Link+ Content

26. 26 Localization PASCAL/ETHZ dataset MSRC dataset MotorbikeCarPeoplesGiraffe BikeCarSheepDoorSign

27. 27 Outline Problem Statement & Our Approach Word Histogram & Network Construction pLSA and LDA based Models Unsupervised Modeling & Recognition Experiments Discussion

28. 28 Conclusion Combination of Topic contents and Link Analysis – Easy Plug-in of geometric information – Relaxation of the ambiguous definition of visual words – Integration between two object recognition approaches Unsupervised Modeling + Recognition Competitive performance

29. 29 Comments? Thank You gunhee@cs.cmu.edu