1 Unsupervised Modeling of Object Categories Using Link Analysis Techniques Gunhee Kim Christos Faloutsos Martial Hebert Gunhee Kim Christos Faloutsos Martial Hebert Computer Science Carnegie Mellon University June 23, 2008, Anchorage, AK

2 Outline Problem Statement & Our Approach Network Construction Link Analysis Techniques – Ranking of features wrt an image/object – Structural Similarity Unsupervised Modeling – Category Discovery – Localization Complexity Conclusion

3 Unsupervised Modeling [1-5] [1] Sivic et al, ICCV 2005 [2] Fritz&Schiele, DAGM 2006 [3] Grauman&Darrell, CVPR 2006 [4] Todorovic&Ahuja, CVPR 2006 [5] Cao&Fei-Fei, ICCV 2007 Category discovery + Localization

4 Previous Work Topic models based on bag of words [1][2][5] [1] Sivic et al, ICCV 2005 [2] Fritz&Schiele, DAGM 2006 [3] Grauman&Darrell, CVPR 2006 [4] Todorovic&Ahuja, CVPR 2006 [5] Cao&Fei-Fei, ICCV 2007 Tree matching [4] Clustering with partial matching [3] w N d z D

5 Intuition Link Analysis Techniques Visual Information Solve Visual tasks A large-scale Network +

6 Statistics of Link Structure

7 Large-Scale Networks WWW [1] Oscars social network [2] Metabolic network [3] Neural network [4] A food web [5] (1): (2): The Academy of Motion Picture Arts and Sciences, (4): (5): Mark ewman

8 Outline Problem Statement & Our Approach Network Construction Link Analysis Techniques – Ranking of features wrt an image/object – Structural Similarity Unsupervised Modeling – Category Discovery – Localization Complexity Conclusion

9 Visual Similarity Network: Vertices Vertices: Any Local Features – Harris Affine + SIFT I1I1 ImIm : Adjacency Matrix of G M nnnn I1I1 ImIm

10 Visual Similarity Network: Edges & Weights Edges: Correspondences by image matching – Spectral Matching [1-2] : Appearance affinity + Geometric Consistency Weights: Stronger geometric consistency, higher values M nnnn IaIa [1][2] Leordeanu & Hebert, ICCV05, ICML06. IaIa IbIb IbIb

11 Outline Problem Statement & Our Approach Network Construction Link Analysis Techniques – Ranking of features wrt an image/object – Structural Similarity Unsupervised Modeling – Category Discovery – Localization Complexity Conclusion

12 1. Ranking of the Features →“models capture the hubs in the visual network” [1] Fergus, Perona, Zisserman, IJCV Fergus et al [1] : “models capture the essence of categories” >>

13 Ranking Removes Noisy Matching

14 How to Rank the Features PageRank [1] – Recursive Definition Vote [1] Brin and Page. WWW 1998

15 Rationale: Why Ranking Works? Consistent Matching Highly varient Matching Hub Outlier

16 2. Structural Similarity Similar vertices → Similar Link structures Node i Node j + (3) similarity of matching behaviors (1) Appearance Similarity + (2) Geometric consistency

17 How to Mine Structural Similarity Automatic Extraction of Synonyms [1] [1] Blondel et al. SIAM review Node i Node j A vertex structural similarity matrix Z nnnn uv : If v appears in the definition of u

18 Outline Problem Statement & Our Approach Network Construction Link Analysis Techniques – Ranking of features wrt an image/object – Structural Similarity Unsupervised Modeling – Category Discovery – Localization Complexity Conclusion

19 Compute Ranking wrt Each Image IaIa PageRank: P-vector for I a n1n1 I1I1 I2I2 ImIm

20 IaIa Less correlated Meaning of Ranking P-vector for I a Ranked importance of the other features wrt image a Ranked importance of features in image a Valuable for Category discovery IaIa IbIb IcIc Highly correlated n1n1

21 Image Affinity Matrix Vertex structural similarity matrix Z I1I1 I2I2 ImIm n1n1 nnnn m PageRank vectors mmmm Image affinity matrix A n >> m (Ex. 1mil >> 600)

22 Category Discovery by Clustering 600 images of 6 Object Classes of Catech-101 k -NN graph [1] Normalized spectral Clustering [2] [1] Luxburg. Statistics and Computing, 2007 [2] Shi & Malik, PAMI 2000 k = 10log(m)

23 Results of Category Discovery TUD/ETHZ dataset – Experimental Setup follows [1] – 75 images per object, 10 repetition 95.47% MotorbikesCarsGiraffes Motorbikes 93.3   2.7 Cars 4.8   Giraffes2.0    1.4 [1] Grauman & Darrell, CVPR 2006

24 Results of Category Discovery Caltech-101 Object classes (100 per object) ACFMW A C F M W obj: 98.55% [1] Grauman & Darrell, CVPR 2006 [2] Sivic et al, ICCV obj: 97.30% 6 obj: 95.42% > [2]: 98%, [1]: 86% A C F M W K ACFMWK A C F M W K ACFM A C F M

25 Compute Ranking wrt a Category P-vector for category C 1 PageRank: n c1  1 n c2  1 n c3  1 P-vector for category C 2 P-vector for category C 3

26 Meaning of Ranking Valuable for Localization Ranked importance of each feature wrt its category IaIa P-vector for a giraffe class nc1nc1

27 Localization – Confidence Values n c1  1n c1  n c1 n c2  1n c2  n c2 n c3  1n c3  n c3 P-vectors and Vertex similarity matrix for each category

28 Examples of Localization

29 Quantitative Results of Localization False Positives [1] Quack et al. ICCV 2007

30 Outline Problem Statement & Our Approach Network Construction Link Analysis Techniques – Ranking of features wrt an image/object – Structural Similarity Unsupervised Modeling – Category Discovery – Localization Complexity Conclusion

31 Complexity Issues The VSN representation is Sparsity of the network – Power iterations for sparse matrices: Scale-free network Ex. 6 objects of Caltech-101  900K nodes Degrees of Vertices Percentage of vertices

32 Outline Problem Statement & Our Approach Network Construction Link Analysis Techniques – Ranking of features wrt an image/object – Structural Similarity Unsupervised Modeling – Category Discovery – Localization Complexity Conclusion

33 Conclusion A new formulation of unsupervised modeling – Statistics of the link structure – Finding communities (categories) and hubs (class representative visual information) Link analysis techniques Competitive performance Future directions – Statistical framework – Scalability

34 Comments? Thank You

35 Supplementary Material If any questions and comments, please send me an at

36 Spectral Matching [1-2] [1] M. Leordeanu and M. Hebert. A spectral technique for correspondence problems using pairwise constraints, ICCV. [2] M. Leordeanu and M. Hebert. Efficient map approximation for dense energy functions, ICML. OK Pairs of wrong correspondences are unlikely to preserve geometry Pairs of correct correspondences are very likely to preserve geometry

37 Weights of Edges 1. Stronger geometric consistency, higher weights matches / 50 features > 10 matches / 100 features C ij > 0.8C ij > 0.7C ij > 0.6C ij > 0.5C ij > 0.4 w ij = w ij = w ij = w ij = w ij =

38 Example of Vertex Similarity [1] Similarity between Vertices of Directed Graphs – Only based on link structures [1] Blondel et al. SIAM review 2004.

39 Ranking for Category Discovery Relative Importance wrt each image O O O O M MaMa - Only consider the relations between Ia and the others IaIa IaIa - Why? To avoid Topic Drift PageRank: IaIa x x P-vector for I a

40 Computation of Relative Importance IaIa IaIa M Before “Category Discovery” O O O O Why? Topic Drift PageRank for I a

41 Relative Importance for Modeling IaIa IaIa M Before “Category Discovery” O O O O Wait ! In General, majority of images are from different object classes. What if the result is distracted by them? PageRank: Recursive Definition!! IaIa The vote by the same object will be much more appreciated ! IbIb IcIc

42 Localization Relative Importance wrt each category M P-vector for each category Mc1Mc1 Mc2Mc2 Mc3Mc3 PageRank:

43 Computation of Relative Importance O O P-vector After “Category Discovery” object category k M Valuable for Localization Ranked importance of features wrt each object category

44 Toy Example: Relative Importance Matching behavior – Consistent between important features in the same class, Highly variant between backgrounds and different objects Matching

45 Toy Example: Relative Importance Image 2Image 1 – Consistent between important features in the same class, Highly variant between backgrounds and different objects

46 Toy Example: Relative Importance – Consistent between important features in the same class, Highly variant between backgrounds and different objects Image 3Image 1

47 Toy Example: Relative Importance – Consistent between important features in the same class, Highly variant between backgrounds and different objects Image 4Image 1 Consistent Highly variant

48 Unsupervised Modeling Category Discovery + Localization IaIa  I1I1 IaIa Ranked importance of the features in I 1 wrt I a P-vector P a Affinity of I 1 to I a This value is not used ! A vertex similarity matrix Z  I1I1 ++ A(a,1) ImIm

49 Localization Category Discovery + Localization O O P-vector object category k M object category 1 O O Z P-vector P c aiai ++ ZcZc aiai   =0.8 P-vector P c