1. Unsupervised Detection of Regions of Interest Using Iterative Link Analysis Gunhee Kim 1 Antonio Torralba 2 1: SCS, CMU 2: CSAIL, MIT Neural Information Processing Systems 2009 November 30, 2009

2. Unsupervised Detection of ROIs A set of images… Rectangular Regions of Interest

3. Why Is the ROI Detection Useful ? Scene recognition [Quattoni&Torralba, CVPR09] Training for Recognition [Bosch et al, ICCV07] Flickr Notes

4. Alternating Optimization One of widely used heuristics for iterative optimization Optimization over two sets of variables is not easy But affordable to optimize one while the other is fixed

5. Goal: Find correspondences between two sets of point clouds [Besl&McKay,1992] Example – Iterative Closest Point Algorithm Trans- formation Estimate transformation parameters Corres- pondences Associate points by NN criteria

6. Goal: Clustering Example – K-means Cluster Membership Find nearest cluster center Cluster Centers Take mean Initialization Pictures from Bishop’s book

7. Goal: Find best ROIs in each image of dataset Unsupervised Detection of ROIs Refine ROIs Detection or Localization Find Examplars Modeling or Ranking examplars Where is butterfly? What are examplars?

8. Our Approach Inspired by alternating optimization Based on link analysis of hypothesis network. Find Examplars = Central and diverse Hubs Refine ROIs = Highly-ranked Hypotheses in each image wrt examplars Easy, Fast and Dynamic –Simple heuristic for linearity of computation wrt dataset size. –Ex. 4.5 hours / 200k images with naïve matlab implementation.

9. ROI Candidates and Description For each, define –At least one of would be good Description: Spatial pyramids of visual words and HOG Similarity measure: Cosine similarity An image15 segments43 ROI hypotheses Visual wordsEdge Gradient

10. Algorithm - Input Image set and its ROI hypothesis set

11. Algorithm - Initialization Best ROI = Image itself !

12. Algorithm - Initialization Initialization is essential for the success ! Why is it a feasible idea for Web images ? –Most pictures are taken from a canonical view so that an object of interest is located in a center with significant size. –Given a similarity network of a sufficiently large number of images, democratic voting reveals the most dominant visual information as hubs [Kim et al 08] Examples of top-ranked Images

13. Algorithm – First Hub Seeking Generate a similarity network and find a hub set

14. Algorithm – First ROI Refinement Bipartite graph between hub sets and All ROIs of an image

15. Algorithm – Second Hub Seeking Keep iterating…

16. Hub Seeking with Centrality & Diversity Mean-shift like hub seeking algorithm Mean Shift [Comaniciu and Meer, PAMI 2002] K-NN similarity matrixPageRank vector G (t) K-NN graph Degree distribution ~ PageRank vector

17. Hub Seeking with Centrality & Diversity Mean-shift like hub seeking algorithm 0.05 0.2 0.5 0.25 0.8 0.5 0.1 Max P-value ! Fixed radius window = max. reachable probability d (= 0.1) Mean Shift

18. ROI Refinement Augmented Bipartite Graph (1-α)W o WoTWoT αW i ROI hypothesisHub setvector ROI hypotheses Hub set PageRank Argmax () i

19. ROI Refinement What does α do? (1-α)W o WoTWoT αW i α = 0α = 0.1 WoWo WoTWoT

20. Example - ROI Refinement T=0T=1T=2T=3T=4T=5T=6T=7 T=0 T=1 T=2 T=3 T=4 T=5T=6 T=7

21. Scalability Setting Bottleneck: Quadratic computation to generate a similarity matrix of selected ROIs If dataset size is too large, –Run the algorithm with N number of images ( N = 10,000) –Re-use x % of previous high-ranked images. Dataset N N N N

22. Experiments Performance Test –PASCAL VOC 2006 Dataset –Weakly-supervised 1 and Unsupervised 2 Scalability Test –Five objects: {butterfly+insect (69,990), classic+car (265,731), motorcycle+bike (106,590), sunflower (165,235), giraffe+zoo (53,620)} –Weakly-supervised 1 1: Input imageset consists of a single object type (only localization is required) 2: Input imageset consists of multiple object types (localization and clustering are required)

23. Performance Tests Weakly Supervised Localization (PR-Curves) [Russell et al. CVPR 2006] http://www.di.ens.fr/russell/projects/mult seg discovery/index.html X-axis: Recall Y-axis: Precision

24. Performance Tests Unsupervised Classification & Localization X-axis: Recall Y-axis: Precision X-axis: FP rate Y-axis: TP rate ROC Curves PR Curves

25. Scalability Tests Weakly-supervised Localization X-axis: Recall Y-axis: Precision

26. Perturbation Tests Robustness of ROI detection of each image against random network formation –100 random sets of size of 200 images Entropy: 0.24191.68462.4331 Dataset An image of interest X-axis: ROI hypotheses Y-axis: Frequencies

27. Localization Examples

28. Conclusion Alternating optimization based Unsupervised ROI detection Simple and Fast Competitive performance on PASCAL 06 Scalable Test with more than 200K Flickr images Critic: Analysis for convexity, convergence, sensitivity to initialization, quality of solution

29. Algorithm