Collective Vision: Using Extremely Large Photograph Collections Mark Lenz CameraNet Seminar University of Wisconsin – Madison February 2, 2010 Acknowledgments: These slides combine and modify slides provided by Yantao Zheng et al. (National University of Singapore/Google)

Last Time Distributed Collaboration Google Goggles –Personal object recognition World-Wide Landmark Recognition – Building the engine

Today World-Wide Landmark Recognition – Querying the engine Building Rome in a Day –Distributed matching and reconstruction Discussion

Unsupervised learning of landmark images Geo- clusters Landmarks from tour articles Noisy image pool Visual clustering Premise: photos from landmark should be similar Clustering based on local features Validate and clean models Visual model validates landmarks! Photo v.s. non-photo classifer to filter out noisy images ……

Object matching based on local features Sim( ) = image match score, Image representation Interest points: Laplacian-of-Gaussian (LoG) filter Local feature: Gabor wavelets match score = Probability that match of and is false positive Probability of at least m out of n features match, if Probability of a feature match by chance

Match Region Graph Image matching Node is match region 2 types of edges: match edge: measures match confidence overlap region edge: measures spatial overlapping Agglomerative hierarchical clustering Visual clusters

False detected images Match is technically correct, but match region is not landmark Match is technically false, due to visual similarity A problem of model generation A problem of image feature and matching mechanism For positive images: 337/417 (80.8%) are correct Identification rate: 337/728 (46.3%) For negative images: False acceptance rate: 1.1%

Landmark Recognition All local features indexed in one k-d tree Match region - interest points that contribute to a match between two images

k-d trees k-dimensional binary tree Sub-trees split at median w.r.t one dim Cycle through dimensions Creates “bins” of NNs Indexing local feature for matching Query time: ~0.2 sec in a P4 computer

Landmark Recognition Detect features on query image For each feature in query image – Find NN features using k-d tree NN features link to their model image Score match regions between query and model images

Scoring Match Regions Query image interest points matching points in model image determined through NN search Match score = 1-P FPij (probability match b/w regions is false positive) – P FPij is based on the number of matched points Match threshold = total score > 5

Intuition Query image should have many interest points with matches in match region = high match score Points should have matches in multiple regions (images) - threshold

Building Rome in a Day Use photos from photo-sharing websites to build 3D models of cities Web photos less structured than automated image capture (e.g. aerial) Increased efficiency through distributed computations

Multi-Stage Parallel Matching Matching is distributed across nodes Vocabulary tree-inspired match proposals – For distributed matching Query Expansion to increase cluster density – Match proposals create only sparse clusters

Conclusion Distributed Collaboration Google Goggles –Personal object recognition World-Wide Landmark Recognition Building Rome in a Day –Distributed matching and reconstruction

Thoughts for Discussion Geo-clustering to filter out seldom traveled/photographed sites Match region graph for view comparison Pre-tag landmarks such as exits Augmented reality Distributed matching of features Ad-hoc wireless network range Other thoughts...