Yan-Tao Zheng1, Ming Zhao2, Yang Song2, Hartwig Adam2 Ulrich Buddemeier2, Alessandro Bissacco2, Fernando Brucher2 Tat-Seng Chua1, and Hartmut Neven2 1 NUS Graduate Sch. for Integrative Sciences and Engineering, National University of Singapore, Singapore 2 Google Inc., U.S.A T OUR THE W ORLD : BUILDING A WEB - SCALE AND LANDMARK RECOGNITION ENGINE ICCV 2009

OUTLINE  Introduction  Approach ( Framework)  Experiments  Conclusion (Future Work)

I NTRODUCTION What is the motivation ? With the vast amount of landmark multimedia data on the web

I NTRODUCTION  Application  Provide clean landmark images for building virtual tourism of a large number of landmarks  Facilitate both content understanding and geo-location detection of images and video  Provide tour guide recommendation and visualization

 Issue must be tackled  No readily available list of landmarks in the world Explore two source : (1) geographically calibrated images in photo sharing websites (2)travel guide articles from websites  Even if, it’s still challenging to collect true landmark image Download landmark images from two sources: (1)photo sharing websites(2)Google Image Search  Efficiency is a challenge for a large-scale system Accomplish by three means: (1)parallel computing (2)efficient clustering algo. (3)efficient image matching by k-d tree indexing I NTRODUCTION

A PPROACH FRAMEWORK

Learning landmarks from GPS-tagged photos >> Perform the agglomerative hierarchical clustering on the photo’s GPS coordinates >> Validation criterion is unique number of authors of photos is larger than a threshold

Set of images A PPROACH FRAMEWORK Learning landmarks from travel guide articles >> with the hierarchy, we can extract city names from country in six continents >> satisfy following criteria, text is deemed to be a landmark candidate

Set of images A PPROACH FRAMEWORK Learning landmarks from travel guide articles >> Validating landmarks (1) if it is too long or most of its words are not capitalized (2) the number of unique authors of images in the cluster >> which reflects the popular appeal of landmarks

A PPROACH DISCOVER LANDMARKS IN THE WORLD Most of users are located in Europe and North America !!

A PPROACH LEARNING OF LANDMARK IMAGES Object matching based on local features  Detect interest point >> LoG filters [11]  Local descriptor >> SIFT [9]  Reduce the feature dimensionality to 40 >> Principle Component Analysis (PCA) [2]  The match interest points of two images are verified >> affine transformation [9]

A PPROACH MATCH SCORE  Match score which is the probability of a false positive Can be estimated by Bayes Theorem [2] By using a cumulative binomial distribution

A PPROACH MATCH REGION Classified into two types: match edge and region overlap edge ---- Match edge ---- Region overlap edge

A PPROACH G RAPH CLUSTERING Do not have a priori knowledge of the # of clusters >> k-means are unsuitable >> exploit the hierarchical agglomerative clustering [2] The distance of region

A PPROACH C LEANING VISUAL MODEL  Photographic v.s non-photographic image classifier  Based on Adaboost algorithm over low level visual features of color histogram and hough transform.  Adopt a multi-view face detector[15]

A PPROACH EFFICIENCY ISSUES  Make efficiency essential in two aspects: (1) the landmark image mining (2) landmark recognition of query images  Achieve efficiency in three measures:  Parallel computing to mine true landmark images  Efficiency in hierarchical clustering  Indexing local feature for matching Use k-d tree[1] ~0.2 sec in a P4 computer

E XPERIMENTS Validated Landmark CitiesCountries GPS-tag photos Tour guide corpus landmarks are found to be common in both lists >> land mark is a perceptional and cognitive concept

E XPERIMENTS  Evaluation of landmark image mining  1000 visual clusters are randomly selected 68 of them are found to be negative outliers (0.68%)  The classifier is trained based on ~5000 photographic and non-photographic images, while the face detector is base on [15]  After cleaning, cluster rate drops to 0.37%

E XPERIMENTS Positive testing 728 images from 124 randomly selected landmarks Negative testing Caltech-256 [5] Pascal VOC 07 [3] Evaluation of landmark recognitioin

E XPERIMENTS Recognition : local feature matching of query image against model images, NN principle A match is found when the match score is larger than the threshold Recognition accuracy: 80.8% fairly satisfactory Image content analysis and geo-location detection: 46.3% moderately satisfactory

C ONCLUSION FUTURE WORK  Conclusion  Build a world-scale landmark recognition engine  Utilize ~21.4M images to build up landmark visual model  Incorporates 5312 landmarks from 1259 cities in 144 countries  Future work  Multi-lingual aspect of landmark engine >> help to discover more landmarks and collect more clean landmark images in their native languages in the Internet

R ELATED WORK Thank You !! Related Work 3D visualization of landmarks