1. Neurocomputing,Neurocomputing, 20122012 Haojie Li Jinhui Tang Yi Wang Bin Liu School of Software, Dalian University of Technology School of Computer Science, Nanjing University of Science and Technology Looking into the world on Google Maps with view direction estimated photos

2. Introduction Google Maps (satellite Layer)

3. Introduction A: current system B:ViewFocus

4. Overview 1. The system first collects a set of photos which fall into the range of 100-meter radius centered at the region. 2. The photos’ view directions are estimated and registered to the map. 3. Finally only photos that are pointing to the region are returned to the users for exploration.

5. Framework

6. SIFT Features SIFT (Scale-invariant feature transform ) ---Transforms an image into a large collection of feature vectors (Gaussian filter) Has good performance in the matching of building dominant and landmark.

7. Photo index 1.Randomly select 6000 SIFT descriptors out of the features extracted from the collected photos 2.Perform k-means on them to build a dictionary of 500 visual words. Video Google: A Text Retrieval Approach to Object Matching in Videos

8. Photo index 3.Visual words are binned using a 2 level pyramid. For retrieval, histogram intersection is used. D: Dimension H Px : Histogram of photo Px L : 2^l cells along each dimension H px (l): numbers of points from Pi and Pj that fall into the lth cell of the grid.

9. Photo matching We select top k=30 nearest neighbors. We can divide the global map into small-size grids and index the photos in each grid separately.

10. Camera pose recovery Using eight-point algorithm to estimate fundamental matrix. 1. For each pair of photos consists of p and p’ which have not been estimated before elsewhere. 2. Determine the correspondences set C by SIFT features matching between p and p’. 3. Randomly choose eight correspondences and compute an initial fundamental matrix F using normalized 8-point algorithm, apply RANSAC to detect outliers, and determine inlier rate induced from current F, if the appropriate inlier rate is achieved, the current F is the robust one, else repeat step3. the F can be written as

11. Camera pose recovery T: Translation vector R: 3x3 rotation matrix Angles: C  = cos  S  = sin  yaw  pitch  roll    

12. Camera pose recovery K describes the cameras intrinsics. f and f’ are the focal length of p and p’ F is parameterized by six parameters, which can be solved by fulfilling the Epipolar( 極限幾何 ) Condition. P’ hom and P hom are noise free homogenous feature points.

13. Camera pose recovery 4.Extract rotation, translation the focal length from F Using a pinhole camera model, the 3D view direction Vp of the photo p can be obtained as follows: V p = R’ * [0 0 -1]’ Where ‘ indicates the transpose of a matrix or vector. We retain the x and y components of V p as the 2D view direction V p of p.

14. View direction geo-registration Suppose there is a total of m photos matched with p, we compute m view directions for p from these photos and create a m x 2 matrix M. Each M is an estimated view direction of p. Then apply RANSAC analysis to M to select inliers as the correct view directions. At last, the average of the inliers is computed as the final view direction of p on the map.

15. System interface Explore places: --The system allows users to select places/regions they are interested in, then it automatically return a set of precise photos related to the target places to users for their focused exploration, by filtering out photos that are pointing other directions. Virtual walking: --when you are using the Google Maps, you can walk and look freely on the map. Sometimes you may want to view what you can see from the your position to a particular viewing angle.

16. System interface

17. Result