IIIT Hyderabad Geometry Directed Browser for Personal Photographs Center for Visual Information Technology IIIT Hyderabad Aditya Deshpande, Siddharth Choudhary, P J Narayanan, Kaustav Kundu, Krishna Kumar Singh, Aditya Singh, Apurva Kumar

IIIT Hyderabad We use SfM and other 3D computer vision techniques to provide intuitive Geometry Directed Photo Browsing. Photo-Browsing Digital Photography - No hard copy - Capture photographs and relive later on display device Photo-Browsers are tools to view digital photographs. E.g. Windows Photo Viewer, iPhoto, FSpot, KSquirrel etc. Photo Browsing model has not evolved much.

IIIT Hyderabad Related Work Face Detection & Tagging on Social Networking Sites. [Zhang et al. MM03], Automatic annotation of family albums. [Davis et al. MM05], Additional contextual data viz. time of capture, geo-tag, indoor/outdoor scene, co-occurring faces. Above techniques only improve photo-browsing experience of social engagements.

IIIT Hyderabad Our Goal Apart from social engagements, a large chunk of users personal photographs consist of tourist places & monuments. [Snavely et al. IJCV08, SIGGRAPH06] (Photosynth) - CPC Storage, local reconstruction to add new cameras Choudhary et al., Li et al., Sattler et al., Irschara et al. etc. - Localize new query images w/o exhaustive search. We combine SfM-Reconstruction + Localization to provide intuitive browsing of user photos in 3D space of the monument.

IIIT Hyderabad Assumptions Our target platform is an off-the-shelf laptop or a desktop. User is expected to click around 5-50 photographs for a particular monument. The system should localize these user photographs in a reasonable time. The system should provide a smooth visualization / transitions of all user photos and ~10 5 points of the monument.

IIIT Hyderabad System Design (1) Heavy SfM Reconstruction done offline in the cloud (2) GDBPackage : reconstruction + addnl. information downloaded to local disk (3) User uploads personal photos through a camera / phone (4) System registers users photos to the point cloud and provides 3D visualization.

IIIT Hyderabad System Block Diagram GDBPackage User Photos Registration Module Visualization Module System is divided in two parts : 1. Registration / Localization Module 2. Visualization Module 2 1 Estimated Cameras

IIIT Hyderabad Localizing User Photos Trivial if photograph is taken from GPS enabled device and is geo-tagged! What if no geo-tag information? Two Localization Approaches : Image based search in a geo-tagged Image Dataset [Panda et al.] Geo-locate digital heritage site photos. Using structure information in SfM Dataset [Irschara et al. CVPR09], match to nearby similar images. [Li et al. ECCV10], visibility prioritized 3D-2D matches. [Sattler et al. ICCV11, ECCV12], visual words to find 2D-3D matches.

IIIT Hyderabad Localization - Choudhary et al. [Choudhary et al. ECCV12] - Triangulate a seed point in the user photograph. - Further 3D-2D search is guided by visibility probabilities. - Find ~20 independent matches. - Use RANSAC to estimate camera parameters. Probability Guided 3D-2D correspondence 3D Position Up Vector View Direction

IIIT Hyderabad Advantages of Localization Method Data for Localization is stored in GDBPackage : (1) Cover Set (2) Visibility Matrix (3) Bi-Partite Visibility Graph CPC images need not be stored, data requirements are minimal. The method is fast and localizes images at the rate of 1sec/photo.

IIIT Hyderabad Non-Localizable Photographs In some cases the images lack sufficient monument geometry for localization to work : - Occluded by people. - Noisy images of nearby scenery/smaller monuments. - Zoomed in images of smaller monument structures etc. Zoomed In View of Small Structure (Pantheon Dataset) Completely Occluded by People (Colosseum Dataset)

IIIT Hyderabad Non-Localizable Photographs Photographs have time of capture stored in their EXIF-tags. A non-localized image is placed at a position that is weighted average of its immediate known predecessor and immediate known successor in time. Similarly, linear interpolation is also done for the view-direction vector to get the complete camera pose. The above method will not give the exact location, but placing it in temporal neighborhood suffices for display purposes.

IIIT Hyderabad Visualization Module 3D Viewer Mouse Navigation Button Navigation Add Screenshot Delete Path Generate Photo- Tour 2D Viewer

IIIT Hyderabad 3D Photo Browser : Geometry Directed Photo-Browsing Initial Mode : 3D Model and small preview (thumbnails) of user photographs. Select Mode : Animate to clicked photo and detailed view. Linear quaternion interpolation of Rotation Matrix for smooth transitions between images. Smooth transitions give a feel of the geometric space of the monument.

IIIT Hyderabad 3D Photo Browser : Generating Custom Photo Tours User can save the current viewpoint (Add Screenshots) Once a set of viewpoints are saved, he can smoothly animate over viewpoints. (Generate Photo-Tour / Animate Path) User can delete the viewpoints and generate a new photo-tour. Photo-Tours are a good way to creatively view personal photos taken at a tourist place.

IIIT Hyderabad Results Monument# Photos# Registered Photos Reg. Time (secs per photo) Colosseum Colosseum Pantheon Stone Chariot (Hampi) (a) Localization Module (b) Visualization Module

IIIT Hyderabad Conclusion and Future Work Minimal System Requirements. Intuitive 3D Visualization of User Photographs. Pipeline for 3D personal photo-viewing from SfM reconstruction. Port our system to a mobile phone and have a touch/gesture interface. 3D Photo-Viewing & Localization App

IIIT Hyderabad Thank you. Questions? More Results (a) Hampi Dataset (Stone Chariot) (b) Pantheon Dataset

IIIT Hyderabad Platform Details ItemSpecification CPUIntel ® CORE i5 Clock Speed2.44GHZ RAM4GB GPUIntel ® HD Graphics Accelerator