1. Capturing, Processing and Experiencing Indian Monuments
BTP Presentation
Dr. C.V. Jawahar
Syed Ahsan Ishtiaque
Kumar Srijan

2. Experiencing a monument
For instance, Golkonda Fort
Many Photographs are taken
How to visualize such a large collection of images.
Allign Text box
Send it to back.
Can these images be visualized it in a better way?
Experiencing a monument
Photographs are taken

3. Previous and Related Work
Technologies for visualizing image collection in 3D exist
Microsoft’s Photosynth
It uses Bundle Adjustment step.
Computes 3D and refines results based on the errors
Highly Iterative
Computationally intensive
Not Scalable: Due to high computational complexity, unable to deal with large image datasets
Not Incremental: Adding a new image needs the whole computation to be done again from the scratch.

4. Improvements We present an approach which is
Incremental
Deals with updation or addition of a new image
Scalable
Works on large image datasets
We are trying to eliminate the need of bundle adjustment, or limit it to smaller dataset.
Achieved by introduction of Matching graph, and dividing it into subgraphs.
Change the title

5. Solution Overview Creating a Matching Graph
Compute SIFT features in the images
Cluster features into Visual Words
Presence of similar visual words indicate similarity
Pose and orientation estimation
Essential matrix(E) between 2 images is computed.
E is decomposed to get R and t between images.
Visualization
Place image in a 3d world according to R and t.
Project neighbouring images onto the current image plane.
For transition synthesize proper intermediate views.

6. Incremental and Scalability
The matching graph makes the process scalable.
Graph is further decomposed into subgraphs.
Computations are performed on these subgraphs.
Relationship between subgraphs is computed using common images.
New image is added incrementally
Compute neighbouring images
Add to the matching graph
Modify the subgraphs which got affected.

7. Generating Matching Graphs
Find images which are geometrically close to each other
Given a set of images
Repeat for all the images
Place an edge between the two
Edge represents that images are geometrically close.
Store SIFT features
Cluster them into K Clusters, in 128 dimensional space – K visual Words - vocabulary
For each SIFT feature, Find its nearest cluster – and represent that feature in terms of that Visual Word.
So, an image can be represented a K-Dimensional vector – i.e. in terms of Bag of Visual Words
Define weights of each Visual Word using tf-idf
Create inverted index for fast lookup
Use weights and check what all images have same VW’s – Define some similarity measure, if > threshold. Place edge temporarily
Spatial Verification – For every edge in graph, estimate best fit F matrix using RANSAC, check for inliers and outliers. If inliers < “Specefic Value” .. Discard the edge.

8. Example*
*Manually Constructed Graph
*Manually Constructed Graph

9. Pose and Orientation Estimation
SIFT Features are detected.
For an edge in Matching graph
Fundamental and Essential Matrix are estimated.
Features are matched using RANSAC, and spurious matches are eliminated x’ F x
Spurious Matches
Correct Matches
Essential Matrix is decomposed to obtain R and T between the two cameras

10. Relative Position and Orientation between two cameras
Example
Relative Position and Orientation between two cameras

11. Visualization Known pose and orientation Two cases exist
Viewing an Image Ci
Translating from Ci to Cj
Plane(Ci) Ci Ci Cj t
1-t

12. Example Video: Gate Dataset, Golkonda.
Video: Way to Hill top, extracted from Photosynth.
Video: Way to Hill top, from our browser.

13. Conclusion
Matching graph computation serves as most important step in the entire process
An offline process, consumes time at start.
Growing this graph is efficient which makes the process incremental.
Computations on subgraph and computing relationships between subgraphs make the process scalable.
Future Work
Automatic construction of Matching graph
Currently done manually
Visualization can be improved with more features.

14. Thank You