1. Inferring Temporal Order of Images from 3D Structure
Grant Schindler Gatech
Frank Dellaert Gatech
Sing Bing Kang MSR, Redmond

2. Outline Problem Definition Algorithm Overview Applications
Things to think about

3. What can be done with n images?

4. What can be done with n images?
Feature Extraction
Correspondence
Structure from Motion
What Now?

5. Temporal Ordering and 4D Walkthrough
1920
1951
1966
2003

6. Outline Problem Definition Algorithm Overview Applications
Things to think about

7. SFM tells us: F1 F2 C1 I1 F3 I2 C2 Camera Matrices
3D points for features
Visibility of 3D points in images

8. C2 F3 C1 I1 I2 F1 F2
SFM info
Image 1 (I1)
Image 2 (I2) F1
Visible
??? F2 F3

9. C2 F3 C1 I1 I2 F1 F2
SFM info
Image 1 (I1)
Image 2 (I2) F1
Visible
??? F2
Occluded F3
Out of View

10. Notion of missing at that timeC2 F3 C1 I1 I2 F1 F2
Notion of missing at that time
SFM info
Image 1 (I1)
Image 2 (I2) F1
Visible
??? F2
Occluded F3
Out of View

11. Classification of 3D point for an Image
Visible – SFM tells us
Out of View – Camera Matrix tells us
Missing / Occluded - ???
for an occluded point, there must be an occluder

12. Point ‘F’ Missing / Occluded ?
Find out occluders
3D Triangulation of all visible points
No occluder should occlude a visible point
Visibility check for F
occluders F1 F2
Camera centre
occluded
missing

13. Visibility Matrix Sij € {visible, occluded, missing, out of view } I1
... In F1
S11
S12 …
S1n F2
S21
S22
S2n Fm
Sm1
Sm2
Smn
Sij € {visible, occluded, missing, out of view }

14. Constraints of Visibility Matrix

15. Combinatorial Algorithm to find Best Configuration
Local search method
Starts at a random configuration
Small moves which reduce the number of constraints violated

16. Issues leading to Finding Approximate Solution
Problems in feature detection
Mislabeling of points
Triangulation strategy
Inaccuracy in SFM
Features occluded by undetected occluders (fog, trees etc)

17. Structural Segmentation from Temporal Ordering
Clustering temporally coherent features
Separate triangulation of each cluster
Texture by projecting on images

18. Algorithm Overview

19. Possible Applications
Historic Preservation
Virtual Tourism
Urban Planning
Spatio-Temporal models as a new way to interact with a vast collection of imagery

20. Things to Think about Feature extraction (done manually here)
Better methods for finding occluders – problems with triangulation method
Very coarse structure
Can have triangles for no occluders
Using Goesele’s work (ICCV 2007) for structural segmentation
High number of images required (this paper used images)
Validation
Correspondence between the best ground truth solution and best approximate solution of ordering
Increasing the scale technically and physically

21. An Interesting Insight….
Assume no building can be demolished once it’s built
Assume every image is a node of graph
Edge from A to B if A precedes B
(B has visible features missing in A )
Directed Graph (acyclic in ideal case)

22. B1 B2 B3 A B C
Input Images C A B
Directed Graph (Acyclic in ideal case)

23. B1 B2 B3 A B C
Input Images C A B
Directed Graph (Acyclic in ideal case) B C A
Topological Sort
Solution !

24. More insights about Graph Model
Every edge has a confidence value based on quality of features and SFM procedure
In general, there can be back edges in this graph
Problem to find the best topological sort maximizing the confidence measure

25. Graph Complexity Increases with more constraints
Modeling constraints involving more than 2 images at a time - how??