Inferring Temporal Order of Images from 3D Structure Grant Schindler Gatech Frank Dellaert Gatech Sing Bing Kang MSR, Redmond

Outline Problem Definition Algorithm Overview Applications Things to think about

What can be done with n images?

What can be done with n images? Feature Extraction Correspondence Structure from Motion What Now?

Temporal Ordering and 4D Walkthrough 1920 1951 1966 2003

Outline Problem Definition Algorithm Overview Applications Things to think about

SFM tells us: F1 F2 C1 I1 F3 I2 C2 Camera Matrices 3D points for features Visibility of 3D points in images

C2 F3 C1 I1 I2 F1 F2 SFM info Image 1 (I1) Image 2 (I2) F1 Visible ??? F2 F3

C2 F3 C1 I1 I2 F1 F2 SFM info Image 1 (I1) Image 2 (I2) F1 Visible ??? F2 Occluded F3 Out of View

Notion of missing at that time C2 F3 C1 I1 I2 F1 F2 Notion of missing at that time SFM info Image 1 (I1) Image 2 (I2) F1 Visible ??? F2 Occluded F3 Out of View

Classification of 3D point for an Image Visible – SFM tells us Out of View – Camera Matrix tells us Missing / Occluded - ??? for an occluded point, there must be an occluder

Point ‘F’ Missing / Occluded ? Find out occluders 3D Triangulation of all visible points No occluder should occlude a visible point Visibility check for F occluders F1 F2 Camera centre occluded missing

Visibility Matrix Sij € {visible, occluded, missing, out of view } I1 ... In F1 S11 S12 … S1n F2 S21 S22 S2n Fm Sm1 Sm2 Smn Sij € {visible, occluded, missing, out of view }

Constraints of Visibility Matrix

Combinatorial Algorithm to find Best Configuration Local search method Starts at a random configuration Small moves which reduce the number of constraints violated

Issues leading to Finding Approximate Solution Problems in feature detection Mislabeling of points Triangulation strategy Inaccuracy in SFM Features occluded by undetected occluders (fog, trees etc)

Structural Segmentation from Temporal Ordering Clustering temporally coherent features Separate triangulation of each cluster Texture by projecting on images

Algorithm Overview

Possible Applications Historic Preservation Virtual Tourism Urban Planning Spatio-Temporal models as a new way to interact with a vast collection of imagery

Things to Think about Feature extraction (done manually here) Better methods for finding occluders – problems with triangulation method Very coarse structure Can have triangles for no occluders Using Goesele’s work (ICCV 2007) for structural segmentation High number of images required (this paper used 20-30 images) Validation Correspondence between the best ground truth solution and best approximate solution of ordering Increasing the scale technically and physically

An Interesting Insight…. Assume no building can be demolished once it’s built Assume every image is a node of graph Edge from A to B if A precedes B (B has visible features missing in A ) Directed Graph (acyclic in ideal case)

B1 B2 B3 A B C Input Images C A B Directed Graph (Acyclic in ideal case)

B1 B2 B3 A B C Input Images C A B Directed Graph (Acyclic in ideal case) B C A Topological Sort Solution !

More insights about Graph Model Every edge has a confidence value based on quality of features and SFM procedure In general, there can be back edges in this graph Problem to find the best topological sort maximizing the confidence measure

Graph Complexity Increases with more constraints Modeling constraints involving more than 2 images at a time - how??