Reconstructing Camera Positions from Visibility Information Elif Tosun Honors Thesis Presentation Advisor: Ileana Streinu May 1 st, 2002

Outline Motivation General Problem Some Definitions Main Problems of Interest Background in Rigidity Solutions in 2D Further Work Summary

Motivation: MIT City Scanning Project Goal: To reconstruct a 3D model of an urban environment using numerous images.

Method: –Robot with a mounted hemispherical camera –Images acquired at different *unknown* positions of an urban environment –Images put together to get “image feature points” using methods in Computer Vision and Computer Graphics. Motivation: MIT City Project A corner may be an image feature point

Method (Cont’d): –Slopes between cameras and image features are inferred from use of hemispherical cameras with known local orientations. –Once the set of slopes are obtained, solution for camera and image feature positions is based on solving a linear system of equations

However… A linear system of equations may be over or under determined. This information is NOT known a priori Therefore, system *sometimes* may fail to give a solution. A SUBSET of edges that give a unique solution is needed. “… determination of the minimal set of distinct pairs needed for a non-degenerate solution is significantly more complicated and depends entirely on the topology of the adjacency graph. ” [Antone-Teller, 2000]

The Problem Given: Visibility information as a set of slopes at n positions of a hemispherical camera Find: Absolute coordinates of camera locations and positions of image features Why? May be used to reconstruct a model of environment. Problem? Over- or under-determined system of equations. Need a subset.

This thesis focuses on: Reformulating this problem and modeling it based on its geometric/combinatorial aspects Exploring it in 2 and 3 dimensions Isolating several feasible sub problems and giving algorithmic solutions Providing implementations for some

General Camera Registration Problem Formulation using “direction networks” Data set: 3 cameras,4 image feature points, slope known for 13 pairs Direction Network: 7 (=3+4) vertices 13 edges

Direction Network? A direction network is a graph G=(V, E) together with slope information for each edge: D = (V, E, S) with S = { s e : s e is the slope information for an edge e  E }

General Camera Registration Problem Given a direction network D and the location of a subset of vertices V’ of V, find the location of all vertices v  V of G. Notes: 1. We don’t treat image features and camera positions differently. Both sets together form the set of vertices V of G. 2. Location of a subset of vertices is needed to fix a frame of reference.

Definitions Realization of a direction network: A geometric embedding of G of D. –i.e. set of points P s.t. if ( ij )  E then the slope between points p i and p j is s ij

Definitions Realization of a direction network: A geometric embedding of G of D. –i.e. set of points P s.t. if ( ij )  E then the slope between points p i and p j is s ij x-axis y-axis (0,0)

Definitions Mixed “direction and length” network: A direction network of the following form: MD = (V, v 1, E 1, E 2, S, L) where: E 1 = set of edges for which slopes are given E 2 = set of edges for which lengths are given S = { s e : s e is the slope for edge e  E 1 } L = { l e : l e is the length for edge e  E 2 } v 1 = a unique vertex that is pinned down (at origin)

How to solve for an MD system To solve for a mixed “direction and length” network… A) Associate a linear equation to each slope information… i.e. for each slope s ij between points p i =(x i, y i ) and p j =(x j, y j ) : (y j -y i ) – s ij (x j -x i ) = 0

B) And associate a quadratic equation to each length information i.e. for each length l ij between points p i =(x i, y i ) and p j =(x j, y j ) : (x j -x i ) 2 + (y i -y j ) 2 = l ij 2

A particular case… The particular case of an MD that we will be considering is the following: There exists an edge e, s.t. –both the length ( l e ) and the slope information ( s e ) is known –one end point of e is pinned down – e is the only edge in E 2 i.e. it’s as if two vertices are pinned down: lele sese So, we’re only dealing with a linear system of equations

More Definitions Tight direction network: A direction network where there is a unique solution to the system.

Loose direction network: A direction network where the system is under- determined.

It has a parallel redrawing!

Over-determined direction network: A direction network where there are more edges than needed to have a unique solution:

Non-realizable direction network: A direction network with no solution where the system is over-determined and the constraints are incompatible

Main Problems Problem 1: Given a direction network, decide whether it’s tight or not. Problem 2: Given a loose direction network, find a set of edges, that needs to be added, so that it becomes tight. - if possible

Problem 3: Given an over-determined direction network: –Find a set of edges that need to be removed so that it becomes tight – if possible –Find the set of edges to be removed to get ‘the best’ tight framework. –Get an approximation of ‘the best’ tight framework.

Why “if possible” for Problems 2 and 3: –The data set might not have the edges necessary for tightness –The given direction network might be partially over- or under-determined All induced sub-graphs are under-determined

Why “if possible” for Problems 2 and 3: –The data set might not have the edges necessary for tightness –The given direction network might be partially over- or under-determined All induced sub-graphs are under-determined Under-determined sub-graph No edge removal can make it tight

Why “if possible” for Problems 2 and 3: –The data set might not have the edges necessary for tightness –The given direction network might be partially over- or under-determined All induced sub-graphs are under-determined Over-determined sub-graph No edge addition can make it tight

Background in Rigidity Geometric Rigidity 2 dual categories: –Fixed Edge Lengths –Fixed Directions (“parallel redrawing”) Duality in 2D: Properties of one hold for the other as well!

Fixed Length Rigidity: Definitions A framework is a graph G=(V, E) together with length information for each edge: F = (V, E, L) with L = { l e : l e is the length information for an edge e  E } lele

A minimally rigid framework A flexible framework (infinitesimal rigidity – velocities) Fixed Length Rigidity: Definitions

A minimally rigid framework A flexible framework (infinitesimal rigidity – velocities)

Fixed Length Rigidity: Definitions An over-braced framework An unrealizable framework

Fixed Length vs. Fixed Direction Framework Minimally Rigid Flexible Direction Network Tight Loose

Fixed Length vs. Fixed Direction Over-braced Unrealizable Over-determined Unrealizable

Fixed Length vs. Fixed Direction Find a non-trivial set of velocities Rigidity Matrix ( M ) ( 2n x m ). Info on derived from point coordinates Find point coordinates Parallel Redrawing Matrix ( M ) ( 2n x m ). Info on slopes

Fixed Length vs. Fixed Direction Solution: Mv = b Why? (fixed rods, perpendicular velocity vectors) ( p i -p j ).( v i -v j ) = 0 Solution: Mv = b Why? (line equations) ( x i -x j ) m ij –( y i -y j ) = 0 V = column vector of unknowns, b = column vector of 0 s. m ij pipi pjpj

Fixed Length vs. Fixed Direction Solution: Mv = b Why? (fixed rods, perp. velocity) ( p i -p j ).( v i -v j ) = 0 Solution: Mv = b Why? (line equations) ( x i -x j ) m ij –( y i -y j ) = 0 V = column vector of unknowns, b = column vector of 0s. mij pi pj

Rigidity in the Plane: Combinatorial Rigidity Based on underlying graph, no numeric data Laman’s Theorem: Given a minimally rigid graph G=(V, E) with n vertices, m edges: (i) m = 2n-3 (ii) For every sub graph G’ of k vertices has <= 2k-3 edges

Rigidity in the Plane: Combinatorial Rigidity Based on underlying graph, no numeric data Laman’s Theorem: Given a minimally rigid graph G=(V, E) with n vertices, m edges: (i) m = 2n-3 (ii) For every sub graph G’ of k vertices has <= 2k-3 edges

Generic Rigidity & Tightness A generic minimally rigid framework(or tight dir. network) : minimally rigid (or tight) for generic positions of points. Generic Non-generic

Generic Rigidity & Tightness A generic minimally rigid framework(or tight dir. network) : minimally rigid (or tight) for generic positions of points. Generic Non-generic

Solution to Problem 1 (Decision Problem) Geometric Approach : Using parallel redrawing matrix If there exists a solution to Mv=B Check dimension, if =0, TIGHT if > 0, UNDER-DETERMINED Else UNREALIZABLE Given a direction network, decide if tight or not

Decision Problem Geometric Approach (Cont’d) PROBLEM: In case of NOISE, geometric approach may return: Unrealizable (although if erroneous edges are not used, there might be a solution) So : we need a subset of these edges necessary and sufficient for a unique solution Use Combinatorial Approach to get a subset, then use Geometric Approach

Solution to Problem 1 (Decision Problem) Combinatorial Approach Main Idea: - Tight direction network  has a unique solution *most* of the time - Underlying graph of a tight D.N. is a minimally rigid graph - Find a way of checking if a given graph is minimally rigid or not

Decision Problem: Combinatorial Approach Sugihara’s Algorithm, based on bipartite matchings –Associate a bipartite graph to original –Modify it for each edge and check for complete bipartite matching Mathematica Notebook

Decision Problem: Combinatorial Approach: Sugihara’s Alg. Complexity O(n 1.5 ).O(n 2 ) =O(n 3.5 )

Solution to Problem 2 (Extension) Problem: Given an under-determined direction network, find a set of edges to extend it to a tight graph (if possible).

Extension Problem The Algorithm Natural Approach: A Greedy Algorithm Add one edge at a time, use “decision” algorithm to test Proof of Correctness!

Matroid Theory and Rigidity Definition: A matroid is an ordered pair M=(S, l) satisfying the following conditions 1. S is a finite non empty set 2. l is a non empty family of subsets of S called the independent subsets, s.t. if B  l and A  B, then A  l 3. If A  l, and | A |<| B |, then there exists an elt x  ( B-a) s.t. A  { x }  l

Matroid Theory and Rigidity 2 Important Properties: 1. Result by Sugihara: “Frameworks demonstrate matroidal properties.” 2. “For matroidal structures, greedy algorithms return optimal solutions correctly” Because of these two properties, our Greedy Approach returns a correct solution

Solution to Problem 3 (Extraction) Given an over-determined direction network, find a set of edges to be removed to get a tight one(if possible).

Extraction: Part 1 The Algorithm Greedy Algorithm Remove one edge at a time, use “decision” algorithm to test Correctness – based on Matroidal Property

For Extension and Extraction Problems: How about if we assign weights to edges? Weights can mean… –Easiest to sample –Most reliable Slight modification in code They help in recognizing a better/best tight direction network

Extraction Problem : Part 2 Connection b/n Combinatorial and Geometric Rigidity The Problem: Given a tight subset of a data set, find another (better?, w/out a non-generic embedding?) tight subset. Data Original Output New Output B C D A B C D A C D A

Further Work Motivating problem is in 3D. Our algorithms… –Emphasis not on efficiency but to show that they are implementable for the sake of experimentation …and implementations –Emphasis was on underlying theory. Used what was already available to us (Mathematica)

Summary Reformulation of Problem

Summary Reformulation of Problem Underlying Theory

Summary Reformulation of Problem Underlying Theory Implementations

Credits Seth Teller, for suggesting the problem Ruth Haas, Brigitte Servatius, Jack Snoeyink, Ileana Streinu, and Walter Whiteley, for their ideas and references

Special Thanks… Ileana Streinu Joseph O’Rourke Smith CS Dept Friends and Family

QUESTIONS ?