L-infinity minimization in geometric vision problems. work with Frederik Schaffalitzky

X x1 x2 x3

X x

Finding the minimax of 1D functions 2 2 3 3 4 4 5 5 6 6 7 7 Minimax point shown by pink circle.

2 3 1

1 1 2 2 3 3 4 4 5 5 6 6 7 7

Example Order Curve intersections 1 2 3 4 5 6 7 1 2 3 4 5 6 7 1 2 3 4

10 4 9 3 2 7 8

Initial heap Initial event queue 1 2 3 4 5 6 7 4 5 1 3 2 7 6 1 2 3 7 6

Scheduling new intersections When two nodes are swapped (pink) new potential intersections with adjacent nodes (blue) must be computed and scheduled. Priority queue (heap) used to order pending intersections.

First transition Updated event queue 4 5 1 3 2 7 6 4 5 2 3 1 7 6 1 2 1

4 5 1 3 2 7 6 4 5 2 3 1 7 6 1 2 5 6 3 7 4 4 1 2 7 5 3 6 4 1 2 3 5 7 6

1 2 5 6 3 7 4 1 2 3 4 5 6 7

1 2 3 4 5 6 7 Only the intersections marked in pink need to be considered

Direction (unit) vectors from cameras (blue) to points (black) are given : Find the positions of the cameras and points.

Multiview methods for Static and Dynamic Scenes

The problem: Dynamic scene reconstruction: From a video of a scene with multiple moving objects (and possibly camera) separate the individual objects and compute motion of each object and the camera.

General approach Track points through the image sequence. Classify points according to which object they belong to. Compute the rigid motion of each set of points. Points on a rigidly moving body must move in a consistent manner in an image sequence.

The simplest case Two images A single rigid motion Motion of object or motion of the camera are equivalent. Constraint is the “epipolar constraint” – matching points lie on corresponding epipolar lines.

Examples of epipolar lines: These are the intersections of the epipolar planes with the images. Corresponding points lie on corresponding lines. Point matching becomes a 1-dimensional search.

More examples of epipolar lines.

Three-view constraints: Constraints on matching features in three images. Line-line-line constraint.

Basic correspondence: the point-line-line correspondence.

Line-line-line correspondence Point-line-line correspondence Point-point-point correspondence

Unprocessed image sequence

Unweeded points

Outliers removed using the fundamental matrix

Outliers removed using the trifocal tensor

Reconstruction of dynamic scenes

Points are classified to one motion or the other according to their epipolar lines.

Results 1.4% misclassification

Results without spatial separation. -- two overlaid images. 0% mis-classification

The End

Other approaches to multiview multibody SFM. Affine factorization: (Kanatani) Basic assumption: affine camera. This means that there is not much perspective depth in the scene. Affine reconstruction is much easier.

Affine Factorization: Advantages: Simpler, handles any number of images Disadvantages: Requires the affine camera assumption. Points need to be seen in all views.

Dimension difference. In projective 3-view dynamic reconstruction we are (effectively) identifying hyperplanes in a 27-dimensional space. In affine dynamic scene reconstruction, point trajectories lie in a 4-dimensional subspace – different subspace for each object. Each method can be seen as fitting linear subspaces to point trajectories (or veronese vectors in perspective case). The Generalized PCA problem (Vidal).

Affine multibody factorization with missing data (Vidal-Hartley-2004). Method of fitting 4-dimensional subspaces to partial vectors using PowerFactorization (Hartley-xxxx). PowerFactorization fills in the missing data.

Thanks to: Rene Vidal for collaboration on dynamic scene segmentation. Andrew Zisserman and Mark Pollefeys for images and experimental results of 3D reconstruction. 2D3 for enhanced video results.

PowerFactorization Richard Hartley and Frederik Schaffalitzky Australian National University, Canberra Australia. And NICTA.