1. Lecture G: Multiple-View Reconstruction from Scene Knowledge
Breakthroughs in 3D Reconstruction and Motion Analysis
Lecture G: Multiple-View Reconstruction from Scene Knowledge
Yi Ma
Perception & Decision Laboratory
Decision & Control Group, CSL
Image Formation & Processing Group, Beckman
Electrical & Computer Engineering Dept., UIUC
September 15, 2003
ICRA2003, Taipei

2. INTRODUCTION: Scene knowledge and symmetry
This is the outline of my talk. Basically we are interested in geometry of multiple images taken for a scene with multiple moving objects, or non-rigid motions, the so-called dynamical scenes. This requires us to generalize existing multiple view geometry developed mostly for static scene to a dynamical scenario. We will first introduce one way to model perspective projection of a scene by embedding its dynamics into a higher dimensional space. This allows us to address conceptual issues such as whether or not a full reconstruction of the scene structure and dynamics is possible, the so-called observability issue from system theoretical viewpoint.
As we will see, in a multiple view setting, the observability is not a critical issue, in a sense that in principle it is always possible to fully recover the scene from sufficiently many views, even a rather rich class of dynamics is concerned. Then, like the classic multiple view geometry, what is important now is to identify all the intrinsic constraints, such as the epipolar constraint, among images which will potentially allow us to recover the structure and dynamics. We know that in multiple view geometry for static scene, these constraints boil down to multilinear constraints. However, it is difficult to generalize them to the dynamical setting, because as we will see that many intrinsic constraints that arise in the dynamical setting is NOT going to be linear, even if the scene dynamics themselves are.
We therefore propose in this talk a different approach. Our previous work has shown that a more global characterization of constraints among multiple images of a static scene is the so called rank conditions on certain matrix. We will show in this talk that the same principle carries into the context of dynamical scenes, even if different types of geometric primitives are considered.
Finally we conclude our talk by pointing out a few open directions and some of our current work on rank related issues.
Symmetry is ubiquitous in man-made or natural environments
September 15, 2003
ICRA2003, Taipei

3. INTRODUCTION: Scene knowledge and symmetry
Parallelism (vanishing point)
Orthogonality
Congruence
Self-similarity
This is the outline of my talk. Basically we are interested in geometry of multiple images taken for a scene with multiple moving objects, or non-rigid motions, the so-called dynamical scenes. This requires us to generalize existing multiple view geometry developed mostly for static scene to a dynamical scenario. We will first introduce one way to model perspective projection of a scene by embedding its dynamics into a higher dimensional space. This allows us to address conceptual issues such as whether or not a full reconstruction of the scene structure and dynamics is possible, the so-called observability issue from system theoretical viewpoint.
As we will see, in a multiple view setting, the observability is not a critical issue, in a sense that in principle it is always possible to fully recover the scene from sufficiently many views, even a rather rich class of dynamics is concerned. Then, like the classic multiple view geometry, what is important now is to identify all the intrinsic constraints, such as the epipolar constraint, among images which will potentially allow us to recover the structure and dynamics. We know that in multiple view geometry for static scene, these constraints boil down to multilinear constraints. However, it is difficult to generalize them to the dynamical setting, because as we will see that many intrinsic constraints that arise in the dynamical setting is NOT going to be linear, even if the scene dynamics themselves are.
We therefore propose in this talk a different approach. Our previous work has shown that a more global characterization of constraints among multiple images of a static scene is the so called rank conditions on certain matrix. We will show in this talk that the same principle carries into the context of dynamical scenes, even if different types of geometric primitives are considered.
Finally we conclude our talk by pointing out a few open directions and some of our current work on rank related issues.
September 15, 2003
ICRA2003, Taipei

4. INTRODUCTION: Wrong assumptions
Ames room illusion
Necker’s cube illusion
This is the outline of my talk. Basically we are interested in geometry of multiple images taken for a scene with multiple moving objects, or non-rigid motions, the so-called dynamical scenes. This requires us to generalize existing multiple view geometry developed mostly for static scene to a dynamical scenario. We will first introduce one way to model perspective projection of a scene by embedding its dynamics into a higher dimensional space. This allows us to address conceptual issues such as whether or not a full reconstruction of the scene structure and dynamics is possible, the so-called observability issue from system theoretical viewpoint.
As we will see, in a multiple view setting, the observability is not a critical issue, in a sense that in principle it is always possible to fully recover the scene from sufficiently many views, even a rather rich class of dynamics is concerned. Then, like the classic multiple view geometry, what is important now is to identify all the intrinsic constraints, such as the epipolar constraint, among images which will potentially allow us to recover the structure and dynamics. We know that in multiple view geometry for static scene, these constraints boil down to multilinear constraints. However, it is difficult to generalize them to the dynamical setting, because as we will see that many intrinsic constraints that arise in the dynamical setting is NOT going to be linear, even if the scene dynamics themselves are.
We therefore propose in this talk a different approach. Our previous work has shown that a more global characterization of constraints among multiple images of a static scene is the so called rank conditions on certain matrix. We will show in this talk that the same principle carries into the context of dynamical scenes, even if different types of geometric primitives are considered.
Finally we conclude our talk by pointing out a few open directions and some of our current work on rank related issues.
September 15, 2003
ICRA2003, Taipei

5. INTRODUCTION: Related literature
Mathematics: Hilbert 18th problem: Fedorov 1891, Hilbert 1901, Bieberbach 1910, George Polya 1924, Weyl 1952
Cognitive Science: Figure “goodness”: Gestalt theorists (1950s),
Garner’74, Chipman’77, Marr’82, Palmer’91’99…
Computer Vision (isotropic & homogeneous textures): Gibson’50,
Witkin’81, Garding’92’93, Malik&Rosenholtz’97, Leung&Malik’97…
Detection & Recognition (2D & 3D): Morola’89, Forsyth’91,
Vetter’94, Mukherjee’95, Zabrodsky’95, Basri & Moses’96,
Kanatani’97, Sun’97, Yang, Hong, Ma’02
This is the outline of my talk. Basically we are interested in geometry of multiple images taken for a scene with multiple moving objects, or non-rigid motions, the so-called dynamical scenes. This requires us to generalize existing multiple view geometry developed mostly for static scene to a dynamical scenario. We will first introduce one way to model perspective projection of a scene by embedding its dynamics into a higher dimensional space. This allows us to address conceptual issues such as whether or not a full reconstruction of the scene structure and dynamics is possible, the so-called observability issue from system theoretical viewpoint.
As we will see, in a multiple view setting, the observability is not a critical issue, in a sense that in principle it is always possible to fully recover the scene from sufficiently many views, even a rather rich class of dynamics is concerned. Then, like the classic multiple view geometry, what is important now is to identify all the intrinsic constraints, such as the epipolar constraint, among images which will potentially allow us to recover the structure and dynamics. We know that in multiple view geometry for static scene, these constraints boil down to multilinear constraints. However, it is difficult to generalize them to the dynamical setting, because as we will see that many intrinsic constraints that arise in the dynamical setting is NOT going to be linear, even if the scene dynamics themselves are.
We therefore propose in this talk a different approach. Our previous work has shown that a more global characterization of constraints among multiple images of a static scene is the so called rank conditions on certain matrix. We will show in this talk that the same principle carries into the context of dynamical scenes, even if different types of geometric primitives are considered.
Finally we conclude our talk by pointing out a few open directions and some of our current work on rank related issues.
Reconstruction (from single view): Kanade’81, Fawcett’93,
Rothwell’93, Zabrodsky’95’97, van Gool et.al.’96, Carlsson’98, Svedberg and Carlsson’99, Francois and Medioni’02, Huang, Yang, Hong, Ma’02,03
September 15, 2003
ICRA2003, Taipei

6. Multiple-View Reconstruction from Scene Knowledge
SYMMETRY & MULTIPLE-VIEW GEOMETRY
Fundamental types of symmetry
Equivalent views
Symmetry based reconstruction
MUTIPLE-VIEW MULTIPLE-OBJECT ALIGNMENT
Scale alignment: adjacent objects in a single view
Scale alignment: same object in multiple views
ALGORITHMS & EXAMPLES
Building 3-D geometric models with symmetry
Symmetry extraction, detection, and matching
Camera calibration
This is the outline of my talk. Basically we are interested in geometry of multiple images taken for a scene with multiple moving objects, or non-rigid motions, the so-called dynamical scenes. This requires us to generalize existing multiple view geometry developed mostly for static scene to a dynamical scenario. We will first introduce one way to model perspective projection of a scene by embedding its dynamics into a higher dimensional space. This allows us to address conceptual issues such as whether or not a full reconstruction of the scene structure and dynamics is possible, the so-called observability issue from system theoretical viewpoint.
As we will see, in a multiple view setting, the observability is not a critical issue, in a sense that in principle it is always possible to fully recover the scene from sufficiently many views, even a rather rich class of dynamics is concerned. Then, like the classic multiple view geometry, what is important now is to identify all the intrinsic constraints, such as the epipolar constraint, among images which will potentially allow us to recover the structure and dynamics. We know that in multiple view geometry for static scene, these constraints boil down to multilinear constraints. However, it is difficult to generalize them to the dynamical setting, because as we will see that many intrinsic constraints that arise in the dynamical setting is NOT going to be linear, even if the scene dynamics themselves are.
We therefore propose in this talk a different approach. Our previous work has shown that a more global characterization of constraints among multiple images of a static scene is the so called rank conditions on certain matrix. We will show in this talk that the same principle carries into the context of dynamical scenes, even if different types of geometric primitives are considered.
Finally we conclude our talk by pointing out a few open directions and some of our current work on rank related issues.
SUMMARY: Problems and future work
September 15, 2003
ICRA2003, Taipei

7. SYMMETRY & MUTIPLE-VIEW GEOMETRY
Why does an image of a symmetric object give away its structure?
Why does an image of a symmetric object give away its pose?
What else can we get from an image of a symmetric object?
September 15, 2003
ICRA2003, Taipei

8. Equivalent views from rotational symmetry
September 15, 2003
ICRA2003, Taipei

9. Equivalent views from reflectional symmetry
September 15, 2003
ICRA2003, Taipei

10. Equivalent views from translational symmetry
September 15, 2003
ICRA2003, Taipei

11. GEOMETRY FOR SINGLE IMAGES – Symmetric Structure
Definition. A set of 3-D features S is called a symmetric structure
if there exists a nontrivial subgroup G of E(3) that acts on it such
that for every g in G, the map
is an (isometric) automorphism of S. We say the structure S has a
group symmetry G.
G is isometric; G is discontinuous. Image of a symmetric object.
September 15, 2003
ICRA2003, Taipei

12. GEOMETRY FOR SINGLE IMAGES – Multiple “Equivalent” Views
September 15, 2003
ICRA2003, Taipei

13. GEOMETRY FOR SINGLE IMAGES – Symmetric Rank Condition
Solving g0 from Lyapunov equations:
with g’i and gi known.
September 15, 2003
ICRA2003, Taipei

14. THREE TYPES OF SYMMETRY – Reflective SymmetryPr
September 15, 2003
ICRA2003, Taipei

15. THREE TYPES OF SYMMETRY – Rotational Symmetry
September 15, 2003
ICRA2003, Taipei

16. THREE TYPES OF SYMMETRY – Translatory Symmetry
September 15, 2003
ICRA2003, Taipei

17. SINGLE-VIEW GEOMETRY WITH SYMMETRY – Ambiguities
“(a+b)-parameter” means there are an a-parameter family of ambiguity in
R0 of g0 and a b-parameter family of ambiguity in T0 of g0. P Pr Pr N
September 15, 2003
ICRA2003, Taipei

18. Symmetry-based reconstruction (reflection)
Reflectional symmetry 2
(1) 1
(2) 4
(3) 3
(4)
Virtual camera-camera
2qThis is the outline of my talk. Basically we are interested in geometry of multiple images taken for a scene with multiple moving objects, or non-rigid motions, the so-called dynamical scenes. This requires us to generalize existing multiple view geometry developed mostly for static scene to a dynamical scenario. We will first introduce one way to model perspective projection of a scene by embedding its dynamics into a higher dimensional space. This allows us to address conceptual issues such as whether or not a full reconstruction of the scene structure and dynamics is possible, the so-called observability issue from system theoretical viewpoint.
As we will see, in a multiple view setting, the observability is not a critical issue, in a sense that in principle it is always possible to fully recover the scene from sufficiently many views, even a rather rich class of dynamics is concerned. Then, like the classic multiple view geometry, what is important now is to identify all the intrinsic constraints, such as the epipolar constraint, among images which will potentially allow us to recover the structure and dynamics. We know that in multiple view geometry for static scene, these constraints boil down to multilinear constraints. However, it is difficult to generalize them to the dynamical setting, because as we will see that many intrinsic constraints that arise in the dynamical setting is NOT going to be linear, even if the scene dynamics themselves are.
We therefore propose in this talk a different approach. Our previous work has shown that a more global characterization of constraints among multiple images of a static scene is the so called rank conditions on certain matrix. We will show in this talk that the same principle carries into the context of dynamical scenes, even if different types of geometric primitives are considered.
Finally we conclude our talk by pointing out a few open directions and some of our current work on rank related issues.
September 15, 2003
ICRA2003, Taipei

19. Symmetry-based reconstruction
Epipolar constraint 2
(1) 1
(2) 4
(3) 3
(4)
Homography
2qThis is the outline of my talk. Basically we are interested in geometry of multiple images taken for a scene with multiple moving objects, or non-rigid motions, the so-called dynamical scenes. This requires us to generalize existing multiple view geometry developed mostly for static scene to a dynamical scenario. We will first introduce one way to model perspective projection of a scene by embedding its dynamics into a higher dimensional space. This allows us to address conceptual issues such as whether or not a full reconstruction of the scene structure and dynamics is possible, the so-called observability issue from system theoretical viewpoint.
As we will see, in a multiple view setting, the observability is not a critical issue, in a sense that in principle it is always possible to fully recover the scene from sufficiently many views, even a rather rich class of dynamics is concerned. Then, like the classic multiple view geometry, what is important now is to identify all the intrinsic constraints, such as the epipolar constraint, among images which will potentially allow us to recover the structure and dynamics. We know that in multiple view geometry for static scene, these constraints boil down to multilinear constraints. However, it is difficult to generalize them to the dynamical setting, because as we will see that many intrinsic constraints that arise in the dynamical setting is NOT going to be linear, even if the scene dynamics themselves are.
We therefore propose in this talk a different approach. Our previous work has shown that a more global characterization of constraints among multiple images of a static scene is the so called rank conditions on certain matrix. We will show in this talk that the same principle carries into the context of dynamical scenes, even if different types of geometric primitives are considered.
Finally we conclude our talk by pointing out a few open directions and some of our current work on rank related issues.
September 15, 2003
ICRA2003, Taipei

20. Symmetry-based reconstruction (algorithm)
2 pairs of symmetric image points
Recover essential matrix or homography
Decompose or to obtain
2qThis is the outline of my talk. Basically we are interested in geometry of multiple images taken for a scene with multiple moving objects, or non-rigid motions, the so-called dynamical scenes. This requires us to generalize existing multiple view geometry developed mostly for static scene to a dynamical scenario. We will first introduce one way to model perspective projection of a scene by embedding its dynamics into a higher dimensional space. This allows us to address conceptual issues such as whether or not a full reconstruction of the scene structure and dynamics is possible, the so-called observability issue from system theoretical viewpoint.
As we will see, in a multiple view setting, the observability is not a critical issue, in a sense that in principle it is always possible to fully recover the scene from sufficiently many views, even a rather rich class of dynamics is concerned. Then, like the classic multiple view geometry, what is important now is to identify all the intrinsic constraints, such as the epipolar constraint, among images which will potentially allow us to recover the structure and dynamics. We know that in multiple view geometry for static scene, these constraints boil down to multilinear constraints. However, it is difficult to generalize them to the dynamical setting, because as we will see that many intrinsic constraints that arise in the dynamical setting is NOT going to be linear, even if the scene dynamics themselves are.
We therefore propose in this talk a different approach. Our previous work has shown that a more global characterization of constraints among multiple images of a static scene is the so called rank conditions on certain matrix. We will show in this talk that the same principle carries into the context of dynamical scenes, even if different types of geometric primitives are considered.
Finally we conclude our talk by pointing out a few open directions and some of our current work on rank related issues.
Solve Lyapunov equation
to obtain and then .
September 15, 2003
ICRA2003, Taipei

21. Symmetry-based reconstruction (reflection)
2qThis is the outline of my talk. Basically we are interested in geometry of multiple images taken for a scene with multiple moving objects, or non-rigid motions, the so-called dynamical scenes. This requires us to generalize existing multiple view geometry developed mostly for static scene to a dynamical scenario. We will first introduce one way to model perspective projection of a scene by embedding its dynamics into a higher dimensional space. This allows us to address conceptual issues such as whether or not a full reconstruction of the scene structure and dynamics is possible, the so-called observability issue from system theoretical viewpoint.
As we will see, in a multiple view setting, the observability is not a critical issue, in a sense that in principle it is always possible to fully recover the scene from sufficiently many views, even a rather rich class of dynamics is concerned. Then, like the classic multiple view geometry, what is important now is to identify all the intrinsic constraints, such as the epipolar constraint, among images which will potentially allow us to recover the structure and dynamics. We know that in multiple view geometry for static scene, these constraints boil down to multilinear constraints. However, it is difficult to generalize them to the dynamical setting, because as we will see that many intrinsic constraints that arise in the dynamical setting is NOT going to be linear, even if the scene dynamics themselves are.
We therefore propose in this talk a different approach. Our previous work has shown that a more global characterization of constraints among multiple images of a static scene is the so called rank conditions on certain matrix. We will show in this talk that the same principle carries into the context of dynamical scenes, even if different types of geometric primitives are considered.
Finally we conclude our talk by pointing out a few open directions and some of our current work on rank related issues.
September 15, 2003
ICRA2003, Taipei

22. Symmetry-based reconstruction (rotation)
2qThis is the outline of my talk. Basically we are interested in geometry of multiple images taken for a scene with multiple moving objects, or non-rigid motions, the so-called dynamical scenes. This requires us to generalize existing multiple view geometry developed mostly for static scene to a dynamical scenario. We will first introduce one way to model perspective projection of a scene by embedding its dynamics into a higher dimensional space. This allows us to address conceptual issues such as whether or not a full reconstruction of the scene structure and dynamics is possible, the so-called observability issue from system theoretical viewpoint.
As we will see, in a multiple view setting, the observability is not a critical issue, in a sense that in principle it is always possible to fully recover the scene from sufficiently many views, even a rather rich class of dynamics is concerned. Then, like the classic multiple view geometry, what is important now is to identify all the intrinsic constraints, such as the epipolar constraint, among images which will potentially allow us to recover the structure and dynamics. We know that in multiple view geometry for static scene, these constraints boil down to multilinear constraints. However, it is difficult to generalize them to the dynamical setting, because as we will see that many intrinsic constraints that arise in the dynamical setting is NOT going to be linear, even if the scene dynamics themselves are.
We therefore propose in this talk a different approach. Our previous work has shown that a more global characterization of constraints among multiple images of a static scene is the so called rank conditions on certain matrix. We will show in this talk that the same principle carries into the context of dynamical scenes, even if different types of geometric primitives are considered.
Finally we conclude our talk by pointing out a few open directions and some of our current work on rank related issues.
September 15, 2003
ICRA2003, Taipei

23. Symmetry-based reconstruction (translation)
2qThis is the outline of my talk. Basically we are interested in geometry of multiple images taken for a scene with multiple moving objects, or non-rigid motions, the so-called dynamical scenes. This requires us to generalize existing multiple view geometry developed mostly for static scene to a dynamical scenario. We will first introduce one way to model perspective projection of a scene by embedding its dynamics into a higher dimensional space. This allows us to address conceptual issues such as whether or not a full reconstruction of the scene structure and dynamics is possible, the so-called observability issue from system theoretical viewpoint.
As we will see, in a multiple view setting, the observability is not a critical issue, in a sense that in principle it is always possible to fully recover the scene from sufficiently many views, even a rather rich class of dynamics is concerned. Then, like the classic multiple view geometry, what is important now is to identify all the intrinsic constraints, such as the epipolar constraint, among images which will potentially allow us to recover the structure and dynamics. We know that in multiple view geometry for static scene, these constraints boil down to multilinear constraints. However, it is difficult to generalize them to the dynamical setting, because as we will see that many intrinsic constraints that arise in the dynamical setting is NOT going to be linear, even if the scene dynamics themselves are.
We therefore propose in this talk a different approach. Our previous work has shown that a more global characterization of constraints among multiple images of a static scene is the so called rank conditions on certain matrix. We will show in this talk that the same principle carries into the context of dynamical scenes, even if different types of geometric primitives are considered.
Finally we conclude our talk by pointing out a few open directions and some of our current work on rank related issues.
September 15, 2003
ICRA2003, Taipei

24. ALIGNMENT OF MULTIPLE SYMMETRIC OBJECTS?
This is the outline of my talk. Basically we are interested in geometry of multiple images taken for a scene with multiple moving objects, or non-rigid motions, the so-called dynamical scenes. This requires us to generalize existing multiple view geometry developed mostly for static scene to a dynamical scenario. We will first introduce one way to model perspective projection of a scene by embedding its dynamics into a higher dimensional space. This allows us to address conceptual issues such as whether or not a full reconstruction of the scene structure and dynamics is possible, the so-called observability issue from system theoretical viewpoint.
As we will see, in a multiple view setting, the observability is not a critical issue, in a sense that in principle it is always possible to fully recover the scene from sufficiently many views, even a rather rich class of dynamics is concerned. Then, like the classic multiple view geometry, what is important now is to identify all the intrinsic constraints, such as the epipolar constraint, among images which will potentially allow us to recover the structure and dynamics. We know that in multiple view geometry for static scene, these constraints boil down to multilinear constraints. However, it is difficult to generalize them to the dynamical setting, because as we will see that many intrinsic constraints that arise in the dynamical setting is NOT going to be linear, even if the scene dynamics themselves are.
We therefore propose in this talk a different approach. Our previous work has shown that a more global characterization of constraints among multiple images of a static scene is the so called rank conditions on certain matrix. We will show in this talk that the same principle carries into the context of dynamical scenes, even if different types of geometric primitives are considered.
Finally we conclude our talk by pointing out a few open directions and some of our current work on rank related issues.
September 15, 2003
ICRA2003, Taipei

25. Correct scales within a single image
Pick the image of a point
on the intersection line
This is the outline of my talk. Basically we are interested in geometry of multiple images taken for a scene with multiple moving objects, or non-rigid motions, the so-called dynamical scenes. This requires us to generalize existing multiple view geometry developed mostly for static scene to a dynamical scenario. We will first introduce one way to model perspective projection of a scene by embedding its dynamics into a higher dimensional space. This allows us to address conceptual issues such as whether or not a full reconstruction of the scene structure and dynamics is possible, the so-called observability issue from system theoretical viewpoint.
As we will see, in a multiple view setting, the observability is not a critical issue, in a sense that in principle it is always possible to fully recover the scene from sufficiently many views, even a rather rich class of dynamics is concerned. Then, like the classic multiple view geometry, what is important now is to identify all the intrinsic constraints, such as the epipolar constraint, among images which will potentially allow us to recover the structure and dynamics. We know that in multiple view geometry for static scene, these constraints boil down to multilinear constraints. However, it is difficult to generalize them to the dynamical setting, because as we will see that many intrinsic constraints that arise in the dynamical setting is NOT going to be linear, even if the scene dynamics themselves are.
We therefore propose in this talk a different approach. Our previous work has shown that a more global characterization of constraints among multiple images of a static scene is the so called rank conditions on certain matrix. We will show in this talk that the same principle carries into the context of dynamical scenes, even if different types of geometric primitives are considered.
Finally we conclude our talk by pointing out a few open directions and some of our current work on rank related issues.
September 15, 2003
ICRA2003, Taipei

26. Correct scale within a single image
This is the outline of my talk. Basically we are interested in geometry of multiple images taken for a scene with multiple moving objects, or non-rigid motions, the so-called dynamical scenes. This requires us to generalize existing multiple view geometry developed mostly for static scene to a dynamical scenario. We will first introduce one way to model perspective projection of a scene by embedding its dynamics into a higher dimensional space. This allows us to address conceptual issues such as whether or not a full reconstruction of the scene structure and dynamics is possible, the so-called observability issue from system theoretical viewpoint.
As we will see, in a multiple view setting, the observability is not a critical issue, in a sense that in principle it is always possible to fully recover the scene from sufficiently many views, even a rather rich class of dynamics is concerned. Then, like the classic multiple view geometry, what is important now is to identify all the intrinsic constraints, such as the epipolar constraint, among images which will potentially allow us to recover the structure and dynamics. We know that in multiple view geometry for static scene, these constraints boil down to multilinear constraints. However, it is difficult to generalize them to the dynamical setting, because as we will see that many intrinsic constraints that arise in the dynamical setting is NOT going to be linear, even if the scene dynamics themselves are.
We therefore propose in this talk a different approach. Our previous work has shown that a more global characterization of constraints among multiple images of a static scene is the so called rank conditions on certain matrix. We will show in this talk that the same principle carries into the context of dynamical scenes, even if different types of geometric primitives are considered.
Finally we conclude our talk by pointing out a few open directions and some of our current work on rank related issues.
September 15, 2003
ICRA2003, Taipei

27. Correct scales across multiple images
September 15, 2003
ICRA2003, Taipei

28. Correct scales across multiple images
This is the outline of my talk. Basically we are interested in geometry of multiple images taken for a scene with multiple moving objects, or non-rigid motions, the so-called dynamical scenes. This requires us to generalize existing multiple view geometry developed mostly for static scene to a dynamical scenario. We will first introduce one way to model perspective projection of a scene by embedding its dynamics into a higher dimensional space. This allows us to address conceptual issues such as whether or not a full reconstruction of the scene structure and dynamics is possible, the so-called observability issue from system theoretical viewpoint.
As we will see, in a multiple view setting, the observability is not a critical issue, in a sense that in principle it is always possible to fully recover the scene from sufficiently many views, even a rather rich class of dynamics is concerned. Then, like the classic multiple view geometry, what is important now is to identify all the intrinsic constraints, such as the epipolar constraint, among images which will potentially allow us to recover the structure and dynamics. We know that in multiple view geometry for static scene, these constraints boil down to multilinear constraints. However, it is difficult to generalize them to the dynamical setting, because as we will see that many intrinsic constraints that arise in the dynamical setting is NOT going to be linear, even if the scene dynamics themselves are.
We therefore propose in this talk a different approach. Our previous work has shown that a more global characterization of constraints among multiple images of a static scene is the so called rank conditions on certain matrix. We will show in this talk that the same principle carries into the context of dynamical scenes, even if different types of geometric primitives are considered.
Finally we conclude our talk by pointing out a few open directions and some of our current work on rank related issues.
September 15, 2003
ICRA2003, Taipei

29. Correct scales across multiple images
This is the outline of my talk. Basically we are interested in geometry of multiple images taken for a scene with multiple moving objects, or non-rigid motions, the so-called dynamical scenes. This requires us to generalize existing multiple view geometry developed mostly for static scene to a dynamical scenario. We will first introduce one way to model perspective projection of a scene by embedding its dynamics into a higher dimensional space. This allows us to address conceptual issues such as whether or not a full reconstruction of the scene structure and dynamics is possible, the so-called observability issue from system theoretical viewpoint.
As we will see, in a multiple view setting, the observability is not a critical issue, in a sense that in principle it is always possible to fully recover the scene from sufficiently many views, even a rather rich class of dynamics is concerned. Then, like the classic multiple view geometry, what is important now is to identify all the intrinsic constraints, such as the epipolar constraint, among images which will potentially allow us to recover the structure and dynamics. We know that in multiple view geometry for static scene, these constraints boil down to multilinear constraints. However, it is difficult to generalize them to the dynamical setting, because as we will see that many intrinsic constraints that arise in the dynamical setting is NOT going to be linear, even if the scene dynamics themselves are.
We therefore propose in this talk a different approach. Our previous work has shown that a more global characterization of constraints among multiple images of a static scene is the so called rank conditions on certain matrix. We will show in this talk that the same principle carries into the context of dynamical scenes, even if different types of geometric primitives are considered.
Finally we conclude our talk by pointing out a few open directions and some of our current work on rank related issues.
September 15, 2003
ICRA2003, Taipei

30. Image alignment after scales corrected
This is the outline of my talk. Basically we are interested in geometry of multiple images taken for a scene with multiple moving objects, or non-rigid motions, the so-called dynamical scenes. This requires us to generalize existing multiple view geometry developed mostly for static scene to a dynamical scenario. We will first introduce one way to model perspective projection of a scene by embedding its dynamics into a higher dimensional space. This allows us to address conceptual issues such as whether or not a full reconstruction of the scene structure and dynamics is possible, the so-called observability issue from system theoretical viewpoint.
As we will see, in a multiple view setting, the observability is not a critical issue, in a sense that in principle it is always possible to fully recover the scene from sufficiently many views, even a rather rich class of dynamics is concerned. Then, like the classic multiple view geometry, what is important now is to identify all the intrinsic constraints, such as the epipolar constraint, among images which will potentially allow us to recover the structure and dynamics. We know that in multiple view geometry for static scene, these constraints boil down to multilinear constraints. However, it is difficult to generalize them to the dynamical setting, because as we will see that many intrinsic constraints that arise in the dynamical setting is NOT going to be linear, even if the scene dynamics themselves are.
We therefore propose in this talk a different approach. Our previous work has shown that a more global characterization of constraints among multiple images of a static scene is the so called rank conditions on certain matrix. We will show in this talk that the same principle carries into the context of dynamical scenes, even if different types of geometric primitives are considered.
Finally we conclude our talk by pointing out a few open directions and some of our current work on rank related issues.
September 15, 2003
ICRA2003, Taipei

31. 1. Specify symmetric objects and correspondence
ALGORITHM: Building 3-D geometric models
1. Specify symmetric objects and correspondence
This is the outline of my talk. Basically we are interested in geometry of multiple images taken for a scene with multiple moving objects, or non-rigid motions, the so-called dynamical scenes. This requires us to generalize existing multiple view geometry developed mostly for static scene to a dynamical scenario. We will first introduce one way to model perspective projection of a scene by embedding its dynamics into a higher dimensional space. This allows us to address conceptual issues such as whether or not a full reconstruction of the scene structure and dynamics is possible, the so-called observability issue from system theoretical viewpoint.
As we will see, in a multiple view setting, the observability is not a critical issue, in a sense that in principle it is always possible to fully recover the scene from sufficiently many views, even a rather rich class of dynamics is concerned. Then, like the classic multiple view geometry, what is important now is to identify all the intrinsic constraints, such as the epipolar constraint, among images which will potentially allow us to recover the structure and dynamics. We know that in multiple view geometry for static scene, these constraints boil down to multilinear constraints. However, it is difficult to generalize them to the dynamical setting, because as we will see that many intrinsic constraints that arise in the dynamical setting is NOT going to be linear, even if the scene dynamics themselves are.
We therefore propose in this talk a different approach. Our previous work has shown that a more global characterization of constraints among multiple images of a static scene is the so called rank conditions on certain matrix. We will show in this talk that the same principle carries into the context of dynamical scenes, even if different types of geometric primitives are considered.
Finally we conclude our talk by pointing out a few open directions and some of our current work on rank related issues.
September 15, 2003
ICRA2003, Taipei

32. 2. Recover camera poses and scene structure
Building 3-D geometric models
2. Recover camera poses and scene structure
This is the outline of my talk. Basically we are interested in geometry of multiple images taken for a scene with multiple moving objects, or non-rigid motions, the so-called dynamical scenes. This requires us to generalize existing multiple view geometry developed mostly for static scene to a dynamical scenario. We will first introduce one way to model perspective projection of a scene by embedding its dynamics into a higher dimensional space. This allows us to address conceptual issues such as whether or not a full reconstruction of the scene structure and dynamics is possible, the so-called observability issue from system theoretical viewpoint.
As we will see, in a multiple view setting, the observability is not a critical issue, in a sense that in principle it is always possible to fully recover the scene from sufficiently many views, even a rather rich class of dynamics is concerned. Then, like the classic multiple view geometry, what is important now is to identify all the intrinsic constraints, such as the epipolar constraint, among images which will potentially allow us to recover the structure and dynamics. We know that in multiple view geometry for static scene, these constraints boil down to multilinear constraints. However, it is difficult to generalize them to the dynamical setting, because as we will see that many intrinsic constraints that arise in the dynamical setting is NOT going to be linear, even if the scene dynamics themselves are.
We therefore propose in this talk a different approach. Our previous work has shown that a more global characterization of constraints among multiple images of a static scene is the so called rank conditions on certain matrix. We will show in this talk that the same principle carries into the context of dynamical scenes, even if different types of geometric primitives are considered.
Finally we conclude our talk by pointing out a few open directions and some of our current work on rank related issues.
September 15, 2003
ICRA2003, Taipei

33. 3. Obtain 3-D model and rendering with images
Building 3-D geometric models
3. Obtain 3-D model and rendering with images
This is the outline of my talk. Basically we are interested in geometry of multiple images taken for a scene with multiple moving objects, or non-rigid motions, the so-called dynamical scenes. This requires us to generalize existing multiple view geometry developed mostly for static scene to a dynamical scenario. We will first introduce one way to model perspective projection of a scene by embedding its dynamics into a higher dimensional space. This allows us to address conceptual issues such as whether or not a full reconstruction of the scene structure and dynamics is possible, the so-called observability issue from system theoretical viewpoint.
As we will see, in a multiple view setting, the observability is not a critical issue, in a sense that in principle it is always possible to fully recover the scene from sufficiently many views, even a rather rich class of dynamics is concerned. Then, like the classic multiple view geometry, what is important now is to identify all the intrinsic constraints, such as the epipolar constraint, among images which will potentially allow us to recover the structure and dynamics. We know that in multiple view geometry for static scene, these constraints boil down to multilinear constraints. However, it is difficult to generalize them to the dynamical setting, because as we will see that many intrinsic constraints that arise in the dynamical setting is NOT going to be linear, even if the scene dynamics themselves are.
We therefore propose in this talk a different approach. Our previous work has shown that a more global characterization of constraints among multiple images of a static scene is the so called rank conditions on certain matrix. We will show in this talk that the same principle carries into the context of dynamical scenes, even if different types of geometric primitives are considered.
Finally we conclude our talk by pointing out a few open directions and some of our current work on rank related issues.
September 15, 2003
ICRA2003, Taipei

34. Extract, detect, match symmetric objects across
ALGORITHM: Symmetry detection and matching
Extract, detect, match symmetric objects across
images, and recover the camera poses.
September 15, 2003
ICRA2003, Taipei

35. Color-based segmentation (mean shift) 2. Polygon fitting
Segmentation & polygon fitting
Color-based segmentation (mean shift)
2. Polygon fitting
September 15, 2003
ICRA2003, Taipei

36. 3. Symmetry verification (rectangles,…)
Symmetry verification & recovery
3. Symmetry verification (rectangles,…)
4. Single-view recovery
September 15, 2003
ICRA2003, Taipei

37. 5. Find the only one set of camera poses that
Symmetry-based matching
5. Find the only one set of camera poses that
are consistent with all symmetry objects
September 15, 2003
ICRA2003, Taipei

38. MATCHING OF SYMMETRY CELLS – Graph Representation
Cell in image 1
Cell in image 2
# of possible matching
Camera transformation 1 2 3
The problem of finding the largest set of matching cells is equivalent to the problem of finding the maximal complete subgraphs (cliques) in the matching graph.
36 possible matchings
September 15, 2003
ICRA2003, Taipei

39. Camera poses and 3-D recovery
Side view
Top view
Generic view
Length ratio
Reconstruction
Ground truth
Whiteboard
1.506
1.51
Table
1.003
1.00
September 15, 2003
ICRA2003, Taipei

40. Multiple-view matching and recovery (Ambiguities)
September 15, 2003
ICRA2003, Taipei

41. Multiple-view matching and recovery (Ambiguities)
September 15, 2003
ICRA2003, Taipei

42. ALGORITHM: Calibration from symmetry
Calibrated homography
Uncalibrated homography
(vanishing point)
This is the outline of my talk. Basically we are interested in geometry of multiple images taken for a scene with multiple moving objects, or non-rigid motions, the so-called dynamical scenes. This requires us to generalize existing multiple view geometry developed mostly for static scene to a dynamical scenario. We will first introduce one way to model perspective projection of a scene by embedding its dynamics into a higher dimensional space. This allows us to address conceptual issues such as whether or not a full reconstruction of the scene structure and dynamics is possible, the so-called observability issue from system theoretical viewpoint.
As we will see, in a multiple view setting, the observability is not a critical issue, in a sense that in principle it is always possible to fully recover the scene from sufficiently many views, even a rather rich class of dynamics is concerned. Then, like the classic multiple view geometry, what is important now is to identify all the intrinsic constraints, such as the epipolar constraint, among images which will potentially allow us to recover the structure and dynamics. We know that in multiple view geometry for static scene, these constraints boil down to multilinear constraints. However, it is difficult to generalize them to the dynamical setting, because as we will see that many intrinsic constraints that arise in the dynamical setting is NOT going to be linear, even if the scene dynamics themselves are.
We therefore propose in this talk a different approach. Our previous work has shown that a more global characterization of constraints among multiple images of a static scene is the so called rank conditions on certain matrix. We will show in this talk that the same principle carries into the context of dynamical scenes, even if different types of geometric primitives are considered.
Finally we conclude our talk by pointing out a few open directions and some of our current work on rank related issues.
September 15, 2003
ICRA2003, Taipei

43. ALGORITHM: Calibration from symmetry
Calibration with a rig is also simplified: we only need
to know that there are sufficient symmetries, not
necessarily the 3-D coordinates of points on the rig.
This is the outline of my talk. Basically we are interested in geometry of multiple images taken for a scene with multiple moving objects, or non-rigid motions, the so-called dynamical scenes. This requires us to generalize existing multiple view geometry developed mostly for static scene to a dynamical scenario. We will first introduce one way to model perspective projection of a scene by embedding its dynamics into a higher dimensional space. This allows us to address conceptual issues such as whether or not a full reconstruction of the scene structure and dynamics is possible, the so-called observability issue from system theoretical viewpoint.
As we will see, in a multiple view setting, the observability is not a critical issue, in a sense that in principle it is always possible to fully recover the scene from sufficiently many views, even a rather rich class of dynamics is concerned. Then, like the classic multiple view geometry, what is important now is to identify all the intrinsic constraints, such as the epipolar constraint, among images which will potentially allow us to recover the structure and dynamics. We know that in multiple view geometry for static scene, these constraints boil down to multilinear constraints. However, it is difficult to generalize them to the dynamical setting, because as we will see that many intrinsic constraints that arise in the dynamical setting is NOT going to be linear, even if the scene dynamics themselves are.
We therefore propose in this talk a different approach. Our previous work has shown that a more global characterization of constraints among multiple images of a static scene is the so called rank conditions on certain matrix. We will show in this talk that the same principle carries into the context of dynamical scenes, even if different types of geometric primitives are considered.
Finally we conclude our talk by pointing out a few open directions and some of our current work on rank related issues.
September 15, 2003
ICRA2003, Taipei

44. SUMMARY: Multiple-View Geometry + Symmetry
Multiple (perspective) images = multiple-view rank condition
Single image + symmetry = “multiple-view” rank condition
Multiple images + symmetry = rank condition + scale correction
Matching + symmetry = rank condition + scale correction
+ clique identification
This is the outline of my talk. Basically we are interested in geometry of multiple images taken for a scene with multiple moving objects, or non-rigid motions, the so-called dynamical scenes. This requires us to generalize existing multiple view geometry developed mostly for static scene to a dynamical scenario. We will first introduce one way to model perspective projection of a scene by embedding its dynamics into a higher dimensional space. This allows us to address conceptual issues such as whether or not a full reconstruction of the scene structure and dynamics is possible, the so-called observability issue from system theoretical viewpoint.
As we will see, in a multiple view setting, the observability is not a critical issue, in a sense that in principle it is always possible to fully recover the scene from sufficiently many views, even a rather rich class of dynamics is concerned. Then, like the classic multiple view geometry, what is important now is to identify all the intrinsic constraints, such as the epipolar constraint, among images which will potentially allow us to recover the structure and dynamics. We know that in multiple view geometry for static scene, these constraints boil down to multilinear constraints. However, it is difficult to generalize them to the dynamical setting, because as we will see that many intrinsic constraints that arise in the dynamical setting is NOT going to be linear, even if the scene dynamics themselves are.
We therefore propose in this talk a different approach. Our previous work has shown that a more global characterization of constraints among multiple images of a static scene is the so called rank conditions on certain matrix. We will show in this talk that the same principle carries into the context of dynamical scenes, even if different types of geometric primitives are considered.
Finally we conclude our talk by pointing out a few open directions and some of our current work on rank related issues.
September 15, 2003
ICRA2003, Taipei

45. SUMMARY Multiple-view 3-D reconstruction in presence of symmetry
Symmetry based algorithms are accurate, robust, and simple.
Methods are baseline independent and object centered.
Alignment and matching can and should take place in 3-D space.
Camera self-calibration and calibration are simplified and linear.
Related applications
Using symmetry to overcome occlusion.
Reconstruction and rendering with non-symmetric area.
Large scale 3-D map building of man-made environments.
This is the outline of my talk. Basically we are interested in geometry of multiple images taken for a scene with multiple moving objects, or non-rigid motions, the so-called dynamical scenes. This requires us to generalize existing multiple view geometry developed mostly for static scene to a dynamical scenario. We will first introduce one way to model perspective projection of a scene by embedding its dynamics into a higher dimensional space. This allows us to address conceptual issues such as whether or not a full reconstruction of the scene structure and dynamics is possible, the so-called observability issue from system theoretical viewpoint.
As we will see, in a multiple view setting, the observability is not a critical issue, in a sense that in principle it is always possible to fully recover the scene from sufficiently many views, even a rather rich class of dynamics is concerned. Then, like the classic multiple view geometry, what is important now is to identify all the intrinsic constraints, such as the epipolar constraint, among images which will potentially allow us to recover the structure and dynamics. We know that in multiple view geometry for static scene, these constraints boil down to multilinear constraints. However, it is difficult to generalize them to the dynamical setting, because as we will see that many intrinsic constraints that arise in the dynamical setting is NOT going to be linear, even if the scene dynamics themselves are.
We therefore propose in this talk a different approach. Our previous work has shown that a more global characterization of constraints among multiple images of a static scene is the so called rank conditions on certain matrix. We will show in this talk that the same principle carries into the context of dynamical scenes, even if different types of geometric primitives are considered.
Finally we conclude our talk by pointing out a few open directions and some of our current work on rank related issues.
September 15, 2003
ICRA2003, Taipei