1. MASKS © 2004 Invitation to 3D vision Lecture 6 Multiple-View Reconstruction from Scene Knowledge Multiple-View Geometry for Image-Based Modeling

2. MASKS © 2004 Invitation to 3D vision Symmetry is ubiquitous in man-made or natural environments INTRODUCTION: Scene knowledge and symmetry

3. MASKS © 2004 Invitation to 3D vision INTRODUCTION: Scene knowledge and symmetry Parallelism (vanishing point)Orthogonality CongruenceSelf-similarity

4. MASKS © 2004 Invitation to 3D vision INTRODUCTION: Wrong assumptions Ames room illusion Necker’s cube illusion

5. MASKS © 2004 Invitation to 3D vision INTRODUCTION: Related literature Cognitive Science: Figure “goodness”: Gestalt theorists (1950s), Garner’74, Chipman’77, Marr’82, Palmer’91’99… Computer Vision (isotropic & homogeneous textures): Gibson’50, Witkin’81, Garding’92’93, Malik&Rosenholtz’97, Leung&Malik’97… Mathematics: Hilbert 18 th problem: Fedorov 1891, Hilbert 1901, Bieberbach 1910, George Polya 1924, Weyl 1952 Detection & Recognition (2D & 3D): Morola’89, Forsyth’91, Vetter’94, Mukherjee’95, Zabrodsky’95, Basri & Moses’96, Kanatani’97, Sun’97, Yang, Hong, Ma’02 Reconstruction (from single view): Kanade’81, Fawcett’93, Rothwell’93, Zabrodsky’95’97, van Gool et.al.’96, Carlsson’98, Svedberg and Carlsson’99, Francois and Medioni’02, Huang, Yang, Hong, Ma’02,03

6. MASKS © 2004 Invitation to 3D vision MUTIPLE-VIEW MULTIPLE-OBJECT ALIGNMENT Scale alignment: adjacent objects in a single view Scale alignment: same object in multiple views SUMMARY: Problems and future work ALGORITHMS & EXAMPLES Building 3-D geometric models with symmetry Symmetry extraction, detection, and matching Camera calibration SYMMETRY & MULTIPLE-VIEW GEOMETRY Fundamental types of symmetry Equivalent views Symmetry based reconstruction Multiple-View Reconstruction from Scene Knowledge

7. MASKS © 2004 Invitation to 3D vision SYMMETRY & MUTIPLE-VIEW GEOMETRY Why does an image of a symmetric object give away its structure? Why does an image of a symmetric object give away its pose? What else can we get from an image of a symmetric object?

8. MASKS © 2004 Invitation to 3D vision Equivalent views from rotational symmetry 90 O

9. MASKS © 2004 Invitation to 3D vision Equivalent views from reflectional symmetry

10. MASKS © 2004 Invitation to 3D vision Equivalent views from translational symmetry

11. MASKS © 2004 Invitation to 3D vision GEOMETRY FOR SINGLE IMAGES – Symmetric Structure Definition. A set of 3-D features S is called a symmetric structure if there exists a nontrivial subgroup G of E(3) that acts on it such that for every g in G, the map is an (isometric) automorphism of S. We say the structure S has a group symmetry G.

12. MASKS © 2004 Invitation to 3D vision GEOMETRY FOR SINGLE IMAGES – Multiple “Equivalent” Views

13. MASKS © 2004 Invitation to 3D vision GEOMETRY FOR SINGLE IMAGES – Symmetric Rank Condition Solving g 0 from Lyapunov equations: with g’ i and g i known.

14. MASKS © 2004 Invitation to 3D vision THREE TYPES OF SYMMETRY – Reflective Symmetry PrPr

15. MASKS © 2004 Invitation to 3D vision THREE TYPES OF SYMMETRY – Rotational Symmetry

16. MASKS © 2004 Invitation to 3D vision THREE TYPES OF SYMMETRY – Translatory Symmetry

17. MASKS © 2004 Invitation to 3D vision SINGLE-VIEW GEOMETRY WITH SYMMETRY – Ambiguities “(a+b)-parameter” means there are an a-parameter family of ambiguity in R 0 of g 0 and a b-parameter family of ambiguity in T 0 of g 0. P PrPr N PrPr

18. MASKS © 2004 Invitation to 3D vision Symmetry-based reconstruction (reflection) Reflectional symmetry Virtual camera-camera 1 2 3 4 (3) (4) (2) (1)

19. MASKS © 2004 Invitation to 3D vision Epipolar constraint Homography 1 2 3 4 (3) (4) (2) (1) Symmetry-based reconstruction

20. MASKS © 2004 Invitation to 3D vision 2 pairs of symmetric image points Decompose or to obtain Solve Lyapunov equation to obtain and then. Recover essential matrix or homography Symmetry-based reconstruction (algorithm)

21. MASKS © 2004 Invitation to 3D vision Symmetry-based reconstruction (reflection)

22. MASKS © 2004 Invitation to 3D vision Symmetry-based reconstruction (rotation)

23. MASKS © 2004 Invitation to 3D vision Symmetry-based reconstruction (translation)

24. MASKS © 2004 Invitation to 3D vision ALIGNMENT OF MULTIPLE SYMMETRIC OBJECTS ?

25. MASKS © 2004 Invitation to 3D vision Pick the image of a point on the intersection line Correct scales within a single image

26. MASKS © 2004 Invitation to 3D vision Correct scale within a single image

27. MASKS © 2004 Invitation to 3D vision Correct scales across multiple images

28. MASKS © 2004 Invitation to 3D vision Correct scales across multiple images

29. MASKS © 2004 Invitation to 3D vision Correct scales across multiple images

30. MASKS © 2004 Invitation to 3D vision Image alignment after scales corrected

31. MASKS © 2004 Invitation to 3D vision ALGORITHM: Building 3-D geometric models 1. Specify symmetric objects and correspondence

32. MASKS © 2004 Invitation to 3D vision Building 3-D geometric models 2. Recover camera poses and scene structure

33. MASKS © 2004 Invitation to 3D vision Building 3-D geometric models 3. Obtain 3-D model and rendering with images

34. MASKS © 2004 Invitation to 3D vision ALGORITHM: Symmetry detection and matching Extract, detect, match symmetric objects across images, and recover the camera poses.

35. MASKS © 2004 Invitation to 3D vision 1.Color-based segmentation (mean shift) 2. Polygon fitting Segmentation & polygon fitting

36. MASKS © 2004 Invitation to 3D vision Symmetry verification & recovery 3. Symmetry verification (rectangles,…) 4. Single-view recovery

37. MASKS © 2004 Invitation to 3D vision Symmetry-based matching 5. Find the only one set of camera poses that are consistent with all symmetry objects

38. MASKS © 2004 Invitation to 3D vision MATCHING OF SYMMETRY CELLS – Graph Representation 1 2 3 1 2 3 36 possible matchings (1,2,1, g) Cell in image 1 Cell in image 2 # of possible matching Camera transformation The problem of finding the largest set of matching cells is equivalent to the problem of finding the maximal complete subgraphs (cliques) in the matching graph.

39. MASKS © 2004 Invitation to 3D vision Camera poses and 3-D recovery Side view Top view Generic view Length ratioReconstructionGround truth Whiteboard1.5061.51 Table1.0031.00

40. MASKS © 2004 Invitation to 3D vision Multiple-view matching and recovery (Ambiguities)

41. MASKS © 2004 Invitation to 3D vision Multiple-view matching and recovery (Ambiguities)

42. MASKS © 2004 Invitation to 3D vision ALGORITHM: Calibration from symmetry (vanishing point) Calibrated homography Uncalibrated homography

43. MASKS © 2004 Invitation to 3D vision ALGORITHM: Calibration from symmetry Calibration with a rig is also simplified: we only need to know that there are sufficient symmetries, not necessarily the 3-D coordinates of points on the rig.

44. MASKS © 2004 Invitation to 3D vision SUMMARY: Multiple-View Geometry + Symmetry Multiple (perspective) images = multiple-view rank condition Single image + symmetry = “multiple-view” rank condition Multiple images + symmetry = rank condition + scale correction Matching + symmetry = rank condition + scale correction + clique identification

45. MASKS © 2004 Invitation to 3D vision Multiple-view 3-D reconstruction in presence of symmetry Symmetry based algorithms are accurate, robust, and simple. Methods are baseline independent and object centered. Alignment and matching can and should take place in 3-D space. Camera self-calibration and calibration are simplified and linear. Related applications Using symmetry to overcome occlusion. Reconstruction and rendering with non-symmetric area. Large scale 3-D map building of man-made environments. SUMMARY