Automatic 3D modeling from range images Daniel Huber Carnegie Mellon University Robotics Institute

2 Outline What is 3D modeling and why automate it? Proposed approach Automatic modeling algorithms Basic modeling operations Automatic modeling example

What is 3D modeling? Applications Reverse engineering Multimedia content creation Preservation of art and architecture Terrain and environment modeling Model-based vision systems conversion to meshes pair-wise registration integration multi-view registration final model original object

4 3D modeling issues and current solutions Registration issues Which views overlap? What are the relative poses? What are the absolute poses? Mechanical solutions Measure absolute poses Measure relative poses Manual solutions Mark correspondences Hand register Specify overlapping views and verify results

5 Example application – handheld modeling User holds object to be modeled Range images are captured from various viewpoints Software determines pose relations and outputs the 3D model

6 Benefits of automatic modeling Enables new applications Handheld modeling Large-scale environment modeling Saves effort Simplifies data collection Eliminates tedious hand registration Provides an alternative to mechanical solutions Less cost & complexity Fewer limits on domain

7 Some existing modeling systems Modeling from range images Mechanical – Rioux (interiors), Miller (terrain), Levoy, Wheeler (objects) Manual – Neugebauer, Ikeuchi, Pulli (statues), Johnson (objects and interiors) Modeling from intensity images Video streams – Tomasi & Kanade, Zisserman (man- made objects) Photographs – Debevec, Zisserman (buildings), Sullivan & Ponce, Kutulakos & Seitz (objects)

8 The automatic modeling problem Don’t assume: Relative poses are known Overlapping views are known Structured scene – planes, corners, etc. Do assume: Views will overlap Views may be noisy or spurious Static scene Given a set of range images of an unstructured scene, automatically construct a consistent 3D model.

9 Overview of automatic modeling approach Select view pairs that are likely to register correctly Perform pair-wise registration and locally verify Robustly combine matches in a network of views (model graph) Infer new links based on topology Verify global consistency Improve quality by simultaneously registering all views

10 Key difficulties Don’t know what the object looks like Don’t know approximate poses or even which views overlap Pair-wise registration fails sometimes: Incorrect matches Missed matches Multiple matches (best match may be wrong) Pair-wise registration takes time – need to choose views to register Local verification cannot always detect errors – use global verification

11 Example of a locally unresolvable ambiguity overhead view views of a room pair-wise registration gives two equally good matches global consistency checks resolve the ambiguity the resulting model hypotheses

12 Surface matching process View selection Pair-wise registration Local verification Proposed automatic modeling framework Working set Pair-wise registration results Inferred matches Model construction process Topological inference Conflict detection Multi-view registration Model hypotheses selected matches Input views selected views registration results inferred matches output model

13 The model graph Encode model topology Nodes are views Arcs are transforms between overlapping views Spanning tree is the minimal complete model Perform model construction Start with no arcs Basic operations add, modify, or delete arcs Encode a priori information as initial arcs incorrect matches correct matches T 1,2

14 Automatic modeling framework Automatic modeling algorithms Model construction process Global verification Topological inference Conflict detection Multi-view registration Surface matching process View selection Pair-wise registration Local verification Basic modeling operations

15 Automatic modeling framework Automatic modeling algorithms Model construction process Global verification Topological inference Conflict detection Multi-view registration Surface matching process View selection Pair-wise registration Local verification Basic modeling operations

16 Automatic modeling algorithms Model construction difficulties One bad match corrupts the model Noisy data and spurious views Incomplete information – missed or untried matches Solutions Combinatorial optimization – deterministic and non- deterministic Study simpler problem first Exhaustive registration – try all pairs up front Selective registration – enable view selection

17 Exhaustive registration Sequential removal – start with all matches and remove Sequential addition – start with seed view and add Merging Repeatedly merge partial models Track multiple hypotheses Non-deterministic methods RANSAC – spanning tree + verification MCMC or annealing – search for maximum likelihood model

18 Selective registration View selection Use order of data collection Cluster similarly shaped views and register in clusters Sort views by shape similarity and register in order Choose self-constraining or unique views Modify exhaustive methods to include view selection

19 Automatic modeling framework Automatic modeling algorithms Model construction process Global verification Topological inference Conflict detection Multi-view registration Surface matching process View selection Pair-wise registration Local verification Basic modeling operations

20 Automatic modeling framework Automatic modeling algorithms Model construction process Global verification Topological inference Conflict detection Multi-view registration Surface matching process View selection Pair-wise registration Local verification Basic modeling operations

21 Uninformed pair-wise registration Align two views with no initial estimate Based on Johnson’s surface matching engine Uses local shape signatures – spin images Searches for similarly shaped regions Fails dramatically sometimes

22 Pair-wise registration refinement Align two views given a close initial estimate Iterative closest point algorithm (Besl & McKay, Zhang) Repeatedly minimize distance between closest points Incorrect correspondences slow convergence Minimize point to tangent plane distance instead (Chen & Medioni) point-point correspondences point-plane correspondences slow sliding fast sliding

23 Automatic modeling framework Automatic modeling algorithms Model construction process Global verification Topological inference Conflict detection Multi-view registration Surface matching process View selection Pair-wise registration Local verification Basic modeling operations

24 Local verification – overlap distance Registration requires significant overlap for success (~ 30%) Correctly registered views should be close wherever they overlap Verify matches based on overlap amount and distance Use as a classifier or as a measure of match quality Views 1 and 2: 48%, 2.1 mmViews 1 and 9: 15%, 3.1 mm

25 Local verification – visibility consistency Consider expected observations from viewpoint of sensor C 1 Consistent – S 2 close to S 1 wherever both observed Free space violation – S 2 blocks visibility of S 1 Occupied space violation – S 1 unobserved, even though it should be (sensor model required) consistent surfaces S1S1 S2S2 C1C1 C2C2 surfaces are close free space violation S1S1 S2S2 C1C1 C2C2 S 2 blocks S 1 occupied space violation S1S1 S2S2 C1C1 C2C2 S 1 is not observed

26 Model construction process Global verification Topological inference Conflict detection Multi-view registration Automatic modeling framework Automatic modeling algorithms Surface matching process View selection Pair-wise registration Local verification Basic modeling operations

27 Global verification Topological inference (positive evidence) Coarse test – bounding box intersection Fine test – local verification procedures Pair-wise register if possible/necessary Conflict detection (negative evidence) Coarse test – frustum or view volume intersection Fine test – visibility consistency

28 Automatic modeling framework Automatic modeling algorithms Model construction process Global verification Topological inference Conflict detection Multi-view registration Surface matching process View selection Pair-wise registration Local verification Basic modeling operations

29 Motivation for multi-view registration T12T12 T23T23 T34T34 T45T45 T56T56 T61T61 T 12 * T 23 * T 34 * T 45 * T 56 * T 61  I Pair-wise registration gives relative pose with small error Errors accumulate with compounding operation, leading to gaps or seams on the final model Multi-view registration distributes the accumulated error in a principled way Close-up of tail - the red, green and blue surfaces should be aligned with the cyan one top view

30 Multi-view registration Simultaneously align two or more surfaces Multi-view ICP Minimize squared distance between corresponding points (Benjemaa & Schmitt) Requires initial pair-wise estimates Suffers from same “sliding” problem as ICP Minimize point-plane distances, instead (Neugebauer)

31 Comparison of multi-view registration methods pair-wise registration only point-point correspondences point-plane correspondences Synchronized cross-section

32 Automatically modeling the angel top view connectivity after local verification horizontal slice initial model (bad matches in red)

33 Inferring new pose relations % overlap1% overlap

34 horizontal slice The final angel model top view

35 Research schedule

36 Conclusion Expected contributions Framework for automatic modeling – model graph, basic operations Algorithms for exhaustive and selective modeling Demonstration in several domains, including handheld modeling application See the proposal for Range image to mesh conversion Registration uncertainty measure Planned experiments Details of related work