Structure Recovery by Part Assembly Chao-Hui Shen 1 Hongbo Fu 2 Kang Chen 1 Shi-Min Hu 1 1 Tsinghua University 2 City University of Hong Kong

Background Consumer level scanning devices Capture both RGB and depth Reconstruction is challenging – Low resolution – Noise – Missing data – …

Example-based Scan Completion Global-to-local and top-down [Kraevoy and Sheffer 2005; Pauly et al. 2005] Rely on the availability of suitable template model However … No suitable model! shape retrieval

Assembly-based 3D Modeling Data-drive suggestion and interaction [Chaudhuri and Koltun 2010; Chaudhuri et al. 2011] – Retrieve suitable parts to match user intent – Aim to support open-ended 3D modeling – Quite different goal from ours Automatic shape synthesis by part composition [Kalogerakis et al. 2012; Jain et al. 2012; Xu et al. 2012] – Result in database that grows exponentially – Significantly enlarge the existing database – But make storage and retrieval challenging

Our solution: Recover the Structure by Part Assembly Structure recovery instead of geometry reconstruction Do NOT prepare a large database Retrieve and assemble suitable parts on the fly

Problem Setup Input Point cloud + Image (Single view) Pre-segmented Repository Models (Parts + Labels) …… Goal: Recover high-level structure Assembly close to geometry Output …… Session: Acquiring and Synthesizing Indoor Scenes An Interactive Approach to Semantic Modeling of Indoor Scenes with an RGBD Camera [Shao et al. 2012] A Search-Classify Approach for Cluttered Indoor Scene Understanding [Nan et al. 2012] Acquiring 3D Indoor Environments with Variability and Repetition [Kim et al. 2012]

Directly searching is computationally prohibitive Need a quick way to explore meaningful structures guided by: – Spatial layout of the parts in the repository models – Acquired data Observations

Complementary characteristics of point cloud & image 3D, more accurate cues for geometry & structure Incomplete and noisy Lack depth information Capture the complete object

Algorithm Overview Candidate Parts SelectionStructure CompositionPart Conjoining ……

Algorithm Overview Candidate Parts SelectionStructure CompositionPart Conjoining ……

Candidate Parts Selection Goal: select a small set of candidates for each category Achieved by retrieving parts that fit well some regions

Straightforward Solution Search for the best-fit parts over the entire domain – Disregards the semantics associated with each part and the interaction between different parts Unlikely to produce good results! X X X X XXX X

Key Fact Man-made objects lie in a low dimensional space – Defined with respect to the relative sizes and positions of shape parts [Ovsjanikov et al. 2011] Employ 3D repository model as a global context – Globally align the models with the input scan first Search in a 3D offset window around the part

Part Matching Scheme Geometric fidelity score Geometric contribution score 3D2D edge map Total matching score 3D offset window

Candidate Parts Select top K parts with highest score for each category Seat Back Arm Front leg ……

Algorithm Overview Candidate Parts SelectionStructure CompositionPart Conjoining ……

Structure Composition Goal: compose the underlying structure by identifying a subset of candidate parts

Constraints for Promising Compositions Geometric fidelityProximityOverlap   having high scoreno isolated partsminimized intersection

Search and Evaluate Search for promising compositions under constraints Globally Evaluate the compositions average geometry fidelity of parts total geometry fidelity total geometry contribution …… optimal composition

Algorithm Overview Candidate Parts SelectionStructure CompositionPart Conjoining ……

Part Conjoining Problem: the parts are loosely placed together Goal: generate a consistent & complete model

Identification of Contact Points Refer to their parent models [Jain et al. 2012]

Matching of Contact Points Greedily match nearby contact points Generate auxiliary contact points when necessary auxiliary contact points

i j identity scale Global Optimization transformed contact points

Results: Chairs 70 repository models, 11 part categories

Results: Tables 61 repository models, 4 part categories

Results: Bicycles 38 repository models, 9 part categories

Results: Airplanes 70 repository models, 6 part categories

Results: Creating New Structures

Results: Impact of Dataset input data Randomly picking some repository models

Summary A bottom-up structure recovery approach – Effectively reuse limited repository models – Automatically compose new structure – Handle single-view inputs by the Kinect system Future work – Multi-view inputs – Include style/functional constraints – Recover Indoor scenes

Thank you! Project Page:

User-Assisted Preprocessing GrabCut to extract foreground object [Rother et al. 2004]

Performance Preprocessing step – 3 minutes user interaction Candidate part selection – 1 minutes for 70 models Structure composition – 2 minutes Part conjoining – 1 second

Limitations Lack of suitable parts in the repository

Limitations Input with severely missing geometry – The global alignment becomes unreliable