1. Structure Recovery by Part Assembly Chao-Hui Shen 1 Hongbo Fu 2 Kang Chen 1 Shi-Min Hu 1 1 Tsinghua University 2 City University of Hong Kong

2. Background Consumer level scanning devices Capture both RGB and depth Reconstruction is challenging – Low resolution – Noise – Missing data – …

3. Example-based Scan Completion Global-to-local and top-down [Kraevoy and Sheffer 2005; Pauly et al. 2005] Rely on the availability of suitable template model However … No suitable model! shape retrieval

4. Assembly-based 3D Modeling Data-drive suggestion and interaction [Chaudhuri and Koltun 2010; Chaudhuri et al. 2011] – Retrieve suitable parts to match user intent – Aim to support open-ended 3D modeling – Quite different goal from ours Automatic shape synthesis by part composition [Kalogerakis et al. 2012; Jain et al. 2012; Xu et al. 2012] – Result in database that grows exponentially – Significantly enlarge the existing database – But make storage and retrieval challenging

5. Our solution: Recover the Structure by Part Assembly Structure recovery instead of geometry reconstruction Do NOT prepare a large database Retrieve and assemble suitable parts on the fly

6. Problem Setup Input Point cloud + Image (Single view) Pre-segmented Repository Models (Parts + Labels) …… Goal: Recover high-level structure Assembly close to geometry Output …… Session: Acquiring and Synthesizing Indoor Scenes An Interactive Approach to Semantic Modeling of Indoor Scenes with an RGBD Camera [Shao et al. 2012] A Search-Classify Approach for Cluttered Indoor Scene Understanding [Nan et al. 2012] Acquiring 3D Indoor Environments with Variability and Repetition [Kim et al. 2012]

7. Directly searching is computationally prohibitive Need a quick way to explore meaningful structures guided by: – Spatial layout of the parts in the repository models – Acquired data Observations

8. Complementary characteristics of point cloud & image 3D, more accurate cues for geometry & structure Incomplete and noisy Lack depth information Capture the complete object

9. Algorithm Overview Candidate Parts SelectionStructure CompositionPart Conjoining ……

10. Algorithm Overview Candidate Parts SelectionStructure CompositionPart Conjoining ……

11. Candidate Parts Selection Goal: select a small set of candidates for each category Achieved by retrieving parts that fit well some regions

12. Straightforward Solution Search for the best-fit parts over the entire domain – Disregards the semantics associated with each part and the interaction between different parts Unlikely to produce good results! X X X X XXX X

13. Key Fact Man-made objects lie in a low dimensional space – Defined with respect to the relative sizes and positions of shape parts [Ovsjanikov et al. 2011] Employ 3D repository model as a global context – Globally align the models with the input scan first Search in a 3D offset window around the part

14. Part Matching Scheme Geometric fidelity score Geometric contribution score 3D2D edge map Total matching score 3D offset window

15. Candidate Parts Select top K parts with highest score for each category Seat Back Arm Front leg ……

16. Algorithm Overview Candidate Parts SelectionStructure CompositionPart Conjoining ……

17. Structure Composition Goal: compose the underlying structure by identifying a subset of candidate parts

18. Constraints for Promising Compositions Geometric fidelityProximityOverlap   having high scoreno isolated partsminimized intersection

19. Search and Evaluate Search for promising compositions under constraints Globally Evaluate the compositions average geometry fidelity of parts total geometry fidelity total geometry contribution …… optimal composition

20. Algorithm Overview Candidate Parts SelectionStructure CompositionPart Conjoining ……

21. Part Conjoining Problem: the parts are loosely placed together Goal: generate a consistent & complete model

22. Identification of Contact Points Refer to their parent models [Jain et al. 2012]

23. Matching of Contact Points Greedily match nearby contact points Generate auxiliary contact points when necessary auxiliary contact points

24. i j identity scale Global Optimization transformed contact points

25. Results: Chairs 70 repository models, 11 part categories

26. Results: Tables 61 repository models, 4 part categories

27. Results: Bicycles 38 repository models, 9 part categories

28. Results: Airplanes 70 repository models, 6 part categories

29. Results: Creating New Structures

30. Results: Impact of Dataset input data Randomly picking some repository models

31. Summary A bottom-up structure recovery approach – Effectively reuse limited repository models – Automatically compose new structure – Handle single-view inputs by the Kinect system Future work – Multi-view inputs – Include style/functional constraints – Recover Indoor scenes

32. Thank you! Project Page: http://cg.cs.tsinghua.edu.cn/StructureRecovery

33. User-Assisted Preprocessing GrabCut to extract foreground object [Rother et al. 2004]

34. Performance Preprocessing step – 3 minutes user interaction Candidate part selection – 1 minutes for 70 models Structure composition – 2 minutes Part conjoining – 1 second

35. Limitations Lack of suitable parts in the repository

36. Limitations Input with severely missing geometry – The global alignment becomes unreliable