A Search-Classify Approach for Cluttered Indoor Scene Understanding Liangliang Nan 1, Ke Xie 1, Andrei Sharf 2 1 SIAT, China 2 Ben Gurion University, Israel
Digitalization of indoor scenes Indoor scenes from Google 3D Warehouse
Acquisition of indoor scenes
Goal Scene understanding
Challenges Clutter –Densely populated –Arbitrary arrangements Partial representation –Occlusions Complex geometry
Classification & Segmentation Two interleaved problems –What are the objects? –Where are the objects? Chicken-egg problem –Classification needs segmentation –Segmentation needs a prior
Our solution Search –Propagate / accumulate patches Classify –Query classifier to detect object
Related Work Indoor scenes ( This Session ) – [Fisher et al. 2012] [Shao et al. 2012] [Kim et al. 2012] Semantic relationship – [Fisher et al. 2010, 2011] Recognition using depth + texture (RGB-D) – [Quigley et al.2009], [Lai and Fox 2010] Outdoor classification – [Golovinskiy et al. 2009] Semantic labeling – [Koppula et al. 2011] Controlled region growing process
Our search-classify idea
Method overview Training Search-Classify
Point cloud features –Height-size ratio of BBox –Aspect ratio of each layer –Bottom-top, mid-top size ratio –Change in COM along horizontal slabs BhBh BdBd BwBw
Classifier Handle missing data –Occlusion Random decision forest –Efficient multi-class classifier Trained with both scanned and synthetic data –Manually segmented and labeled –510 chairs –250 tables –110 cabinets –40 monitors etc. [Shotton et al. 2008, 2011]
Search-Classify Starts from seeds –Random patch triplets –Remove seeds with low confidence Accumulating neighbor patches –Highest classification confidence Stop condition –Steep decrease in classification confidence Seed
Segmented - classified objects problems –Overlap, outliers, ambiguities etc. Refinement –Outliers = patches with large distance Segmentation refinement by template fitting
Template deformation Different styles for each class Predefined scalable parts Templates can deform [Xu et al. 2010]
Template deformation Different styles for each class Predefined scalable parts Templates can deform [Xu et al. 2010]
Fitting via template deformation ConfidenceFitting errorBest fitting Best matching template –One-side Euclidean distance from points to template
Results and discussion
Scalability test with varied object density 0 (25) 1 (45) 5 (60)
Results and discussion Comparison Lai et al Ours
Limitation Upward assumption –Features –Template fitting
Future work Contextual information
Thank you