1. Indoor Segmentation and Support Inference from RGBD Images Nathan Silberman, Derek Hoiem, Pushmeet Kohli, Rob Fergus

2. Goal: Infer Support for Every Region Lamp Supported by Nightstand Nightstand Supported by Floor

3. Goal: Infer Support for Every Region

4. Why infer physical support? Interacting with objects may have physical consequences!

5. Why infer physical support: Recognition Object on top of desk Object hanging from file cabinet

6. Why infer physical support: Recognition

7. Working with RGB+Depth Captured with Microsoft Kinect Restricted to Indoor Scenes

8. NYU Depth Dataset Version 2.0 Collected new NYU Depth Dataset Much larger than NYU Depth 1.0 – 464 Scenes – 1449 Densely Labeled frames – Over 400,000 Unlabeled frames – Over 800 Semantic Classes – Full videos available Larger variation in scenes Dense Labels much higher quality http://cs.nyu.edu/~silberman/datasets/nyu_depth_v2.html

9. High Quality Semantic Labels Bed Pillow 1 Pillow 2 Headboard Nightstand Lamp Window Dresser Picture 1 Wall 1 Wall Picture 3 Doll 1 Doll 2 Floor Picture 2 Pillow 3

10. High Quality Support Labels Support from behind Support from belowSupport from hidden region

11. Segmentation Support Inference RGBD Image

12. Input Scene Parsing

13. Major Surfaces Surface Normals Aligned Point Cloud Major Surfaces Surface Normals Aligned Point Cloud Input Scene Parsing

14. Input Major Surfaces Surface Normals Aligned Point Cloud Major Surfaces Surface Normals Aligned Point Cloud Segmentation Scene Parsing

15. Segmentation Scheme similar to: Recovering Occlusion Boundaries from a Single Image D. Hoiem, A.N. Stein, A.A. Efros, and M. Hebert, ICCV 2007. Hierarchical Segmentation

16. Segmentation Scheme similar to: Recovering Occlusion Boundaries from a Single Image D. Hoiem, A.N. Stein, A.A. Efros, and M. Hebert, ICCV 2007. Hierarchical Segmentation

17. Segmentation Scheme similar to: Recovering Occlusion Boundaries from a Single Image D. Hoiem, A.N. Stein, A.A. Efros, and M. Hebert, ICCV 2007. Hierarchical Segmentation

18. Segmentation Scheme similar to: Recovering Occlusion Boundaries from a Single Image D. Hoiem, A.N. Stein, A.A. Efros, and M. Hebert, ICCV 2007. Hierarchical Segmentation

19. Segmentation Scheme similar to: Recovering Occlusion Boundaries from a Single Image D. Hoiem, A.N. Stein, A.A. Efros, and M. Hebert, ICCV 2007. Hierarchical Segmentation

20. Input Major Surfaces Surface Normals Aligned Point Cloud Major Surfaces Surface Normals Aligned Point Cloud Segmentation Scene Parsing

21. Input Major Surfaces Surface Normals Aligned Point Cloud Major Surfaces Surface Normals Aligned Point Cloud Segmentation Support Inference

22. Segmentation Support Inference RGBD Image

23. Modeling Choice #1 All objects supported by a single object except – Floor requires no support. 4 4 2 2 3 3 1 1 4. Floor 3. Wall 1. Chair 2. Picture (Inverted) Tree Representation Image Regions

24. Modeling Choice #2 All objects are either supported by another region in the image OR a hidden region.

25. Modeling Choice #2 All objects are either supported by another region in the image OR a hidden region. Deoderant supported by counter

26. Modeling Choice #2 All objects are either supported by another region in the image OR a hidden region. Cabinet supported by hidden region

27. Modeling Choice #3 Every object is either supported from below or from behind.

28. Modeling Choice #3 Every object is either supported from below or from behind. Deoderant supported from below

29. Modeling Choice #3 Every object is either supported from below or from behind. Mirror supported from behind

30. Modeling Support: Structure Classes ‘Structure Classes’ encode high level support prior knowledge (1) Ground (2) Furniture (3) Prop or (4) Structure

31. Modeling Support Goal: For each region in regions, infer: 1.Supporting region 2.Support Type 3.Structure class

32. Modeling Support Goal: For each region in regions, infer: 1.Supporting region 2.Support Type 3.Structure class The formal problem per image:

33. Local Support Local Structure ClassPrior - supporting region - support type - structure class Joint Energy Factorizes into three terms:

34. comes from logistic regressor trained on pairwise features Local Support Energy 1 1 2 2 - supporting region - support type - structure class

35. Local Structure Class Energy from logistic regressor trained on features from each individual region - supporting region - support type - structure class

36. A region’s structure class helps predict its support. Structure Furniture Structure Floor OR Prior (1/4): Transitions - supporting region - support type - structure class

37. Supporting regions should be nearby OR Prior (2/4): Support Consistency - supporting region - support type - structure class

38. A region requires no support if and only if its structure class is ‘floor’ Prior (3/4): Ground Consistency - supporting region - support type - structure class

39. A region is unlikely to be the floor if another floor region is lower than it OR Prior (4/4): Global Ground Consistency Floor Prop - supporting region - support type - structure class

40. Integer Program Formulation Relaxed to Linear Program

41. Experiments

42. Evaluating Support Accuracy = # of Correctly Labeled Support Relationships # of Total Labeled Support Relationships Evaluation with features extracted from: Regions from Ground Truth LabelsRegions from Segmentation

43. Baseline #1: Image Plane Rules Heuristic: look at neighboring regions for support

44. Baselines #2: Structure Class Rules Heuristic: Support is deterministic given Structure Classes Floor Furniture Prop Structure

45. Baselines #3: Support Classifier Use only the output of support classifier

46. Evaluating Support (Regions from Ground Truth Labels) Examples of Manually Labeled Regions

47. Evaluating Support (Regions from Ground Truth Labels) Examples of Manually Labeled Regions

48. Results Ground Truth Regions Floor Furniture Prop Structure

49. Results Ground Truth Regions Correct Prediction

50. Results Ground Truth Regions Correct Prediction Incorrect Prediction

51. Results Ground Truth Regions Correct Prediction Incorrect Prediction Support from below

52. Results Ground Truth Regions Correct Prediction Incorrect Prediction Support from behind Support from below

53. Results Ground Truth Regions Correct Prediction Incorrect Prediction Support from behind Support from below Support from hidden region

54. Results Ground Truth Regions Correct Prediction Incorrect Prediction Support from behind Support from below Support from hidden region

55. Results Ground Truth Regions Correct Prediction Incorrect Prediction Support from behind Support from below Support from hidden region

56. Evaluating Support (Regions from Segmentation) Examples of Regions from Segmentation

57. Results Automatically Segmented Regions Correct Prediction Incorrect Prediction Support from behind Support from below Support from hidden region

58. Results Automatically Segmented Regions Correct Prediction Incorrect Prediction Support from behind Support from below Support from hidden region

59. Conclusion Algorithm for inferring Physical Support Novel Integer Program Formulation 3D Cues for segmentation Dataset: – http://cs.nyu.edu/~silberman/datasets/nyu_depth_v2.html Code: – http://cs.nyu.edu/~silberman/projects/indoor_scene_ seg_sup.html