IIIT Hyderabad Learning Semantic Interaction among Graspable Objects Swagatika Panda, A.H. Abdul Hafez, C.V. Jawahar Center for Visual Information Technology, IIIT-Hyderabad, India

IIIT Hyderabad How do we pick objects … Possibility of Damage Objects are removed in an order Automatically find the order (Support Order)using RGBD data. AIM

IIIT Hyderabad Types of support relationships … Support from Below Support from Side Containment

IIIT Hyderabad The framework … Input: RGB Image and Depthmap captured using Kinect. Depthmap RGB Image

IIIT Hyderabad The framework … Segmentation Depthmap RGB Image Segmented Image Segmentation: Over-segmentation using Arbelaez et al. (PAMI’11) Hierarchical segmentation using Hoiem et al. (IJCV’11) Segmentation: Over-segmentation using Arbelaez et al. (PAMI’11) Hierarchical segmentation using Hoiem et al. (IJCV’11)

IIIT Hyderabad The framework … Segmentation Object Detection Object of interest Depthmap RGB Image Segmented Image Detected Region Object Detection: SIFT feature matching. RANSAC applied to discard outliers. Matching segmented regions are merged. Object Detection: SIFT feature matching. RANSAC applied to discard outliers. Matching segmented regions are merged.

IIIT Hyderabad The framework … Segmentation Object Detection Object of interest Depthmap Support Matrix RGB Image Segmented Image Detected Region Support Inference Tree of Support Support Inference: Infers support relationships among the regions and stores in a support matrix.

IIIT Hyderabad The framework … Support Order Segmentation Object Detection Object of interest Depthmap Support Matrix RGB Image Segmented Image Detected Region Support Inference Tree of Support Support order prediction Support Order Prediction: Support relationship captured in a tree. Identification of scenarios to avoid damage. Tree traversal to generate Support Order. Support Order Prediction: Support relationship captured in a tree. Identification of scenarios to avoid damage. Tree traversal to generate Support Order.

IIIT Hyderabad Support Inference Support Matrix Tree of Support Structure class Classifier Segmented Image Support Classifier Floor, Wall, Furniture Graspable objects {O i } Object of interest O(O in ) {O s } Q In Out (O i – pa (O in ), O in ) O in Structure class classifier: Logistic Regression Stochastic Gradient Descent Algorithm 4 classes: Floor, Wall, Furniture, Graspable Objects Structure class classifier: Logistic Regression Stochastic Gradient Descent Algorithm 4 classes: Floor, Wall, Furniture, Graspable Objects

IIIT Hyderabad Support Inference Support Matrix Tree of Support Structure class Classifier Segmented Image Support Classifier Floor, Wall, Furniture Graspable objects {O i } Object of interest O(O in ) {O s } Q In Out (O i – pa (O in ), O in ) O in Support classifier: 3-layer feed-forward neural network classifier Hierarchical support inference Given regions (A, B), predict if B supports A. Support types: from below/ from side/ containment/none Support classifier: 3-layer feed-forward neural network classifier Hierarchical support inference Given regions (A, B), predict if B supports A. Support types: from below/ from side/ containment/none

IIIT Hyderabad Support Inference Support Matrix Tree of Support Structure class Classifier Segmented Image Support Classifier Floor, Wall, Furniture Graspable objects {O i } Object of interest O(O in ) {O s } Q In Out (O i – pa (O in ), O in ) O in Hierarchical Support Inference: Begin with object of interest O. Compare each object with other objects except its parents and grand-parents. Iterate until Q is empty. #comparisons: O(nlogn) Hierarchical Support Inference: Begin with object of interest O. Compare each object with other objects except its parents and grand-parents. Iterate until Q is empty. #comparisons: O(nlogn)

IIIT Hyderabad Illustration of Hierarchical Support Inference … Supported Region Supporting Region Support Type Below Side O in O out Support Matrix

IIIT Hyderabad Features… Close Proximity, f p < 1 At distance, f p > 1 Proximity Significant overlap Less overlap Boundary Ratio Visual Occlusion Side view: actual contact Side view: no contact Depth Boundary ContainmentNo containment Containment Relative Stability Stable object Unstable object

IIIT Hyderabad Beyond pairwise support relations … Case1 : Support In Hierarchy

IIIT Hyderabad Case 2 : Simultaneous support in multiple hierarchy O , Beyond pairwise support relations …

IIIT Hyderabad Case 3 : Containment 1.1 O O O Beyond pairwise support relations …

IIIT Hyderabad O9O9 O3O3 O2O2 O O1O1 O4O4 O7O7 O8O8 O O1O1 O4O4 O2O2 O3O3 O5O5 O6O6 O9O9 O7O7 O8O8 Support Order Prediction… Build Tree of Support O6O6 O5O5 O O1O1 O2O2 RootSupporting Region Supported Region

IIIT Hyderabad O9O9 O3O3 O2O2 O O1O1 O4O4 O7O7 O8O8 O O1O1 O4O4 O2O2 O3O3 O5O5 O6O6 O9O9 O7O7 O8O8 Support Order Prediction… Build Tree of Support Prune the redundant edges O6O6 O5O5

IIIT Hyderabad O9O9 O3O3 O6O6 O2O2 O O5O5 O1O1 O4O4 O7O7 O8O8 O O1O1 O4O4 O2O2 O3O3 O5O5 O6O6 O9O9 O7O7 O8O8 Support Order Prediction… Build Tree of Support Prune the redundant edges Skip the contained objects

IIIT Hyderabad O9O9 O3O3 O6O6 O2O2 O O5O5 O1O1 O4O4 O7O7 O8O8 O O1O1 O4O4 O2O2 O3O3 O5O5 O6O6 O9O9 O7O7 O8O8 Support Order Prediction… Build Tree of Support Prune the redundant edges Skip the contained objects Perform Reverse Level Order Tree Traversal

IIIT Hyderabad O9O9 O3O3 O6O6 O2O2 O O5O5 O1O1 O4O4 O7O7 O8O8 O O1O1 O4O4 O2O2 O3O3 O5O5 O6O6 O9O9 O7O7 O8O8 Support Order Prediction… Build Tree of Support Prune the redundant edges Skip the contained objects Perform Reverse Level Order Tree Traversal Support Order: O 3 → O 9 → O 2 → O 8 → O 7 → O 4 → O 1 → O

IIIT Hyderabad RGBD Dataset… Collected 50 images in clutter using Kinect. Data includes: RGB images, Depth maps, Point clouds Data for individual objects at different orientation. Annotation: Object Label Structure class Label Object instance Label Object Category Label

IIIT Hyderabad Results Illustration of Support from Below

IIIT Hyderabad Results Illustration of Support from Side

IIIT Hyderabad Results Illustration of Containment

IIIT Hyderabad Accuracy of Support Inference InferenceStructure class Inference Support Class Inference TypeTrainingTestingTrainingTesting Ground Truth Regions Segmented Regions

IIIT Hyderabad Conclusion Learned semantic interaction among objects in clutter by support inference and support order prediction Created a RGBD dataset with objects in clutter involving contact and overlap. Future work: Improvement in Segmentation Support Order Prediction using multiple views Support Order Prediction in more complex settings

IIIT Hyderabad Thank You!