1. Real-Time Hierarchical Scene Segmentation and Classification Andre Uckermann, Christof Elbrechter, Robert Haschke and Helge Ritter John Grossmann

2. Real-Time Scene Segmentation/Classification Purpose: Correctly segment various objects in a scene for the end goal of classifying or identifying each segmented object. Important for robots to be able to classify objects in real-time. o Collision Detection o Don’t want to harm humans o Grasping correct objects o Pointing gestures to specific objects or areas

3. Existing Work Classify based on features from existing Object Model library Limited to existing object models Segment objects in a scene based on generic rules Color, Smoothness, Convexity Not good with ambiguous situations Use World knowledge and heuristic or learned segment relations Predict connectivity of object surface patches based on learned relations (too slow) Employ Heuristic rules to connect surface patches in a weighted graph

4. Improving the Existing Work Create a full hierarchy of grouping hypotheses Tree-like connectivity structure of the scene Results in spatial scene analysis grouping spatially connected objects Higher Level Decision Process to Determine Optimal Grouping Hypothesis Implement world knowledge Change level of granularity on task specific manner

5. Model-Free Segmentation Pre-Segmentation into Surface Patches Separate scene surfaces into uniform regions 1.Three-stage smoothing (median, temporal, Gaussian) 2.Calculation of surface normals from cross products 3.Detection of edges from angle between normal and distance of adjacent points 4.Connected component analysis to assign unique IDs to all surface regions 5.Assignment of edge points to closest surface region

6. Connectivity Graph Creation Criteria 1.Cut-Free Adjacency 2.Co-planarity 3.Similar Curvature of Surface Pairs

7. Cut-Free Adjacency Consider adjacent surfaces If one surface cuts another surface they most likely are not part of the same object

8. Co-Planarity Combine object parts separated by occlusion Check if both surfaces are separated by occlusion or by space Match shapes of non-adjacent surface patches Co-Planar surfaces must span a common plane

9. Curvature Matching

10. Handling Remaining Edge Points Some larger edge-point blobs were not assigned to a neighbored surface patch during pre-segmentation Normally small, separate objects Separate object “blobs” by a region growing algorithm Consider maxima in depth image as “separation-clefts” All new found edges added as nodes to connectivity graph https://www.youtube.com/watch?v=gI7c9RC7gKg

11. Hierarchical Segmentation Takes connectivity graph created in previous step, and attempts to find hierarchical clustering of graph nodes. Groups of increasing connectivity strength 1.Assignment of Initial Edge Weights (Cutting cost) 2.Fine-Tuning Edge Weights

12. Initializing Edge Weights

13. Fine-Tuning Edge Weights

14. Hierarchical Segmentation Tree Each least cut creates a new branch Repeat until all edges cut Each leaf is a single surface The branching nodes are potential grouping hypotheses

15. Hierarchical Classification Fast NN-based classifier traverses tree from top down Computes class probability distribution of the object hypothesis at each branching node Peaks of classification confidence will indicate salient entities of objects or groupings of objects Segmentation hierarchy efficiently connects: 1.Generic low-level regularities about structures governing segmentation (surfaces and edges) 2.Specialized, high level feature correlations indicative of objects and object parts or configurations (handle and mug, door and door knob)

16. Nearest-Neighbor Classifier Uses a bag of features model For each feature type, there is a normalized distance function between a pair of features Features which exactly match have 0 distance The distance between feature vectors is the max of all individual feature distances Adding new features and classes can be performed very fast Main features used are a Size, Elongation and Color

17. Feature: Size

18. Feature: Elongation

19. Feature: Color

20. Applying the Classifier https://www.youtube.com/watch?v=gI7c9RC7gKg