Real-Time Hierarchical Scene Segmentation and Classification Andre Uckermann, Christof Elbrechter, Robert Haschke and Helge Ritter John Grossmann

Real-Time Scene Segmentation/Classification Purpose: Correctly segment various objects in a scene for the end goal of classifying or identifying each segmented object. Important for robots to be able to classify objects in real-time. o Collision Detection o Don’t want to harm humans o Grasping correct objects o Pointing gestures to specific objects or areas

Existing Work Classify based on features from existing Object Model library Limited to existing object models Segment objects in a scene based on generic rules Color, Smoothness, Convexity Not good with ambiguous situations Use World knowledge and heuristic or learned segment relations Predict connectivity of object surface patches based on learned relations (too slow) Employ Heuristic rules to connect surface patches in a weighted graph

Improving the Existing Work Create a full hierarchy of grouping hypotheses Tree-like connectivity structure of the scene Results in spatial scene analysis grouping spatially connected objects Higher Level Decision Process to Determine Optimal Grouping Hypothesis Implement world knowledge Change level of granularity on task specific manner

Model-Free Segmentation Pre-Segmentation into Surface Patches Separate scene surfaces into uniform regions 1.Three-stage smoothing (median, temporal, Gaussian) 2.Calculation of surface normals from cross products 3.Detection of edges from angle between normal and distance of adjacent points 4.Connected component analysis to assign unique IDs to all surface regions 5.Assignment of edge points to closest surface region

Connectivity Graph Creation Criteria 1.Cut-Free Adjacency 2.Co-planarity 3.Similar Curvature of Surface Pairs

Cut-Free Adjacency Consider adjacent surfaces If one surface cuts another surface they most likely are not part of the same object

Co-Planarity Combine object parts separated by occlusion Check if both surfaces are separated by occlusion or by space Match shapes of non-adjacent surface patches Co-Planar surfaces must span a common plane

Curvature Matching

Handling Remaining Edge Points Some larger edge-point blobs were not assigned to a neighbored surface patch during pre-segmentation Normally small, separate objects Separate object “blobs” by a region growing algorithm Consider maxima in depth image as “separation-clefts” All new found edges added as nodes to connectivity graph

Hierarchical Segmentation Takes connectivity graph created in previous step, and attempts to find hierarchical clustering of graph nodes. Groups of increasing connectivity strength 1.Assignment of Initial Edge Weights (Cutting cost) 2.Fine-Tuning Edge Weights

Initializing Edge Weights

Fine-Tuning Edge Weights

Hierarchical Segmentation Tree Each least cut creates a new branch Repeat until all edges cut Each leaf is a single surface The branching nodes are potential grouping hypotheses

Hierarchical Classification Fast NN-based classifier traverses tree from top down Computes class probability distribution of the object hypothesis at each branching node Peaks of classification confidence will indicate salient entities of objects or groupings of objects Segmentation hierarchy efficiently connects: 1.Generic low-level regularities about structures governing segmentation (surfaces and edges) 2.Specialized, high level feature correlations indicative of objects and object parts or configurations (handle and mug, door and door knob)

Nearest-Neighbor Classifier Uses a bag of features model For each feature type, there is a normalized distance function between a pair of features Features which exactly match have 0 distance The distance between feature vectors is the max of all individual feature distances Adding new features and classes can be performed very fast Main features used are a Size, Elongation and Color

Feature: Size

Feature: Elongation

Feature: Color

Applying the Classifier