Action-Grounded Push Affordance Bootstrapping of Unknown Objects Barry Ridge & Aleš Ude Humanoid and Cognitive Robotics Lab Department of Automation, Biocybernetics & Robotics IROS Nov 4th, 2013 1

Introduction

Grounding 3-D Shape Features in Push Actions RGB image + segmented 3-D point cloud Shape primitives fitted to object parts Learn object push affordances when objects are pushed from different positions & directions. Dense 3D point clouds from modern depth sensors capture object surface shape with reasonably high accuracy. Object models can be formed dynamically by fitting simple shape primitives to object parts in such point clouds. Shape & position of object parts can be used to learn & predict affordances when objects are pushed from different positions & directions.

Point Cloud Library (PCL)

Calibrated Point Clouds & Push Trajectories

Action Coordinate Frame Object point clouds before/after interaction + hand push trajectory Point clouds + trajectory transformed to push action coordinate frame Transform object point clouds before & after interaction to push action coordinate frame. We define the push action coordinate frame w.r.t. the push contact point & direction: - Origin centred at object contact point. - Frame orientation found by fitting line to push trajectory using least squares linear regression.

Action-Grounded Shape Features Fitting Planes to Object Parts Before Interaction Fitting Planes to Object Parts After Interaction Point Clouds + Trajectory transformed to Push Action Coordinate Frame Divide object point cloud into part cells: 1 cell for overall point cloud. 2 cells by evenly splitting point cloud by its extent in X-dimension. - 2 cells by evenly splitting point cloud by its extent in Y-dimension. - 2 cells by evenly splitting point cloud by its extent in Z-dimension. Find point centroids + Fit planes in each cell. Derive 35-dimensional object shape feature vector.

Action-Grounded Shape Features

Learning Goal Cross-View Co-Occurrences Input Feature Space (Pre-Push) Output Feature Space (Post-Push)

Self-supervised Multi-view Learning Vector Quantization

Class Discovery In order to discover affordance classes in the output layer, we perform unsupervised clustering of the prototypes using the X-means algorithm to find the optimal k* clustering...

Class Prediction ...then finally, class labels may be assigned to input layer prototypes using the following labeling function: ...and input samples may then be classified by matching them to their nearest-neighbour input layer prototype.

Experiments 5 Push Types 4 Affordance Types 5 push types: push forward from centre of object, top of object, bottom of object, left of object and right of object. 4 affordance types: forward translation, forward topple, left rotation and right rotation.

Experiments 134 object push trials performed on 5 household objects in various positions and orientations on table surface. 10-Fold Incremental Cross Validation. Comparison with incremental supervised learning methods based on vector quantization, including: GLVQ (Generalized Learning Vector Quantization) GRLVQ (Generalized Relevance Learning Vector Quantization) SRNG (Supervised Relevance Neural Gas)

Results: Short-Term Training

Results: Long-Term Training

Results: Reduced Feature Set, Short-Term Training

Results: Reduced Feature Set, Long-Term Training

Conclusions Main contribution: an action grounded 3-D visual feature descriptor to be used for bootstrapping object affordances when objects are unknown. When used in combination with a self-supervised learning algorithm, this feature descriptor is effective at facilitating both affordance class discovery and prediction in an online learning setting with a number of initially unknown objects and object affordances. A feature relevance determination extension to the self-supervised algorithm was also shown to boost affordance class prediction results by emphasizing the discriminative contribution of particular features within the descriptor.