Classifying Objects as New or Learned with Convolutional Networks and SGD By Kevin Xiong and Evan Phibbs Mentored by Yufei Wang

Introduction The turtlebot runs on Robotic Operating System (ROS). ROS allows us to interface with the turtlebot’s motors and Xbox kinect’s camera and depth data.

Goal Our goal was for the turtlebot to be placed anywhere in the room, move around the room to find objects around it, recognize each object as known or new and learn the object if it was new, as an attempt in open-ended learning.

Libraries We wrote our program entirely in python using two libraries: Caffe: A convolutional neural network library Sklearn: a general-purpose machine learning library, which includes a linear svm

Methods The robot captures depth data and rgb data from the turtlebot’s kinect. a binary mask is created from the depth data such that each 1 corresponds to an object pixel and each 0 to a background pixel in order to reduce the effect of the background on classification rgb data is multiplied by the mask and set as input into the CNN

Methods (continued) The convolutional network’s last hidden layer activations are used as input into an svm classifier The distances to the separating hyperplanes of each input into the svm are used as inputs into another svm classifier to determine whether an object is new or known.

Methods (continued) rgb depth mask rgb mask linear svm classifie r clas s decision_function method (distance to each hyperplane) new or old linear svm classifie r eighth layer activations (4096d vector) output Caffe convolutional network

Hurdles We encountered many limitations while using the turtlebot The turtlebot’s movements are not accurate The turtlebot can only rotate, move forward, or move backward. Kinect has very noisy depth data and is not aligned with camera rgb data

Results We tested on 4 types of tea, 2 types of cubes, a bottle, magic eight ball, and a stuffed animal cat.

Video Demonstration

Improvements Our program could be improved with the following: a more robust way of circling the objects an algorithm for moving about a room to ensure no objects are left undiscovered Combining depth data and object partitioning algorithms in order to create a finer, more accurate mask

Improvements (continued) relying solely on object partitioning in order to recognize known objects and remove them from the scene, leaving only new objects to be focused on and trained using an object mask such that background pixels are set to random color noise