1. Interactive Learning of the Acoustic Properties of Objects by a Robot
Jivko Sinapov
Mark Wiemer
Alexander Stoytchev
Iowa State University

2. Motivation: why study sound?
Sound Producing Event
1. Conscious - mention
Gaver: “In particular, such an examination suggests that a given sound provides information about an interaction of materials at a location in an environment.”
Human beings and many animals have the remarkable ability to infer and detect events in the world using acoustic information. This example, adopted from Gaver, illustrates how sound waves allow us to experience events that are beyond the reach of range of the rest of our senses. Based on our experience in life, we can easily tell that a car is approaching just from the sound we hear. Gaver in particular argues that when we hear a non-speech sound, our brain tries to figure out the physical event that caused that generated that sound.
[Gaver, 1993]

3. Motivation (2) Why should a robot use acoustic information?
Human environments are cluttered with objects that generate sounds
Help robot perceive events and objects outside of field of view
Help robot perceive material properties of objects

4. Related Work Krotkov et al. (1996) and Klatzky et al. (2000):
Perception of material using contact sounds.
Learned sound models for tapping aluminum, brass, glass, wood, and plastic (one object per material)
Richmond and Pai (2000)
Robotic platform for measuring contact sounds between robot’s end effector and object surfaces
Models the contact sounds from different materials using spectrogram averaging
[Richmond and Pai, 200]

5. Related Work (2) Torres-Jara, Natale and Fitzpatrick (2005)
Robot taps objects and records spectrogram of sound
Recognize objects using spectrogram matching
Recognized 4 test objects used during training.
Tapping objects
Spectrogram of tapping

6. Our Study
Demonstrate object recognition using acoustic features from interaction
18 Different Objects
3 Different behaviors:
push, grasp, drop
Evaluate different machine learning algorithms

7. Robot and Objects 7-DOF Barret WAM arm with Barret Hand
18 Different objects:
Get rid of figure text, paste list of objects in ppt

8. Robot Behaviors Three behaviors: grasp, push, drop Grasping:
Talk to kevin
Segment it better, skip first few seconds

9. Robot Behaviors
Three behaviors: grasp, push, drop
Pushing:

10. Robot Behaviors
Three behaviors: grasp, push, drop
Dropping:

11. Sound Feature Representation
Step 1: segment sound wave during interaction:
Step 2: Compute Discrete Fourier transform (DFT) of sound wave:
Step 3: Compute 2-D histogram of DFT matrix using block averaging:
Put labels for time and frequency on dft picture
5 frequency bins
Frequency
Time
10 temporal bins

12. Object Recognition using Acoustic Properties of Objects
Problem: given robot’s behavior and detected sound features from interaction, predict the object.
Example:
Behavior:
Sound Features:
Object Class:
grasp

13. Problem Formulation Let be the set of exploratory behaviors
Let be the set of objects,
Let be a data point such that:
, , and
For each behavior learn a model that can estimate

14. Learning Algorithms K-NN Support Vector Machine (SVM) Bayesian Network
Simple instance-based algorithm
Uses Euclidean distance function
Support Vector Machine (SVM)
Discriminative approach, uses Kernel trick
Bayesian Network
Probabilistic graphical model
Sound Features are discretized into bins
Add picture for each algorithm

15. Learning Algorithms: k-NN, SVM, and Bayesian Network
k-NN: memory-based learning algorithm
With k = 3:
neighbors
neighbors
Test point ?
Therefore,
Pr(red) = 0.66
Pr(blue) = 0.33

16. Learning Algorithms: k-NN, SVM, and Bayesian Network
Support Vector Machine: discriminative learning algorithm
Finds maximum margin hyperplane that separates two classes
Uses Kernel trick to map data points into a feature space in which such a hyperplane exists [

17. Learning Algorithms: k-NN, SVM, and Bayesian Network
Bayesian Network: a probabilistic graphical model
Full power of statistical modeling and inference
Learning: learns both the structure of the network and the parameters (conditional probability tables)
Numerical features are discretized into bins A B C D E

18. Using Multiple Behaviors
Given trained models , ,
Given novel sounds , , from behaviors performed on the same object
Assign prediction to object class that maximizes:
Fix cropping from paper

19. Evaluation
6 trials recorded with each of the 18 objects with each of the 3 behaviors
Leave-one-out cross-validation
Compared performance of learning algorithms as well as behaviors
Performance Measure:

20. Results
Chance accuracy = 1/18 = %

21. Confusion Matrix for model Mpush using Bayesian Network
Predicted → 4 - 2 5 1 6 3
Perfect classification and no false positives for:
Add pictures of rest of the objects

22. Confusion Matrix for model Mcombined using Bayesian Network
Predicted → 6 - 1 5
Conclusion:
The errors made by models Mgrasp, Mpush and Mdrop are uncorrelated.

23. Learning rate of algorithms
Compare performance of the model Mgrasp as a function of dataset size for:
k-NN
Support Vector Machine
Bayesian Network
Get rid of figure caption,
Add next figure after end of talk in case

24. Learning Rate per Behavior with Bayesian Network

25. Summary and Conclusions
Accurate acoustic-based object recognition with 18 objects and 3 behaviors
Using multiple behaviors improves recognition regardless of learning algorithm
Bayesian network performed best with given feature representation
Grasping and Pushing interaction produces sound features that are more informative of the object than Dropping
“improves recognition regardless of the type of learning algorithm being used”

26. Future Work Scaling up:
Increase number of objects
Vary object and robot pose
Autonomous interaction
Use unsupervised learning to form object sound categories
More powerful feature representations
Temporal features (i.e. periodicity) of sounds
Use models to detect events in the world performed by others (humans or other robots)
Change to arial