1. Interactive Object Recognition Using Proprioceptive Feedback
Taylor Bergquist, Connor Schenck, Ugonna Ohiri,
Jivko Sinapov, Shane Griffith and Alexander Stoytchev
Developmental Robotics Lab
Iowa State University, Ames, IA, U.S.A.
{knexer | cschenck | ucohiri |
jsinapov | shaneg | alexs iastate.edu

2. What is Proprioception?
“It is the sense that indicates whether the body is moving with required effort, as well as where the various parts of the body are located in relation to each other.”
- Wikipedia

3. The Importance of Proprioception
Full
Empty

4. The Importance of Proprioception
Hard
Soft

5. Exploratory Behaviors in Children
[Power, 2000]

6. Lifting: Weight, Gravity, Effort[

7. Shaking: Weight, Inertia, Contents

8. Dropping: Gravity and Physics

9. Pushing: Physics and Objects

10. Crush: Compliance, Flexibility

11. Five Exploratory Behaviors
Lift:
Crush:
Shake:
Push:
Drop:

12. Robot Platform

13. Objects Used in the Experiments
50 household objects
Different materials: metal, paper, plastic, wood, etc.
Some objects have contents inside of them (e.g., pill bottle)
All are graspable by the Barrett Hand

15. Torque Data Preprocessing
Joint torque data was sampled and recorded at 500 Hz using the robot’s API
The raw data was filtered to remove outliers
J1…J7

16. Feature Extraction Joint torque data (500 samples/second in ℝ7)
Some way is needed to compress the data
Discretize to obtain a sequence Pi of tokens from a finite alphabet
J1 … J7
SOM

17. Training the Self-Organizing Map

18. Object Recognition Model
Problem Formulation
Proprioceptive sequence
Object Recognition Model

19. Predictive Models k-NN and global alignment
Emphasizes temporal structure of the proprioceptive sequences
Multinomial Naïve Bayes and n-gram
Emphasizes distributional structure of the sequences

20. k-NN k-NN: memory-based learning algorithm
With k = 3:
neighbors
neighbors
Test point ?
Therefore,
Pr(red) = 2/3
Pr(blue) = 1/3
Spend less time on k-NN; emphasize sequence comparison, add question slide with more detail/notes
Uses Needleman-Wunsch global alignment as a measure of similarity between two sequences
“A guided tour to approximate string matching”, by Navarro, G. in ACM Computing Surveys, v.33, 2001

21. Multinomial Naïve Bayes (N-grams)
Sample Sequence
1-grams
2-grams 10 11 9 1 8 6 5 3 4

22. Multinomial Naïve Bayes (2-grams)
Sample Sequence
2-grams 1 8 6 5 3 4
Animate this
where n(wt, di) is the number of occurrences of word wt V in the feature vector di.

23. Multinomial Naïve Bayes (2-grams)
Sample Sequence
2-grams 1 8 6 5 3 4
Animate this
where n(wt, di) is the number of occurrences of word wt V in the feature vector di.

24. Evaluation Ten-fold cross validation
2500 total interactions, 250 per fold
Evaluated on recognition accuracy, where:
Chance accuracy: 1/50 = 2%
# correct predictions
% Accuracy =
x 100
# total predictions

25. Recognition from a Single Behavior

26. Recognition from Multiple Behaviors
How to combine predictions from multiple behaviors?
assume that all behaviors are equally useful
weight behaviors according to their accuracy

27. Object Recognition Results
What happens when the robot uses information from multiple interactions with the same object?

28. Multimodal Recognition
# Behaviors
Single
Multiple
# Modalities

29. Multimodal Recognition
# Behaviors
Single
Multiple
# Modalities
This paper (proprioception only)

30. Multimodal Recognition
# Behaviors
Single
Multiple
# Modalities
Follow up paper (proprioception + audio)
“Interactive Object Recognition Using Proprioceptive and Auditory Feedback”
Submitted to IEEE Robotics and Automation Magazine (under review).

31. Audio Data (for the same dataset)
Audio data was recorded during data collection and transformed into spectrograms
Raw Sound:
Discrete Fourier Transform:

32. Multimodal Training (Two SOMs)
Training a self-organizing map (SOM) using sampled joint torques:
Training an SOM using sampled frequency distributions:

33. Multimodal Feature Extraction
Discretization of joint-torque records using a trained SOM
Discretization of the DFT of a sound using a trained SOM
is the sequence of activated SOM nodes over the duration of the interaction
is the sequence of activated SOM nodes over the duration of the sound

34. Multimodal Recognition
Proprioception sequence
Audio sequence
Proprioceptive Recognition Model
Auditory Recognition Model
Weighted Combination

35. Multimodal Recognition Results

36. Related Work Audition Proprioception
Kubus, Kröger, and Wahl, (3 objects)
Richmond and Pai, (4 objects)
Torres-Jara, Natale, and Fitzpatrick, 2005 (4 objects)
Sinapov, Weimer, Stoytchev, ICRA (36 objects)
Proprioception
Natale, Metta, and Sandini, (7 objects)

37. Accuracy vs. Number of Objects

38. Multiple Modalities + Multiple Behaviors
Combination of behaviors and modalities compensates for this effect

39. Conclusions and Future Work
Robots can and should use proprioception as a source of information about the world
Better results can be obtained by combining multiple interactions and multiple modalities
Future Work:
More complex behaviors and more objects
Integrate proprioception with more modalities (vision, haptics, etc.)

40. Take Home Message Number of objects recognition accuracy
Number of behaviors recognition accuracy
Number of modalities recognition accuracy

41. Thank you
w00t
Any questions?

42. THE END

43. Object Recognition Results
The dotted lines show the best and worst case for the weighted and unweighted combination.

44. Recognition from Multiple Behaviors
How to combine predictions from multiple behaviors?
Previous work (Sinapov, Weimer, Stoytchev, ICRA 2009):
assume that all behaviors are equally useful
Choose the object Oi that maximizes ∑B P(Oi | PB), where B is an exploratory behavior performed on the object.
This assumption fails to hold for this work;
weight behaviors according to accuracy instead
Choose Oi to maximize ∑B P(Oi | PB) * wB, where wB is the estimated reliability of the model given a sequence from behavior B.