1. Learning about Objects
by Experiment
– Paul Fitzpatrick –
MIT Artificial Intelligence Laboratory

2.     machine perception algorithms robust perception training data
parameter
tweaking  

3. self-training perceptual system
algorithms
opportunities
training data  
robust
perception
more
algorithms  

4. making opportunities
In robotics, vision is often used to guide manipulation
But manipulation can also guide vision
Important for…
Experimentation progressing when perception is ambiguous
Correction recovering when perception is misleading
Development bootstrapping when perception is primitive

5. a simple scene? Edges of table and cube overlap
Color of cube and table are poorly separated
Cube has misleading surface pattern
Maybe some cruel grad-student
faked the cube with paper,
or glued it
to the table

6. active segmentation
Object boundaries are not always easy to detect visually

7. active segmentation
Object boundaries are not always easy to detect visually
Solution: Cog sweeps through ambiguous area

8. active segmentation
Object boundaries are not always easy to detect visually
Solution: Cog sweeps through ambiguous area
Resulting object motion helps segmentation

9. active segmentation
Object boundaries are not always easy to detect visually
Solution: Cog sweeps through ambiguous area
Resulting object motion helps segmentation
Robot can learn to recognize and segment object without further contact

10. active segmentation

11. active segmentation

12. integrated behavior

13. segmentation examples

14. segmentation examples
table
car

15. boundary fidelity cube car bottle ball
0.05
0.1
0.15
0.2
0.25
0.3
0.04
0.08
0.12
0.16
cube
car
bottle
ball
(square root of) second Hu moment
measure of anisiotropy
measure of size (area at poking distance)

16. affordance exploitation
poking
object segmentation
edge catalog
object detection
(recognition, localization,
contact-free segmentation)
affordance exploitation
(rolling)
manipulator detection
(robot, human)

17. poking
object segmentation

18. opportunities for segmentation
robot, poking
(Paul Fitzpatrick,
Giorgio Metta)
wearable, illumination
(Charlie Kemp)
camera, periodicity
(Artur Arsenio)

19. segmentation on a wearable
System detects when wearer reaches for an object, requests wearer to hold it up, then illuminates it

20. segmentation by watching a human
System detects periodic motion – waving, tapping, etc. – and extracts seed points for segmentation

21. poking
object segmentation
edge catalog

22. sampling oriented regions

23. sample samples

24. most frequent samples 1st 21st 41st 61st 81st 101st 121st 141st 161st

25. Red = horizontal Green = vertical
some tests
Red = horizontal Green = vertical

26. natural images 00
22.50
900
22.50
450
22.50
450
22.50

27. poking object segmentation edge catalog object detection
(recognition, localization,
contact-free segmentation)

28. geometry+appearance Advantages: more selective; fast
angles + colors + relative sizes
invariant to scale, translation,
In-plane rotation
Advantages: more selective; fast
Disadvantages: edges can be occluded; 2D method
Property: no need for offline training

29. in operation

30. yellow on yellow

31. open object recognition
robot’s
current
view
recognized
object (as seen
during poking)
pokes,
segments
ball
sees ball,
“thinks” it is cube
correctly differentiates
ball and cube

32. open object recognition

33. manipulator detection
poking
object segmentation
edge catalog
manipulator detection
(robot, human)
object detection
(recognition, localization,
contact-free segmentation)

34. manipulator recognition

35. affordance exploitation object segmentation
poking
affordance exploitation
(rolling)
object segmentation
edge catalog
manipulator detection
(robot, human)
object detection
(recognition, localization,
contact-free segmentation)

36. objects roll in different ways
a bottle
it rolls along its side
a toy car
it rolls forward
a toy cube
it doesn’t roll easily
a ball
it rolls in
any direction

37. affordance exploitation

38. mimicry test Demonstration by human Mimicry in similar situation
Mimicry when object is rotated
Invoking the object’s natural rolling affordance
Going against the object’s natural rolling affordance

39. mimicry test

40. affordance exploitation object segmentation
poking
affordance exploitation
(rolling)
object segmentation
edge catalog
manipulator detection
(robot, human)
object detection
(recognition, localization,
contact-free segmentation)

41. referring to objects

42. beyond objects familiar activities
use constraint of familiar activity to
discover unfamiliar entity used within it
reveal the structure of unfamiliar activities by tracking familiar entities into and through them
familiar entities (objects, actors, properties, …)

43. learning activity structure
/ylo/
/lft/
/grin/
/ryt/
/ylo/
/lft/
/grin/
/ryt/
/ylo/
/lft/
/grin/
/ryt/
/ylo/
/lft/
/grin/
/ryt/
/ylo/
/lft/
/grin/
/ryt/
/ylo/
/lft/
/grin/
/ryt/
/ylo/
/lft/
/grin/
/ryt/
/ylo/
/lft/
/grin/
/ylo/
/lft/
/ylo/
/lft/
/ryt/
Extending Vocabulary
/lft/
/yn/
/ryt/
Extending Vocabulary
/lft/
/ylo/
/ryt/
/grin/
/lft/
/ylo/
/ryt/

44. conclusions active segmentation (through contact) appearance catalog
(for oriented features)
open object recognition
for correction, enrollment
affordance recognition
(for rolling)
open speech recognition
(for isolated words)
virtuous circle of development
learning about and through activity