1. Object Lesson: Discovering and Learning to Recognize Objects Object Lesson: Discovering and Learning to Recognize Objects – Paul Fitzpatrick – MIT CSAIL USA

2. Paul Fitzpatrick, MIT CSAIL, Humanoids 2003 robots and learning  Robots have access to physics, and physics is a good teacher  Physics won’t let you believe the wrong thing for long  Robot perception should ideally integrate experimentation, or at least learn from (non- fatal) mistakes forward seems safe…

3. Paul Fitzpatrick, MIT CSAIL, Humanoids 2003 a challenge: object perception  Object perception is a key enabling technology  Many components: –Object detection –Object segmentation –Object recognition  Typical systems require human-prepared training data; can we use autonomous experimentation?

4. Paul Fitzpatrick, MIT CSAIL, Humanoids 2003 a challenge: object perception  Object perception is a key enabling technology  Many components: –Object detection –Object segmentation –Object recognition  Typical systems require human-prepared training data; can we use autonomous experimentation? Fruit detection

5. Paul Fitzpatrick, MIT CSAIL, Humanoids 2003 a challenge: object perception  Object perception is a key enabling technology  Many components: –Object detection –Object segmentation –Object recognition  Typical systems require human-prepared training data; can we use autonomous experimentation? Fruit segmentation

6. Paul Fitzpatrick, MIT CSAIL, Humanoids 2003 a challenge: object perception  Object perception is a key enabling technology  Many components: –Object detection –Object segmentation –Object recognition  Typical systems require human-prepared training data – can’t adapt to new situations autonomously Fruit recognition

7. Paul Fitzpatrick, MIT CSAIL, Humanoids 2003 talk overview  Perceiving through experiment –Example: active segmentation  Learning new perceptual abilities opportunistically –Example: detecting edge orientation –Example: object detection, segmentation, recognition  An architecture for opportunistic learning –Example: learning about, and through, search activity

8. Paul Fitzpatrick, MIT CSAIL, Humanoids 2003 talk overview  Perceiving through experiment –Example: active segmentation  Learning new perceptual abilities opportunistically –Example: detecting edge orientation –Example: object detection, segmentation, recognition  An architecture for opportunistic learning –Example: learning about, and through, search activity

9. Paul Fitzpatrick, MIT CSAIL, Humanoids 2003 active segmentation  Object boundaries are not always easy to detect visually  Solution: Cog sweeps arm through ambiguous area  Any resulting object motion helps segmentation  Robot can learn to recognize and segment object without further contact

10. Paul Fitzpatrick, MIT CSAIL, Humanoids 2003 active segmentation  Detect contact between arm and object using fast, coarse processing on optic flow signal  Do detailed comparison of motion immediately before and after collision  Use minimum-cut algorithm to generate best segmentation 12345 987610

11. Paul Fitzpatrick, MIT CSAIL, Humanoids 2003 active segmentation

12. Paul Fitzpatrick, MIT CSAIL, Humanoids 2003 active segmentation  Not always practical!  No good for objects the robot can view but not touch  No good for very big or very small objects  But fine for objects the robot is expected to manipulate Head segmentation the hard way!

13. Paul Fitzpatrick, MIT CSAIL, Humanoids 2003 listening to physics  Active segmentation is useful even if robot normally depends on other segmentation cues (color, stereo)  If passive segmentation is incorrect and robot fails to grasp object, active segmentation can use even clumsy collision to get truth  Seems silly not to use this feedback from physics and keep making the same mistake table car

14. Paul Fitzpatrick, MIT CSAIL, Humanoids 2003 talk overview  Perceiving through experiment –Example: active segmentation  Learning new perceptual abilities opportunistically –Example: detecting edge orientation –Example: object detection, segmentation, recognition  An architecture for opportunistic learning –Example: learning about, and through, search activity

15. Paul Fitzpatrick, MIT CSAIL, Humanoids 2003 talk overview  Perceiving through experiment –Example: active segmentation  Learning new perceptual abilities opportunistically –Example: detecting edge orientation –Example: object detection, segmentation, recognition  An architecture for opportunistic learning –Example: learning about, and through, search activity

16. Paul Fitzpatrick, MIT CSAIL, Humanoids 2003 opportunistic learning  To begin with, Cog has three categories of perceptual abilities: –Judgements it can currently make (e.g. about color, motion, time) –Judgements it can sometimes make (e.g. boundary of object, identity of object) –Judgements it cannot currently make (e.g. counting objects)  With opportunistic learning, the robot takes judgements in the sometimes category and works to promote them to the can category by finding reliable correlated features that are more frequently available –Example: analysis of boundaries detected through motion yields purely visual features that are predictive of edge orientation –Example: assuming a non-hostile environment (some continuity in time and space) segmented views of objects can be grouped and purely visual features inferred that are characteristic of distinct objects

17. Paul Fitzpatrick, MIT CSAIL, Humanoids 2003 training a model of edge appearance  Robot initially only perceives oriented edges through active segmentation procedure  Robot collects samples of edge appearance along boundary, and builds a look- up table from appearance to orientation angle  Now can perceive orientation directly  This is often built in, but it doesn’t have to be

18. Paul Fitzpatrick, MIT CSAIL, Humanoids 2003 most frequent samples 1 st 21 st 41 st 61 st 81 st 101 st 121 st 141 st 161 st 181 st 201 st 221 st

19. Paul Fitzpatrick, MIT CSAIL, Humanoids 2003 some tests Red = horizontal Green = vertical

20. Paul Fitzpatrick, MIT CSAIL, Humanoids 2003 natural images 00000000 90 0 45 0  45 0  22.5 0

21. Paul Fitzpatrick, MIT CSAIL, Humanoids 2003 talk overview  Perceiving through experiment –Example: active segmentation  Learning new perceptual abilities opportunistically –Example: detecting edge orientation –Example: object detection, segmentation, recognition  An architecture for opportunistic learning –Example: learning about, and through, search activity

22. Paul Fitzpatrick, MIT CSAIL, Humanoids 2003 on to object detection…

23. Paul Fitzpatrick, MIT CSAIL, Humanoids 2003 on to object detection… look for this……in this

24. Paul Fitzpatrick, MIT CSAIL, Humanoids 2003 on to object detection…

25. Paul Fitzpatrick, MIT CSAIL, Humanoids 2003 on to object detection…

26. Paul Fitzpatrick, MIT CSAIL, Humanoids 2003 on to object detection… geometry alone geometry + color

27. Paul Fitzpatrick, MIT CSAIL, Humanoids 2003 other examples

28. Paul Fitzpatrick, MIT CSAIL, Humanoids 2003 other examples

29. Paul Fitzpatrick, MIT CSAIL, Humanoids 2003 other examples

30. Paul Fitzpatrick, MIT CSAIL, Humanoids 2003 just for fun look for this……in this result

31. Paul Fitzpatrick, MIT CSAIL, Humanoids 2003 real object in real images

32. Paul Fitzpatrick, MIT CSAIL, Humanoids 2003 yellow on yellow

33. Paul Fitzpatrick, MIT CSAIL, Humanoids 2003 multiple objects camera image response for each object implicated edges found and grouped

34. Paul Fitzpatrick, MIT CSAIL, Humanoids 2003 attention

35. Paul Fitzpatrick, MIT CSAIL, Humanoids 2003 first time seeing a ball sees ball, “thinks” it is cube correctly differentiates ball and cube robot’s current view recognized object (as seen during poking) pokes, segments ball

36. Paul Fitzpatrick, MIT CSAIL, Humanoids 2003 open object recognition

37. Paul Fitzpatrick, MIT CSAIL, Humanoids 2003 talk overview  Perceiving through experiment –Example: active segmentation  Learning new perceptual abilities opportunistically –Example: detecting edge orientation –Example: object detection, segmentation, recognition  An architecture for opportunistic learning –Example: learning about, and through, search activity

38. Paul Fitzpatrick, MIT CSAIL, Humanoids 2003 talk overview  Perceiving through experiment –Example: active segmentation  Learning new perceptual abilities opportunistically –Example: detecting edge orientation –Example: object detection, segmentation, recognition  An architecture for opportunistic learning –Example: learning about, and through, search activity

39. Paul Fitzpatrick, MIT CSAIL, Humanoids 2003 physically-grounded perception active segmentation object segmentation edge catalog object detection (recognition, localization, contact-free segmentation) affordance exploitation [with Giorgio Metta] (rolling) manipulator detection (robot, human)

40. Paul Fitzpatrick, MIT CSAIL, Humanoids 2003 socially-grounded perception

41. Paul Fitzpatrick, MIT CSAIL, Humanoids 2003 socially-grounded perception

42. Paul Fitzpatrick, MIT CSAIL, Humanoids 2003 opportunistic architecture: a virtuous circle familiar activities familiar entities (objects, actors, properties, …) use constraint of familiar activity to discover unfamiliar entity used within it reveal the structure of unfamiliar activities by tracking familiar entities into and through them

43. Paul Fitzpatrick, MIT CSAIL, Humanoids 2003 a virtuous circle poking, chatting car, ball, cube, and their names discover car, ball, and cube through poking; discover their names through chatting

44. Paul Fitzpatrick, MIT CSAIL, Humanoids 2003 a virtuous circle poking, chatting, search car, ball, cube, and their names follow named objects into search activity, and observe the structure of search

45. Paul Fitzpatrick, MIT CSAIL, Humanoids 2003 learning about search

46. Paul Fitzpatrick, MIT CSAIL, Humanoids 2003 a virtuous circle poking, chatting, search car, ball, cube, and their names follow named objects into search activity, and observe the structure of search

47. Paul Fitzpatrick, MIT CSAIL, Humanoids 2003 a virtuous circle poking, chatting, searching car, ball, cube, toma, and their names discover novel object through poking, learn its name (e.g. ‘toma’) indirectly during search

48. Paul Fitzpatrick, MIT CSAIL, Humanoids 2003 finding the toma

49. Paul Fitzpatrick, MIT CSAIL, Humanoids 2003 conclusion: why do this?  The quest for truly flexible robots  Humanoid form is general-purpose, mechanically flexible  Robots that really live and work amongst us will need to be as general-purpose and adaptive perceptually as they are mechanically