1. Learning complex visual concepts

2. detecting hands direction of gaze
Difficult to detect, appear early, important for subsequent learning of agents, goals, interactions

3. Why are hands difficult to detect?
Van Gogh
Kirchner
multiple appearances
small and inconspicuous

4. Selectivity to hands appears early in infancy
using a head camera to study visual experience
“overall… hands were in view and dynamically acting on an object in over 80% of the frames”
Yoshida & Smith 2008
What makes hands learnable by humans?

5. Motion, hand as ‘mover’ (7 months old)
Saxe, Carey (2006) The perception of causality in infancy

6. Early sensitivity to special motion types
high sensitivity to motion in general
(detecting motion, motion segmentation, tracking)
specific sub-classes of motion: self-motion, passive, and ‘mover’
A specific motion is highly indicative of hands

7. Detecting ‘mover’ events Ullman, Harari, Dorfman (2012)
• mover event = moving image region causing a stationary region to move or change after contact
(E) Is with short tracking. F are falses, we get about 65% yield. Some complete movies are without ‘movers’.
• simple and primitive, prior to objects or figure-ground segmentation
• ‘mover’ as an innate teaching signal for learning the
concept of a hand

8. Training videos
Movies of scenes, people moving, manipulating objects, moving hands
‘Mover’ events are detected in all movies and used for training

9. ‘Mover’ events extracted from videos
High fraction of detected mover events involve hands
(90% recall, 65% precision)
(b) With tracking, infants track the motion
Extract many images of objects entering cells of mover events & use images to train a hand object classifier

10. Hand detection in still images
These are detections following the learning of a hand detector from movers. Here they are detected in static images, not motion.
Trained classifier mainly detected hands in object manipulation scenes

11. Continued learning Two detection algorithms:
Detect hands with greater range of appearance by tracking
Detect more hands by the body context

12. Hand by surrounding context
face shoulder upper-arm lower-arm hand
Amano, Kezuka, Yamamoto, 2004
Slaughter, Heron-Delaney, 2010
Slaughter, Neary, 2011 12

13. Start from bottom row – detections
Start from bottom row – detections. In C, the top is by shape, the bottom by context. (d) by shape, (e) by context (a) performance
a – comparison of precision-recall graphs between the learned models in different learning phases.
b – comparison of precision-recall graphs between the learned models of combined appearance+context, no context and no tracking.
(c)-(e) detection results: (c) – example test images and detection results (color temperature indicate detection confidence), (d) – new-appearances (non-manipulating) detections, (e) – detections of difficult appearances
appearance
context

14. On the Apparent Direction of Eyes in a Portrait
Gaze direction
W.H. Wollaston (1824)
On the Apparent Direction of Eyes in a Portrait
Very early seminal study, Scaife & Bruner Nature 1975, got some gaze following for 2-4 months. The experiment always started with directed eye contact between the infant and the ‘subject’. The ‘subject’ head turn was large – 90 degrees.
D’Entremont, B., Hains, S.M.J., and Muir, D.W. (1997). A demonstration of gaze following in 3- to 6-month-olds. Infant Behav. Dev. 20, 569–572. Like in the original Scaife & Bruner, they engaged the infant (direct looking at them, talking to them). Got reliable turns as early as 3 months. 90 deg in each direction, towards a lion-puppet.
infants follow the gaze of others
starts at 3-6 months and continues to develop
head orientation first, eye cues later
important in development of communication & language
modeling mainly head direction

15. Gaze cues are subtle and inconspicuous
mover events supply
the teaching signal

16. Gaze extraction Training Testing model humans
Use facial appearances and gaze directions derived from mover events as a training set to learn direction of gaze