Lasting Relationship Cornerstones of Social Robotics in HRI Teamwork Social LearningSocial Intelligence Interdependence Transparent Communication Cognitive Compatibility Perspective Taking User Studies, Psychology & Social Development

Being a body with a mind in a world of like bodies with like minds yields multi-modal associations the self and the appearance of other agents the behaviors (events in time) of the self and the behaviors of other agents the internal workings of oneself and other agents Human Social Development

Simulation Theory & Mindreading We use our own cognitive system off- line to simulate others (R. Gordon) Cognitive processes are dual-use Generate own actions from our mental states Infer the mental states responsible others actions by stepping into their shoes We use our own cognitive system off- line to simulate others (R. Gordon) Cognitive processes are dual-use Generate own actions from our mental states Infer the mental states responsible others actions by stepping into their shoes

Neural Mechanisms of Mindreading Dual Use SELF-TOM Evidence of overlapping brain regions involved in SELF and TOM SELF: meta- representational cognitive capacity to apply a self perspective TOM: mindreading capacity to model someone elses state of mind Evidence of overlapping brain regions involved in SELF and TOM SELF: meta- representational cognitive capacity to apply a self perspective TOM: mindreading capacity to model someone elses state of mind TOM+SELF+ TOM-SELF+ TOM+SELF- TOM-SELF- Vogeley et al, Neuro Image 14, (2001)

Simulation Theory & Social Learning Andrew Meltzoff posits that we LEARN to simulate via early infant imitation-based interactions with adults Social Learning Implications: The experience of others can be mapped to self --- enabling the development of learning by observation, imitation, social referencing, etc. Andrew Meltzoff posits that we LEARN to simulate via early infant imitation-based interactions with adults Social Learning Implications: The experience of others can be mapped to self --- enabling the development of learning by observation, imitation, social referencing, etc.

Architecture

Imitation & Mirror Systems Dual Use Recognition/Production Visual- Motor Xform Motor Out Motor In Vision In Interpret Observed Actions wrt Motor Repertoire Meltzoff&Moore AIM Model Synthesize Action from Motor Repertoire Motor Knowledge Breazeal et al, Artificial Life (2005)

Social Structure of Imitation Human Engages robot in imitation game by mimicking the robots facial expressions Social Interaction Affords Learning Body Maps: how robots face (body) maps onto social others Mirror System: Dual use of motor representations for recognition of action in others and production of own action Ability to mimic others actions Human Engages robot in imitation game by mimicking the robots facial expressions Social Interaction Affords Learning Body Maps: how robots face (body) maps onto social others Mirror System: Dual use of motor representations for recognition of action in others and production of own action Ability to mimic others actions

Development of Social Referencing Social Referencing (~12 mos) Understand meaning of affective signal from adult (~6 mos) Shared attention to understand referent of adult (~9 mos) Associate that appraisal with beliefs and memory (attitudes) toward that referent Interact with novel object accordingly Learn how to appraise novel objects (~18 mos) Social Referencing (~12 mos) Understand meaning of affective signal from adult (~6 mos) Shared attention to understand referent of adult (~9 mos) Associate that appraisal with beliefs and memory (attitudes) toward that referent Interact with novel object accordingly Learn how to appraise novel objects (~18 mos)

Social Referencing Challenges #1 Understand the affective meaning of anothers expression

Evoke Affect from Observed Expression Exploit Bi-Directional Body-Affect Pathways to learn affective meaning of observed facial expression Body- affect loop Imitate Facial expression

Empathic Mechanism Learn association to evoke empathic response in robot

Recognition of Vocal Affective Intent Four cross-cultural contours of infant- directed speech A. Fernald Exaggerated prosody matched to infants innate responses Four cross-cultural contours of infant- directed speech A. Fernald Exaggerated prosody matched to infants innate responses time (ms) pitch, f (kHz) o approval Thats agood bo-o-y! No nobaby. time (ms) pitch, f (kHz) o prohibition Can you get it? Can you get it? time (ms) pitch, f (kHz) o attention time (ms) pitch, f (kHz) o MMMMOh, honey. comfort

Evidence for Fernald-like Contours for (Cute) Robot Directed Speech

Valence and Arousal in Feature Space prohibition & high-energy neutral attention & approval soothing & low-energy neutral Breazeal & Aryananda, Autonomous Robots (2002)

Results, Multiple Languages Objective scorer classifies as strong, medium, weak Good overall performance for strong instances Random perf. = 20% very good for caregivers good for naive subjects Acceptable misclassifications minimal confusion of valence some confusion of arousal

Responding to Vocal Affect

Social Referencing Challenges #1 Understand the affective meaning of anothers expression #2 Understand the referent that their emotive reaction is about #1 Understand the affective meaning of anothers expression #2 Understand the referent that their emotive reaction is about

Saliency & Visual Attention (Adapted from J. Wolfe VGS 2.0) Visual attention allows robot to look at salient objects/events around it --- sets focus of robots attention inhibit reset Frame Grabber Eye Motor Control Top down, task-driven influences wwww skin tonehabituationmotioncolor attention

Seek face – high skin gain, low color saliency gain Looking time 80% face, 20% block Seek toy – low skin gain, high saturated-color gain Looking time 28% face, 72% block Looking Preference Internal influences bias how salience is measured The robot is not a slave to its environment Prefers behaviorally relevant stimuli

Directing Attention in Interaction Breazeal & Scassellati, IJCAI 1999

Socially Directed Attention Add social cues as stimuli that explicitly contributes to saliency in addition to environmental pop outs Gaze to PointingGaze to Head Pose

Shared Attention: Aboutness Extend attention model to distinguish focus of attention (what is salient right now) verses referential focus (what this interaction is about) Keep track of relative-looking-time of objects in scene that robot and human look at Hypothesis: object with highest relative looking time is the shared object referent. Extend attention model to distinguish focus of attention (what is salient right now) verses referential focus (what this interaction is about) Keep track of relative-looking-time of objects in scene that robot and human look at Hypothesis: object with highest relative looking time is the shared object referent.

Social Referencing Challenges #1 Understand the affective meaning of anothers expression #2 Understand the referent that their emotive reaction is about #3 Use their appraisal to bootstrap its own appraisal of novel objects #1 Understand the affective meaning of anothers expression #2 Understand the referent that their emotive reaction is about #3 Use their appraisal to bootstrap its own appraisal of novel objects

Object Beliefs Incoming perceptual features are bound into beliefs about objects Object beliefs are tracked perceptual histories over time: What this object was like recently... + What this object is like right now... + What we expect this object to be like soon. Incoming perceptual features are bound into beliefs about objects Object beliefs are tracked perceptual histories over time: What this object was like recently... + What this object is like right now... + What we expect this object to be like soon.

Object Beliefs & Working Memory Object templates are long-term prototypical representations of objects Sets expectations of what this thing typically is like… Learning Associations Bind labels to object templates to learn names of objects Bind affect to object templates to learn attitude toward objects (i.e., somatic markers) Object templates are long-term prototypical representations of objects Sets expectations of what this thing typically is like… Learning Associations Bind labels to object templates to learn names of objects Bind affect to object templates to learn attitude toward objects (i.e., somatic markers)

Social Referencing Thomaz et al, Ro-Man 2005

Goal: robots that can learn in the real-world from anyone Most people dont have experience with Machine Learning techniques, but they bring a lifetime of experience with social learning interactions Social-cognitive skills Social interaction and socio- affective-cognitive skills does heavy lifting of framing the learning problem as a collaborative process. If done correctly, improves learning process and performance for human and robot. Robot learns what is intended in a transparent way. Summary: HRI meets ML Taking Learning Experience Seriously

Students Matt Berlin Andrew Brooks Jesse Gray Guy Hoffman Our collaborators Stan Winston Studio Cory Kidd Jeff Lieberman Andrea Thomaz Dan Stiehl Contributors Funding TTT & DL Media Lab Consortia ONR YIP DARPA MARS, BICA Toyota Thank You!

For more info robotic.media.mit.edu Designing Sociable Robots (2002) MIT Press robotic.media.mit.edu Designing Sociable Robots (2002) MIT Press