1. Presentation by Sasha Beltinova
A Survey of Autonomous Human Affect Detection Methods for Social Robots Engaged in Natural HRI
Derek McColl, Alexander Hong, Naoaki Hatakeyama, Goldie Nejat, Beno Benhabib
Presentation by Sasha Beltinova

2. How everyone’s title slides look
The Office reference...hopefully at least some people in the class have watched it!

3. A Survey of Autonomous Human Affect Detection Methods for Social Robots Engaged in Natural HRI
This is essentially the agenda. Unpack the title to inform what the presentation will be about.

4. A Survey of Autonomous Human Affect Detection Methods for Social Robots Engaged in Natural HRI
Emotional State

5. A Survey of Autonomous Human Affect Detection Methods for Social Robots Engaged in Natural HRI
Emotional State
Robots that interact with humans using natural human communication methods like speech or body language

6. A Survey of Autonomous Human Affect Detection Methods for Social Robots Engaged in Natural HRI
Emotional State
Robots that interact with humans using natural human communication methods like speech or body language
Human Robot Interaction

7. A Survey of Autonomous Human Affect Detection Methods for Social Robots Engaged in Natural HRI
Emotional State
Robots that interact with humans using natural human communication methods like speech or body language
Human Robot Interaction
Non-acted

8. A Survey of Autonomous Human Affect Detection Methods for Social Robots Engaged in Natural HRI
A survey of how social robots recognize humans’ emotional state in natural, non-acted interactions with humans.

9. HRI

10. What is HRI? Human-Robot Interaction (HRI):
Development of robots that engage humans in various scenarios
Study of human-robot interactions during these scenarios
Requires interaction with a physical robot

11. Social HRI
Robots interact using human modalities like speech, body language, and facial expressions
Correctly determining humans’ social cues is necessary for responding appropriately
Affect detection should not interfere with or influence human’s behavior during HRI
Should be able to interpret and respond to natural (non-acted) displays of emotion
Affective HRI:
Collaborative
Assistive
Mimicry
General or multi-purpose

12. Human Affect

13. Affect: Categorical
A set of discrete states, e.g.happiness, surprise, fear, disgust, anger, and sadness
Most commonly used: Ekman’s Facial Action Coding System (FACS)
Position of facial AU correspond to distinct facial expressions of emotions
Strength: affective states are easily distinguishable
Weakness: Affect not included in the model cannot be categorized

14. Affect: Dimensional
Affective state is a continuous spectrum across several dimensions
Most commonly used: two-dimensional valence-arousal model
Strength: can account for all possible affective states and their variations
Weakness: sometimes hard to distinguish one affect from another

15. Methods of Affect Recognition during HRI
Single input mode:
Facial affect recognition
Body language based affect recognition
Voice based affect recognition
Recognition of affective physiological signals
Multimodal inputs: combination of the above

16. Affect Recognition: Single Mode of Input

17. Facial Affect Recognition
Facial expressions are a natural way to communicate opinions, intentions, and emotions in face-to-face human interactions.
To emulate empathy, social robots need to be able to:
Interpret and recognize human affective state based on humans’ facial expressions
Display (or simulate the display of) their own affect
Express their perspective intentions to humans
Most systems consist of :
2D onboard cameras to detect facial features
Classifiers to estimate affect

18. Facial Affect Recognition
Collaborative HRI Scenario:
Children play chess with iCat
2D webcam to detect facial features
iCat determined affect and responded appropriately
Children reported perceived quality of interaction
Robots displaying empathy is important in interactions with humans
Affect categorized as positive, neutral, or negative valence
Affect model consisted of:
probability of a smile (calculated using SVMs nd facial geometric features from 2D and 3D facial landmarks extracted using Seeing Machine’s faceAPI
eye gaze,
game state,
game evolution
Children in several control groups, ones in adaptive empathic control group believed iCat knew how they felt

19. Facial Affect Recognition
Mimicry HRI Scenario:
AIBO robot dog mimics human facial expression
External camera to capture person’s facial expression
Facial expression recognized from streaming video using time-series representation of the tracked facial actions
AIBO mimics same expression with LED display
Facial Expression detection method had 80% accuracy
AIBO was able to mimic human expression only with slight delay

21. Body Language Based Affect Recognition
Body language like arm positions or head posture can communicate affective state in social interactions.
Identifying body language displays can help social robots identify and influence human behavior in a variety of HRI tasks.
Many systems include:
Kinetic sensors to capture and model person’s trunk and arm poses during interaction
Interpretation and classification systems for body poses and gestures

22. Body Language Based Affect Recognition
Assistive HRI:
Brian2.1 used to provide social assistance to elderly in long-term care facility during meal eating to encourage engament in eating activity
Kinect sensor mounted on robot’s chest to identify body language displays
Features extracted included head, trunk, arm position to classify valence and arousal
Robot prompted humans to eat or drink items in one-on-one interaction
Kinetic sensor used to detect body language in non-meal actions (manipulating utensils or cup etc).
Valence and arousal calculated from 3D data
2D video data used for human baseline coding
77.9% recognition rate for valence and 93.6% recognition rate for arousal

23. Body Language Based Affect Recognition

24. Voice Based Affect Recognition
People often communicate affect with their voices.
Identifying vocal affect is important for effective bi-directional communication with humans
Systems usually include:
Onboard microphones to detect emotions from human voice
Software to identify emotional state
Some vocal intonation features:
Voice intensity and pitch
Speech rate
Word articulation
Vocal intonation features like voice intensity, speech rate, or word articulation directly correspond to specific affective states.

25. Voice Based Affect Recognition
Collaborative HRI:
Cyton robot arm picked up object based on user affect
Voice signal recorded from microphone headset
Robot arm guided by human voice intonation picked desired objects
Unfamiliar users found easy to use
high arousal is usually associated with cognitive effort and attentional processes
high unpredictability and surprise are usually related to attentional shifts
High arousal -> robot should be attentive
High predictability -> robot is doing well
Low predictability -> human is uncertain and robot should hesitate or switch targets

26. Recognition of Affective Physiological Signals
Physiological responses like heart rate, skin conductance, breathing rate, or pupil dialation can indicate emotional state.
Systems usually include wearable tech like EKG and skin-conductance sensors

27. Recognition of Affective Physiological Signals
Humans watched a industrial robot arm perform tasks
Robot arm planned a safe path or a standard path and executed
Executed at various speeds
Person’s arousal measured
Estimated arousal levels strongly correlated with self-reported anxiety levels
The safe path planner is similar to the potential field method, with the addition of a danger criterion, comprising of factors that affect the impact force during a collision between the robot and the human, which is minimized along the path.
pick location was specified to the right and away from the subject, and the place location was directly in front and close to the subject
lower levels of anxiety and surprise, and higher levels of calm, for the safe planned paths
anxiety response for the fast pick and place trajectory was found to be significantly higher (α=0.05, student t-test) for the PF planned path

30. Affect Recognition: Multimodal Input

31. Multimodal Affect Recognition
Benefits:
Alternative ways to determine affect when one modality fais
Complementary and diverse information can increase system robustness and performance
Challenges:
More difficult to acquire and process

32. Discussion

33. Current State of the World
Autonomous detection of emotional state of humans is important for good HRI
Forms of affective interactions include:
Collaborative HRI
Assistive HRI
Mimicry HRI
Mutli-purpose HRI
Categorical model of human affect is most commonly used
Using classifiers like SVMs, HMMs, and NNs
Facial expressions is the most popular input mode for detecting affect
Typically done with 2D camera

34. Challenges
Current systems can identify only a small number of affective states.
Humans experience and display a large variety of emotions during social interactions.
Facial affect recognition, the most commonly used modality, relies on 2D camera in most cases.
Performs well only when humans face is directly in front of the camera.
Many studies carried out in short game-like interaction scenarios.
Affect recognition techniques should be tested in long-term studies across wide variety of tasks and scenarios.

35. Don’t shave it for later