A project to develop software to assist people with autism to recognise, understand and express emotions through facial expressions, voice and body gestures

Autism & Emotions Difficulties recognizing and understanding mental states: – Their own – In other people Aspects of difficulty include: – Facial expressions – Vocal intonation – Body gestures and body language – The integration of the above in context Difficulties in expressing emotions non-verbally, in all the above aspects. Emotion recognition and expression are not learned spontaneously >>> need to be taught explicitly. Risk of social exclusion for individuals with autism is significantly increased without these skills.

Project Introduction A project to develop interactive games to assist people with autism to recognise, understand and express emotions through facial expressions, voice and body gestures Primarily for children aged 5 to 10, high functioning Can serve teenagers and young adults (with proper accommodations) Can serve more populations that are socially challenged, such as: ADHD, Hearing impaired

Project Partners Prof. Simon Baron-Cohen, head of the Autism Research Centre at Cambridge University, UK Prof. Sven Bölte, Karolinska Institutet, Sweden Dr. Ofer Golan, Bar-Ilan University, Israel Prof. Peter Robinson, head of Cambridge Computer Lab, UK Prof. Björn Schuller, Technische Universität München, Germany Prof. Antonio Camurri, University of Genova, Italy Autism-Europe (AE) Executive comitee

Project Method & Rationale The ASC-Inclusion software will: Analyse the children’s facial expressions, vocal intonations and gestures (standard microphones and webcams); Train children to recognise and express emotions through Interactive lessons, fun games, animations, video clips; Personalise settings according to children’s individual needs; Support therapists and parents with professional information, reports on children’s progress and forums to interact with other carers.

How the software will work Innovation meets intervention The program will consist of 3 state-of-the-art subsystems for: Facial expressions Vocal intonation Body gestures The subsystems will be integrated together in an online virtual world platform to provide the user with a comprehensive and motivating learning experience.

How the software will work Facial expressions

How the software will work Vocal intonation

How the software will work Body gestures

Environment – Narrative Environment narrative: Complete fun learning tasks to earn more fun stuff! The players are researchers in an international research camp located in the jungle. They research human behaviour and emotional expression.

Avatar: Create your character Click on different shapes of mouth, eyes, eyebrows etc. to express your mood

Main area: The research Center Main area: The research Center Get a task, see your research status, meet the mentor, use research tools

Motivational Elements: Home Motivational Elements: Home Take a canoe to your home or visit a friend’s home, design your home

Motivational Elements: Goodies Motivational Elements: Goodies Use the virtual coins you earned by completing “research missions” (=learning tasks) to purchase items for your home, avatar and collections

Motivational Elements: Games Motivational Elements: Games Play cool games using your virtual money, meet friends

Games: Facial expression Mahjong: Attention to details Initial level: Match happy faces. Advanced level: Match different people who feel the same.

Games: Emotional tone of voice Recognise what the speakers feel according to their voice

Games: Body language hints Edit your own movie using the hints you’ve learned

Games: Cross-modal training Match picture to sound: Listen to Timmy. Click on the picture of the character who feels like Timmy.

Interactive books: Social scenes Enjoy funny stories on everyday social situations Create your own ending, Invent more comics stories with the characters

Testing Game A highly rewarding learning: Answer the questions and win a million coins!