Work in Collaboration with Isbister (NYU-Poly), Jeffery Ventrella, Leslie Bishko, Jacki Morie EMIIE Lab

Nonverbal Behavior: From real to virtual Figure from A. Vinciarelli, M. Pantic, H. Bourlard, and A. Pentland, “Social signals, their function, and automatic analysis: a survey,” ICMI, 2008.

 Enhance Communication  Express Emotions and Reactions  Social Signals : agreement, confusion, alignment, turn-taking  Applications:  Virtual Meetings (IBM, …)  Online Education  Performances  Social Hang Out Nonverbal Behavior: Importance

CVEs, Virtual Worlds support for Social Signals

Social Com Software: Nonverbal Behavior (Tiny Chat)

Social Com Software: Nonverbal Behavior (Mingleverse)

Social Com Software: Nonverbal Behavior (Shami et al.) N. S. Shami, L.-T. Cheng, S. Rohall, A. Sempere, and J. Patterson, "Avatars Meet Meetings: Design Issues in Integrating Avatars in Distributed Corporate Meetings," in Group, 2010.

Funded by AFRL

Research Goal  Understand how people use Social Signaling in Second Life:  Proximity  Gaze  Appearance  Gestures  Facial Expressions

Social Signaling Study in Second Life  28 Participants in 7 sessions  10 males 18 females, average age 40 (range 20-60)  Belonging to the following communities: Experimental Performance Arts 1 group, 6 participants Theatrical Arts2 groups 14 participants Building / Animating1 group 3 participants Music Composition1 group 4 participants Role Play1 group 2 participants Educators1 group 2 participants

Social Signaling Study in Second Life (SL)  Participant Recruitment  Online Recruiting in SL and Ning sites  More than 50 friends made visiting 40 locations  Specific community and location targeted  Attended events & distributed bulletins via and SL  Participants also made referrals to their friends  What sort of activity are they doing?  Focus Group Show and Tell on SS topics derived from lit review  Sessions took place on Lakamaka Islands (owned by SRI) ▪ Privacy and performance concerns wearing scripts ▪ Participants joined the Lakamaka Community David’s Avatar

Social Signaling Study in Second Life  Data Collection  Background Survey  Semi-Structured Interview (during focus group)  Real Time Proximity and Chat logging Sample Screenshots on Lakamaka

Focus Group – Sample Questions  Physical Appearance  How do you make your character different from the other characters in the virtual world?  Gesture and Posture  Do you use gestures as part of your online communication?  Face and Eye Behavior  Do you change facial expressions and direct your avatar gaze?  Voice  How do you use voice and text chat to communicate?  Space and Environment  Do you have any sense of a “personal bubble” around your avatar?  Technological Affordances  What scripts do you use if any?

Show video

 No standard way of puppeteering avatars  Few research studies:  what behavioral cues are important  how they are used

Show video

 Participants:  groups of 2-3  11 participants  Situation Setup: an argument  Video Tapped the Session  Logged all button pushing and interaction  Total: minutes

 "Being able to visually interact with other Skype users through emotions is great, when you do not wish to use you camera or do not have access to one."  "bring the avatar back to the screen having fun making the avatar move around"  "Can’t see the point of the system while there is webcam and chat options.”

 “It difficult to concentrate for both moving the avatar around and talking at the same time.”  “As time goes on the content of the conversation becomes more important, the visuals becomes unnecessary.”

 Context is very important  Important Signals:  Thumbs Up and down  Turn taking  Applause  Proximity  Gaze

 Social Signaling is Important  Identified some important behavioral cues depend on context  It is still unclear:  The right UI and control  The right semantic buttons  Conscious vs. non-conscious

 FLAME (1998) – modeling Emotions and Learning in believable characters  ELE (2003) – Expressive Lighting Engine: system that allocates and adjusts lights to accommodate interaction and aesthetic goals  Mirage (2002) – Interactive Narrative Architecture using theatre acting and filmmaking principles  Player modeling (2002) – figuring out style of play and adapting an interactive narrative accordingly

 Adaptive Systems  Believable characters  Visual Design and Presentation  User Experience Research  Games User Research Group (IGDA)  Assassin’s Creed II Study (Ubisoft)  Façade studies  Triangulation of eye tracking, physio, game metrics (Electronic Arts)  Co-op patterns (Bardel Entertainment)  Virtual world monitoring through visualization and Metrics (God Mode Games and Relic Entertainment)  Laban + Delsarte for modeling expressive behavoior