1. Work in Collaboration with Isbister (NYU-Poly), Jeffery Ventrella, Leslie Bishko, Jacki Morie EMIIE Lab

2. Nonverbal Behavior: From real to virtual Figure from A. Vinciarelli, M. Pantic, H. Bourlard, and A. Pentland, “Social signals, their function, and automatic analysis: a survey,” ICMI, 2008.

3.  Enhance Communication  Express Emotions and Reactions  Social Signals : agreement, confusion, alignment, turn-taking  Applications:  Virtual Meetings (IBM, …)  Online Education  Performances  Social Hang Out Nonverbal Behavior: Importance

4. CVEs, Virtual Worlds support for Social Signals

5. Social Com Software: Nonverbal Behavior (Tiny Chat)

6. Social Com Software: Nonverbal Behavior (Mingleverse)

7. Social Com Software: Nonverbal Behavior (Shami et al.) N. S. Shami, L.-T. Cheng, S. Rohall, A. Sempere, and J. Patterson, "Avatars Meet Meetings: Design Issues in Integrating Avatars in Distributed Corporate Meetings," in Group, 2010.

8. Funded by AFRL

9. Research Goal  Understand how people use Social Signaling in Second Life:  Proximity  Gaze  Appearance  Gestures  Facial Expressions

10. Social Signaling Study in Second Life  28 Participants in 7 sessions  10 males 18 females, average age 40 (range 20-60)  Belonging to the following communities: Experimental Performance Arts 1 group, 6 participants Theatrical Arts2 groups 14 participants Building / Animating1 group 3 participants Music Composition1 group 4 participants Role Play1 group 2 participants Educators1 group 2 participants

11. Social Signaling Study in Second Life (SL)  Participant Recruitment  Online Recruiting in SL and Ning sites  More than 50 friends made visiting 40 locations  Specific community and location targeted  Attended events & distributed bulletins via e-mail and SL  Participants also made referrals to their friends  What sort of activity are they doing?  Focus Group Show and Tell on SS topics derived from lit review  Sessions took place on Lakamaka Islands (owned by SRI) ▪ Privacy and performance concerns wearing scripts ▪ Participants joined the Lakamaka Community David’s Avatar

12. Social Signaling Study in Second Life  Data Collection  Background Survey  Semi-Structured Interview (during focus group)  Real Time Proximity and Chat logging Sample Screenshots on Lakamaka

13. Focus Group – Sample Questions  Physical Appearance  How do you make your character different from the other characters in the virtual world?  Gesture and Posture  Do you use gestures as part of your online communication?  Face and Eye Behavior  Do you change facial expressions and direct your avatar gaze?  Voice  How do you use voice and text chat to communicate?  Space and Environment  Do you have any sense of a “personal bubble” around your avatar?  Technological Affordances  What scripts do you use if any?

14. Show video

15.  No standard way of puppeteering avatars  Few research studies:  what behavioral cues are important  how they are used

22. Show video

23.  Participants:  groups of 2-3  11 participants  Situation Setup: an argument  Video Tapped the Session  Logged all button pushing and interaction  Total: 15-18 minutes

27.  "Being able to visually interact with other Skype users through emotions is great, when you do not wish to use you camera or do not have access to one."  "bring the avatar back to the screen having fun making the avatar move around"  "Can’t see the point of the system while there is webcam and chat options.”

28.  “It difficult to concentrate for both moving the avatar around and talking at the same time.”  “As time goes on the content of the conversation becomes more important, the visuals becomes unnecessary.”

29.  Context is very important  Important Signals:  Thumbs Up and down  Turn taking  Applause  Proximity  Gaze

30.  Social Signaling is Important  Identified some important behavioral cues depend on context  It is still unclear:  The right UI and control  The right semantic buttons  Conscious vs. non-conscious

31.  FLAME (1998) – modeling Emotions and Learning in believable characters  ELE (2003) – Expressive Lighting Engine: system that allocates and adjusts lights to accommodate interaction and aesthetic goals  Mirage (2002) – Interactive Narrative Architecture using theatre acting and filmmaking principles  Player modeling (2002) – figuring out style of play and adapting an interactive narrative accordingly

32.  Adaptive Systems  Believable characters  Visual Design and Presentation  User Experience Research  Games User Research Group (IGDA)  Assassin’s Creed II Study (Ubisoft)  Façade studies  Triangulation of eye tracking, physio, game metrics (Electronic Arts)  Co-op patterns (Bardel Entertainment)  Virtual world monitoring through visualization and Metrics (God Mode Games and Relic Entertainment)  Laban + Delsarte for modeling expressive behavoior