Cursive: Controlling Expressive Avatar Gesture using Pen Gesture Introduce authors Avatars are controlled by a person, used for interpersonal communication Desktop virtual reality applications Francesca A. Barrientos John F. Canny UC Berkeley Computer science division CVE’02, September 30-October 2, 2002, Bonn, Germany.

Avatar world communication Verbal communication Chat Spoken Nonverbal communication Change image or play animation Controls Select menu item Select image from palette of images Interface creates disruption between verbal and nonverbal The palace These are examples of typical virtual worlds. Primary form of communication is verbal. They have some kinds of nonverbal communication. But interface creates disruption between verbal and nonverbal communication that doesn’t exist in the physical world during face-to-face communication blaxxun 2 October 2002 CVE '02 • Bonn, Germany Barrientos

Language embedded in physical world In physical world, language embedded in matrix of sounds and visuals including movement Communication through multiple channels is seamless and continuous Avatar, as a virtual body, can send nonverbal communication Language is learned with many channels received simultaneously Invention of writing and reading makes verbal seem more important The avatar sends visual signals. Want to have an avatar to help me communicate verbally and nonverbally at the same time Problem is controlling the avatar through interacting with a computer 2 October 2002 CVE '02 • Bonn, Germany Barrientos

CVE '02 • Bonn, Germany Barrientos Problem Controlling a highly articulated avatar using simple computer input devices is difficult Goals for gesture control technique Seamless with verbal communication Create expressive movement Unobtrusive interaction at a desktop or palmtop computer Problem is that it’s hard to control an avatar input devices like the mouse or a keyboard Want interaction technique that heals the disruption 2 October 2002 CVE '02 • Bonn, Germany Barrientos

Avatar gesture from pen gesture Body gesture has symbolic and qualitative aspects Pen gesture (handwriting) carries discrete and continuous data Pen gesture simultaneously selects an avatar gesture and controls expressive movement Our solution is to generate avatar gesture from pen gesture Here’s what it would look like 2 October 2002 CVE '02 • Bonn, Germany Barrientos

Illustration of technique Writing pen gesture triggers animation Body gesture is “sweep” to side Symbol is letter ‘s’ l Quality being varied is size Here is what it looks like; This is the end of the introduction 2 October 2002 CVE '02 • Bonn, Germany Barrientos

CVE '02 • Bonn, Germany Barrientos Overview Background Nonverbal communication Previous approaches Interaction technique Cursive application Conclusions Now that you know what the interaction technique is, here is a guide to the rest of the talk. First discuss the aspects of nonverbal communication which interest us and what previous approaches have been Then interaction technique Then out implementation Then some conclusions Many kinds of nonverbal communication Among most studied are facial displays and body posture and many more One that interests me is communicative gesture movements of hands while talking 2 October 2002 CVE '02 • Bonn, Germany Barrientos

Communicative gesture Closely tied to speech Encodes ideas shared with speech Clarifies meaning when speech is ambiguous Useful when outside noise interferes with speech Multidimensional in meaning and form Encodes through shape, timing, and quality of movement Indicates mood/emotion The aspect of nonverbal communication we wanted to tackle is gesture because it is so closely tied to speech Interested because its particularly challenging, though we don’t want to limit ourselves A number of researchers have developed interaction techniques for nonverbal communication, not just gesture 2 October 2002 CVE '02 • Bonn, Germany Barrientos

CVE '02 • Bonn, Germany Barrientos Previous approaches Specialized widgets Vlnet expression panel Select Modulate Trigger 3 step process ComicChat EmotionWheel Select and modulate Hunt and click diverts attention Avatar agency BodyChat and ComicChat Driven by chat text Lacks direct control All of these methods lack the sense of movement that makes gesture feel expressive to the person speaking People who have studied digital media believe that the continuous interaction makes computer interaction more satisfying 2 October 2002 CVE '02 • Bonn, Germany Barrientos

Continuous interaction Computers fragment our thinking by substituting discrete events for continuous actions. -Malcolm McCullough (Abstracting Craft, p. 53) [Gestures] can...enhance the experience of agency through kinesthetic involvement and the feeling of directness. -Brenda Laurel (Computers as Theatre, p. 158) Substituting discrete events this is what specialized widgets do agency and kinesthetic involvement Avatars as agents lose the kinesthetic involvement For these reasons we are interested in pen gestures 2 October 2002 CVE '02 • Bonn, Germany Barrientos

CVE '02 • Bonn, Germany Barrientos Why pen gestures Natural People doodle while talking and listening Expressive Reflects emotional state Very personal Can be intentionally manipulated Dual nature form: Analog and digital Multi-dimensional Now on to the interaction technique 2 October 2002 CVE '02 • Bonn, Germany Barrientos

Interaction technique Library Generated offline Input User writes letter in GUI Generation Selection and synthesis Animation Gesture performed w- s- r- This is what happens during the interaction. Here is a closer look at how the system it library GUI Gesture generator Avatar animator 2 October 2002 CVE '02 • Bonn, Germany Barrientos

Interaction schematic When the pen gesture is written, both symbolic and continuous data is extracted from the digital ink. The way we produce the expressive gesture is by blending together different samples of the avatar gesture. Cursive interaction technique. Copyright © 2002 by Barrientos. 2 October 2002 CVE '02 • Bonn, Germany Barrientos

Synthesizing expressive gesture An avatar gesture is represented by a set of motion samples Each motion sample has different movement qualities Samples recorded using human performance capture system Samples made up of set of joint angle trajectories Synthesize expressive gesture instance by blending samples So, for example 2 October 2002 CVE '02 • Bonn, Germany Barrientos

Motion blending example sample 2 sample 1 interpolated gesture instance The method we use to interpolate is multilinear interpolation. 2 October 2002 CVE '02 • Bonn, Germany Barrientos

Modulation through multilinear interpolation Gesture sample I Rotation angles over time Gesture type β Set of 2n prototypes - G n style parameters Prototypes represent extremal trajectories Gesture instance Iβ(u,v ) Vector of joint trajectories Multilinear interpolation on type produces instance Speed u Size v Iβ(0,1) Iβ(0,0) Iβ(1,1) Iβ(1,0) Iβ(l,k) Iβ(0,k) Iβ(1,k) k l This method will work when the gesture samples have the same duration. 2 October 2002 CVE '02 • Bonn, Germany Barrientos

CVE '02 • Bonn, Germany Barrientos Interpolating speed Interpolation along curves of different durations Time dilation step Determine duration for interpolated curve Choose sample rate on interpolated curve Compress slower curve - sample at proportionately slower rate Sample faster trajectory at proportionately faster rate Angle θ We have written an application that implements this interaction technique t 2 October 2002 CVE '02 • Bonn, Germany Barrientos

CVE '02 • Bonn, Germany Barrientos Cursive Designed to make it easy to test the interaction technique with existing standard vrml virtual worlds 2 October 2002 CVE '02 • Bonn, Germany Barrientos

Independent of specific avatar world software Cursive communicates directly with all avatar copies Avatar driver runs Cursive to control gesturing avatar Other viewers can see gestures without installing Cursive 2 October 2002 CVE '02 • Bonn, Germany Barrientos

CVE '02 • Bonn, Germany Barrientos Communication viewer host driver host Driver logs into Vworld server Other viewers receive notification Other viewers request and download avatar copy Avatar opens socket connection to Cursive Cursive sends gesture commands via socket Web server request avatar Download avatar open socket sends commands So far we have only used Cursive in our own lab login notification VWorld server 2 October 2002 CVE '02 • Bonn, Germany Barrientos

CVE '02 • Bonn, Germany Barrientos Evaluation Very simple to control one parameter at a time More complicated to control size and speed simultaneously Effective usage requires practice Require further investigation into mapping handwriting features to movement parameters Viable technique for controlling avatar gesture 2 October 2002 CVE '02 • Bonn, Germany Barrientos

CVE '02 • Bonn, Germany Barrientos Summary Wanted: Richly expressive, spontaneous gesture in avatar worlds Solution: An interaction technique employing pen gesture input Cursive: An implementation of this interaction technique 2 October 2002 CVE '02 • Bonn, Germany Barrientos

CVE '02 • Bonn, Germany Barrientos Future work User studies Experiments with public multi-user virtual worlds Investigate avatar gesture vocabulary How many gestures What types of gestures 2 October 2002 CVE '02 • Bonn, Germany Barrientos

CVE '02 • Bonn, Germany Barrientos Conclusion Transmitting bodily nonverbal communication through the internet is an exciting possibility Continuous interaction strengthens the connection between the user and the avatar Pen gestures are a powerful technique for controlling expressive avatar gesture Expressive writing is natural Interaction technique is compact and unobtrusive Maps continuous actions to continuous events 2 October 2002 CVE '02 • Bonn, Germany Barrientos

CVE '02 • Bonn, Germany Barrientos Acknowledgements My gesture model: Erin Dare. 2 October 2002 CVE '02 • Bonn, Germany Barrientos