Cursive: Controlling Expressive Avatar Gesture using Pen Gesture Francesca A. Barrientos John F. Canny UC Berkeley Computer science division CVE’02, September 30-October 2, 2002, Bonn, Germany.

CVE '02 Bonn, Germany Barrientos 2 2 October 2002 Avatar world communication Verbal communication –Chat –Spoken Nonverbal communication –Change image or play animation –Controls Select menu item Select image from palette of images Interface creates disruption between verbal and nonverbal The palace blaxxun

CVE '02 Bonn, Germany Barrientos 3 2 October 2002 Language embedded in physical world In physical world, language embedded in matrix of sounds and visuals including movement Communication through multiple channels is seamless and continuous Avatar, as a virtual body, can send nonverbal communication

CVE '02 Bonn, Germany Barrientos 4 2 October 2002 Problem Controlling a highly articulated avatar using simple computer input devices is difficult Goals for gesture control technique –Seamless with verbal communication –Create expressive movement –Unobtrusive interaction at a desktop or palmtop computer

CVE '02 Bonn, Germany Barrientos 5 2 October 2002 Body gesture has symbolic and qualitative aspects Pen gesture (handwriting) carries discrete and continuous data Pen gesture simultaneously selects an avatar gesture and controls expressive movement Avatar gesture from pen gesture

CVE '02 Bonn, Germany Barrientos 6 2 October 2002 Illustration of technique Writing pen gesture triggers animation Body gesture is “sweep” to side Symbol is letter ‘s’ l Quality being varied is size

CVE '02 Bonn, Germany Barrientos 7 2 October 2002 Overview Background –Nonverbal communication –Previous approaches Interaction technique Cursive application Conclusions

CVE '02 Bonn, Germany Barrientos 8 2 October 2002 Communicative gesture Closely tied to speech Encodes ideas shared with speech –Clarifies meaning when speech is ambiguous –Useful when outside noise interferes with speech Multidimensional in meaning and form –Encodes through shape, timing, and quality of movement Indicates mood/emotion

CVE '02 Bonn, Germany Barrientos 9 2 October 2002 Previous approaches Specialized widgets Vlnet expression panel 1.Select 2.Modulate 3.Trigger 3 step process ComicChat EmotionWheel Select and modulate Hunt and click diverts attention Avatar agency BodyChat and ComicChat Driven by chat text Lacks direct control

CVE '02 Bonn, Germany Barrientos 10 2 October 2002 Continuous interaction Computers fragment our thinking by substituting discrete events for continuous actions. -Malcolm McCullough (Abstracting Craft, p. 53) [Gestures] can...enhance the experience of agency through kinesthetic involvement and the feeling of directness. -Brenda Laurel (Computers as Theatre, p. 158)

CVE '02 Bonn, Germany Barrientos 11 2 October 2002 Why pen gestures Natural –People doodle while talking and listening Expressive –Reflects emotional state –Very personal –Can be intentionally manipulated Dual nature –form: Analog and digital Multi-dimensional

CVE '02 Bonn, Germany Barrientos 12 2 October 2002 Interaction technique Library Generated offline Input User writes letter in GUI Generation Selection and synthesis Animation Gesture performed GUI library Gesture generator Avatar animator w- s- r-

CVE '02 Bonn, Germany Barrientos 13 2 October 2002 Interaction schematic Cursive interaction technique. Copyright © 2002 by Barrientos.

CVE '02 Bonn, Germany Barrientos 14 2 October 2002 Synthesizing expressive gesture An avatar gesture is represented by a set of motion samples –Each motion sample has different movement qualities –Samples recorded using human performance capture system –Samples made up of set of joint angle trajectories Synthesize expressive gesture instance by blending samples

CVE '02 Bonn, Germany Barrientos 15 2 October 2002 Motion blending example sample 1 interpolated gesture instance sample 2

CVE '02 Bonn, Germany Barrientos 16 2 October 2002 Modulation through multilinear interpolation Gesture sample I –Rotation angles over time Gesture type β –Set of 2 n prototypes - G –n style parameters –Prototypes represent extremal trajectories Gesture instance I β (u,v ) –Vector of joint trajectories –Multilinear interpolation on type produces instance Speed u Size v k l GβGβ I β (0,1) I β (0,0) I β (1,1) I β (1,0) Iβ(l,k)Iβ(l,k) I β (0,k)I β (1,k)

CVE '02 Bonn, Germany Barrientos 17 2 October 2002 Interpolating speed Interpolation along curves of different durations Time dilation step –Determine duration for interpolated curve –Choose sample rate on interpolated curve –Compress slower curve - sample at proportionately slower rate –Sample faster trajectory at proportionately faster rate Angle θ t

CVE '02 Bonn, Germany Barrientos 18 2 October 2002 Cursive

CVE '02 Bonn, Germany Barrientos 19 2 October 2002 Independent of specific avatar world software Cursive communicates directly with all avatar copies Avatar driver runs Cursive to control gesturing avatar Other viewers can see gestures without installing Cursive

CVE '02 Bonn, Germany Barrientos 20 2 October 2002 Communication Driver logs into Vworld server Other viewers receive notification Other viewers request and download avatar copy Avatar opens socket connection to Cursive Cursive sends gesture commands via socket Web server VWorld server driver hostviewer host login notification request avatarDownload avatar open socket sends commands

CVE '02 Bonn, Germany Barrientos 21 2 October 2002 Evaluation Very simple to control one parameter at a time More complicated to control size and speed simultaneously –Effective usage requires practice –Require further investigation into mapping handwriting features to movement parameters Viable technique for controlling avatar gesture

CVE '02 Bonn, Germany Barrientos 22 2 October 2002 Summary Wanted: Richly expressive, spontaneous gesture in avatar worlds Solution: An interaction technique employing pen gesture input Cursive: An implementation of this interaction technique

CVE '02 Bonn, Germany Barrientos 23 2 October 2002 Future work User studies Experiments with public multi-user virtual worlds Investigate avatar gesture vocabulary –How many gestures –What types of gestures

CVE '02 Bonn, Germany Barrientos 24 2 October 2002 Conclusion Transmitting bodily nonverbal communication through the internet is an exciting possibility Continuous interaction strengthens the connection between the user and the avatar Pen gestures are a powerful technique for controlling expressive avatar gesture –Expressive writing is natural –Interaction technique is compact and unobtrusive –Maps continuous actions to continuous events

CVE '02 Bonn, Germany Barrientos 25 2 October 2002 Acknowledgements My gesture model: Erin Dare.

CVE '02 Bonn, Germany Barrientos 26 2 October 2002 Related work Commercial worlds –Blend of text virtual communities and computer graphics –Worlds Chat – first 3D world in 1995 Multi-user virtual environment research –Vlnet (Guye-Vuillème et al ’98) –ComicChat (Kurlander+Skelly+Salesin ‘96) –Autonomous avatars (Vilhjálmsson+Cassell’98, Cassell et al ‘94) –Acting in virtual reality (Slater et al ‘00) Synthetic characters –Improv (Perlin+Goldberg ’96) –Alive (Blumberg+Galyean ’95, Maes et al ‘97) –Jack (Badler ’97) Computer mediated conversation visualization –Collaboration-at-a-glance (Donath ’95) –Chat Circles (Viegas+Donath ’99)