1. Cursive: Controlling Expressive Avatar Gesture using Pen Gesture
Introduce authors
Avatars are controlled by a person, used for interpersonal communication
Desktop virtual reality applications
Francesca A. Barrientos John F. Canny
UC Berkeley Computer science division
CVE’02, September 30-October 2, 2002, Bonn, Germany.

2. Avatar world communication
Verbal communication
Chat
Spoken
Nonverbal communication
Change image or play animation
Controls
Select menu item
Select image from palette of images
Interface creates disruption between verbal and nonverbal
The palace
These are examples of typical virtual worlds.
Primary form of communication is verbal.
They have some kinds of nonverbal communication.
But interface creates disruption between verbal and nonverbal communication that doesn’t exist in the physical world during face-to-face communication
blaxxun
2 October 2002
CVE '02 • Bonn, Germany Barrientos

3. Language embedded in physical world
In physical world, language embedded in matrix of sounds and visuals including movement
Communication through multiple channels is seamless and continuous
Avatar, as a virtual body, can send nonverbal communication
Language is learned with many channels received simultaneously
Invention of writing and reading makes verbal seem more important
The avatar sends visual signals.
Want to have an avatar to help me communicate verbally and nonverbally at the same time
Problem is controlling the avatar through interacting with a computer
2 October 2002
CVE '02 • Bonn, Germany Barrientos

4. CVE '02 • Bonn, Germany Barrientos
Problem
Controlling a highly articulated avatar using simple computer input devices is difficult
Goals for gesture control technique
Seamless with verbal communication
Create expressive movement
Unobtrusive interaction at a desktop or palmtop computer
Problem is that it’s hard to control an avatar input devices like the mouse or a keyboard
Want interaction technique that heals the disruption
2 October 2002
CVE '02 • Bonn, Germany Barrientos

5. Avatar gesture from pen gesture
Body gesture has symbolic and qualitative aspects
Pen gesture (handwriting) carries discrete and continuous data
Pen gesture simultaneously selects an avatar gesture and controls expressive movement
Our solution is to generate avatar gesture from pen gesture
Here’s what it would look like
2 October 2002
CVE '02 • Bonn, Germany Barrientos

6. Illustration of technique
Writing pen gesture triggers animation
Body gesture is “sweep” to side
Symbol is letter ‘s’ l
Quality being varied is size
Here is what it looks like;
This is the end of the introduction
2 October 2002
CVE '02 • Bonn, Germany Barrientos

7. CVE '02 • Bonn, Germany Barrientos
Overview
Background
Nonverbal communication
Previous approaches
Interaction technique
Cursive application
Conclusions
Now that you know what the interaction technique is, here is a guide to the rest of the talk.
First discuss the aspects of nonverbal communication which interest us and what previous approaches have been
Then interaction technique
Then out implementation
Then some conclusions
Many kinds of nonverbal communication
Among most studied are facial displays and body posture and many more
One that interests me is communicative gesture
movements of hands while talking
2 October 2002
CVE '02 • Bonn, Germany Barrientos

8. Communicative gesture
Closely tied to speech
Encodes ideas shared with speech
Clarifies meaning when speech is ambiguous
Useful when outside noise interferes with speech
Multidimensional in meaning and form
Encodes through shape, timing, and quality of movement
Indicates mood/emotion
The aspect of nonverbal communication we wanted to tackle is gesture because it is so closely tied to speech
Interested because its particularly challenging, though we don’t want to limit ourselves
A number of researchers have developed interaction techniques for nonverbal communication, not just gesture
2 October 2002
CVE '02 • Bonn, Germany Barrientos

9. CVE '02 • Bonn, Germany Barrientos
Previous approaches
Specialized widgets
Vlnet expression panel
Select
Modulate
Trigger
3 step process
ComicChat EmotionWheel
Select and modulate
Hunt and click diverts attention
Avatar agency
BodyChat and ComicChat
Driven by chat text
Lacks direct control
All of these methods lack the sense of movement that makes gesture feel expressive to the person speaking
People who have studied digital media believe that the continuous interaction makes computer interaction more satisfying
2 October 2002
CVE '02 • Bonn, Germany Barrientos

10. Continuous interaction
Computers fragment our thinking by substituting discrete events for continuous actions.
-Malcolm McCullough
(Abstracting Craft, p. 53)
[Gestures] can...enhance the experience of agency through kinesthetic involvement and the feeling of directness.
-Brenda Laurel
(Computers as Theatre, p. 158)
Substituting discrete events
this is what specialized widgets do
agency and kinesthetic involvement
Avatars as agents lose the kinesthetic involvement
For these reasons we are interested in pen gestures
2 October 2002
CVE '02 • Bonn, Germany Barrientos

11. CVE '02 • Bonn, Germany Barrientos
Why pen gestures
Natural
People doodle while talking and listening
Expressive
Reflects emotional state
Very personal
Can be intentionally manipulated
Dual nature
form: Analog and digital
Multi-dimensional
Now on to the interaction technique
2 October 2002
CVE '02 • Bonn, Germany Barrientos

12. Interaction technique
Library Generated offline
Input User writes letter in GUI
Generation Selection and synthesis
Animation Gesture performed w- s- r-
This is what happens during the interaction. Here is a closer look at how the system it
library
GUI
Gesture generator
Avatar animator
2 October 2002
CVE '02 • Bonn, Germany Barrientos

13. Interaction schematic
When the pen gesture is written, both symbolic and continuous data is extracted from the digital ink.
The way we produce the expressive gesture is by blending together different samples of the avatar gesture.
Cursive interaction technique.
Copyright © 2002 by Barrientos.
2 October 2002
CVE '02 • Bonn, Germany Barrientos

14. Synthesizing expressive gesture
An avatar gesture is represented by a set of motion samples
Each motion sample has different movement qualities
Samples recorded using human performance capture system
Samples made up of set of joint angle trajectories
Synthesize expressive gesture instance by blending samples
So, for example
2 October 2002
CVE '02 • Bonn, Germany Barrientos

15. Motion blending example
sample 2
sample 1
interpolated gesture instance
The method we use to interpolate is multilinear interpolation.
2 October 2002
CVE '02 • Bonn, Germany Barrientos

16. Modulation through multilinear interpolation
Gesture sample I
Rotation angles over time
Gesture type β
Set of 2n prototypes - G
n style parameters
Prototypes represent extremal trajectories
Gesture instance Iβ(u,v )
Vector of joint trajectories
Multilinear interpolation on type produces instance
Speed u
Size v
Iβ(0,1)
Iβ(0,0)
Iβ(1,1)
Iβ(1,0)
Iβ(l,k)
Iβ(0,k)
Iβ(1,k) k l
This method will work when the gesture samples have the same duration.
2 October 2002
CVE '02 • Bonn, Germany Barrientos

17. CVE '02 • Bonn, Germany Barrientos
Interpolating speed
Interpolation along curves of different durations
Time dilation step
Determine duration for interpolated curve
Choose sample rate on interpolated curve
Compress slower curve - sample at proportionately slower rate
Sample faster trajectory at proportionately faster rate
Angle θ
We have written an application that implements this interaction technique t
2 October 2002
CVE '02 • Bonn, Germany Barrientos

18. CVE '02 • Bonn, Germany Barrientos
Cursive
Designed to make it easy to test the interaction technique with existing standard vrml virtual worlds
2 October 2002
CVE '02 • Bonn, Germany Barrientos

19. Independent of specific avatar world software
Cursive communicates directly with all avatar copies
Avatar driver runs Cursive to control gesturing avatar
Other viewers can see gestures without installing Cursive
2 October 2002
CVE '02 • Bonn, Germany Barrientos

20. CVE '02 • Bonn, Germany Barrientos
Communication
viewer host
driver host
Driver logs into Vworld server
Other viewers receive notification
Other viewers request and download avatar copy
Avatar opens socket connection to Cursive
Cursive sends gesture commands via socket
Web server
request avatar
Download avatar
open socket
sends commands
So far we have only used Cursive in our own lab
login
notification
VWorld server
2 October 2002
CVE '02 • Bonn, Germany Barrientos

21. CVE '02 • Bonn, Germany Barrientos
Evaluation
Very simple to control one parameter at a time
More complicated to control size and speed simultaneously
Effective usage requires practice
Require further investigation into mapping handwriting features to movement parameters
Viable technique for controlling avatar gesture
2 October 2002
CVE '02 • Bonn, Germany Barrientos

22. CVE '02 • Bonn, Germany Barrientos
Summary
Wanted: Richly expressive, spontaneous gesture in avatar worlds
Solution: An interaction technique employing pen gesture input
Cursive: An implementation of this interaction technique
2 October 2002
CVE '02 • Bonn, Germany Barrientos

23. CVE '02 • Bonn, Germany Barrientos
Future work
User studies
Experiments with public multi-user virtual worlds
Investigate avatar gesture vocabulary
How many gestures
What types of gestures
2 October 2002
CVE '02 • Bonn, Germany Barrientos

24. CVE '02 • Bonn, Germany Barrientos
Conclusion
Transmitting bodily nonverbal communication through the internet is an exciting possibility
Continuous interaction strengthens the connection between the user and the avatar
Pen gestures are a powerful technique for controlling expressive avatar gesture
Expressive writing is natural
Interaction technique is compact and unobtrusive
Maps continuous actions to continuous events
2 October 2002
CVE '02 • Bonn, Germany Barrientos

25. CVE '02 • Bonn, Germany Barrientos
Acknowledgements
My gesture model: Erin Dare.
2 October 2002
CVE '02 • Bonn, Germany Barrientos