Animated Speech Therapist for Individuals with Parkinson Disease Supported by the Coleman Institute for Cognitive Disabilities J. Yan, L. Ramig and R. Cole Center for Spoken Language Research University of Colorado at Boulder, Campus Box 594 Boulder, Colorado ,USA,

CURRENT SITUATION The prevalence of disordered communication is particularly high in the one and one half million individuals diagnosed with ldiopathic Parkinson disease (IPD). There is a growing need for an accessible, inexpensive, effective treatment for the disordered communication in these individuals. At least 89% of these individuals have disordered speech and voice(Logemann et al., 1978), but only 3-4% (Hartelius et al., 1994) receive speech treatment. Our goal is to provide accessible, effective treatment to all who need it.

RESEARCH OBJECTIVE Develop a fully automated agent who interacts with a patient much like a human therapist. Create an animated therapist that will have the ability to increase the loudness and improve the quality and intelligibility of the patient’s speech. Develop a computer-based LSVT program, and demonstrate the feasibility of making LSVT treatment widely accessible through computer-based LSVT. Create tools that will run on off-the-shelf computer platforms, and thus be widely available to individuals with IPD for home use or through access to computers in clinics or public institutions. Create a computer-based LSVT program that will use engaging intelligent animated agents that are lifelike 3d characters. They will be designed to interact with individuals with IPD much like effective LSVT clinicians.

SYSTEM ARCHITECTURE

Patient’s behavior analysis component Audio: 1. Measure the volume of phonation 2. Measure the duration of phonation. Video: 1. Capture the patients' lip motion and show it on the screen. This will allow the patient to notice their mouth shape and how wide it opens during therapy. 2. Capture the patients head orientation through head tracking so that the agent will face the patient.

Synthesis component 1. Display the volume and pitch of the phonation in real time as the patient practices therapy. 2. Plot the parameter as a function of time to track how volume and pitch improve over time. 3. Build up a photo-realistic 3d virtual therapist model 4. Render this character model in 3d virtual environment. 5. Create the basic animation library for the treatment session, typical facial expressions, eye contact, head movements, hand gestures sequences. 6. Create the contextually appropriate gestures and speeches for the therapist and patient exchanges during the therapy according to patients’ phonation and pitch quality. 7. Record the real human voice database of the therapy.

Lori’s avatar

3D ANIMATED THRAPIST TECHNIQUES HEAD ANIMATION Visible speech Visible speech is produced by morphing between viseme targets. Facial expression To generate a large number of facial expressions, we enable independent control of separate facial components. Facial expression is controlled by 38 parameters using sliders in a dialog box. Head gesture Three types of head movements have been designed: head turning, head nodding and circular head movement Eye gesture Eye blink control and eye ball movement control, including circular eye ball movement. Smoothing facial expressions Three types of smoothing algorithms were designed to meet this requirement (1) An “easy in or easy out” algorithm, used to control animation speeds at different times so that the animation is more realistic; (2) Kochanek-Bartels cubic splines, in which three parameters such as tension, continuity and bias are used to produce smooth motion; (3) B-Splines.

BODY ANIMATION A parameter driven skeleton/bone model is used to generate lifelike gestures. The skeleton/bones are considered as rigid objects. Each bone is driven by the specific joints' rotation parameters referenced by the three rotating axes. The movement of the skeleton/bone is controlled by the pre-defined rotation parameters set for each joint.

Multi-level gesture description module Multi-level gesture description module has a three-level structure. The first level, the hand shape transcriber, is used to build the hand shape data. The second level, the sign transcriber, relies on the hand shape database and allows users to specify the location and motion of the two (left and right) arms. The third level, the animation transcriber, generates realistic animation sequences according to the target frames generated from the sign transcriber.

Hand shape transcriber Low-level parameter controller The skeleton comprises 22 degrees of freedom (DOF) in 15 joints of each hand, it provides direct and precise manipulation of each finger joint. High-level trajectory controller A set of commands are defined to describe one specific hand gestures using six trajectories: "spread", "bend", "hook", "separate" "yaw", "pitch“. Hand shape library A total of 20 basic hand shapes were selected for the primary hand shape library.

Body posture transcriber Built on top of the hand shape transcriber. Allows users to specify the body posture in terms of hand shape, location and orientation for both hands and arms. Body posture library. Hand gesture examples

Animation transcriber Enables users to define the animation speed and route as a specific sequence of key frames. One key frame is described by one particular body posture status. A cubic splines-based interpolation algorithm generates the animation sequence according to the hierarchical structure of the body. Animation library includes animation sequences such as bowing, "thumbs up," clapping and others.

CURRENT RESULTS REVIEW Designed a photo-realistic 3d character model of the real therapist-Professor Ramig. Videotaped 8 tapes of Dr. Ramig conducting LSVT therapy with four different patients. Collected the fruitful primary video materials of the treatment session. Received invaluable feedback from the patients about the benefits and barriers of using the animated characters as LSVT therapists. Implemented a working prototype of the seated LSVT therapist agent. Build up the basic database of the speech, facial expressions and upper body gestures of the animated LSVT therapist. Seated therapist