1 Chinese-Speaking 3D Talking Head Project No: H08040 Sang Siew Hoon Supervisor: Dr Ng Teck Khim

2 System Objectives create a realistic Chinese-speaking 3D talking model create a realistic Chinese-speaking 3D talking model -automatic -accurate and realistic -portable

3 Problems Lack of references on Chinese Lack of references on Chinese Involves various fields Involves various fields -Face modeling -Text-to-speech -Face Animation (Focus)

4 System Overview 5 modules 1. Face Model Preparation 2. Chinese Visemes 3. Text-to-Speech (TTS) 4. Animation 5. Rendering

5

6 Face Model Preparation

7 Face Modeling 3DS Max Face Modeling 3DS Max

8 Face Model Preparation Defining MPEG-4 feature points Defining MPEG-4 feature points

9 Face Model Preparation Assigning lip vertices Assigning lip vertices

10 Face Model Preparation Cloning morph targets Cloning morph targets FAP#3 open_jawFAP#4 lower_t_midlip

11 Chinese Visemes Visemes - the visual equivalent of a phoneme 1. Classification of Chinese Syllables 2. Definition of Chinese Visemes 3. Refining Dynamic Visemes

12 Chinese Visemes Definition of Chinese Visemes Definition of Chinese Visemes

13 Chinese Visemes Definition of Chinese Visemes Definition of Chinese Visemes

14 Chinese Visemes Definition of Chinese Visemes Definition of Chinese Visemes

15 Chinese Visemes Refining Dynamic Visemes Refining Dynamic Visemes 1. + e only  replace er 2. zh, z, y + I  drop i 3. d, l, g, j, zh, z, y, w + u | ü  drop 3. d, l, g, j, zh, z, y, w + u | ü  drop 4. j, y + ending with an  replace the an by en 5. j + complex finals headed by i and followed by more than one  drop i in complex finals

16 Text-to-Speech

17 Text-to-Speech

18 Animation 1. Coarticulation 2. Automatic Generation of Face Animation 3. Enhancing Realism

19 Animation Enhancing realism Enhancing realism - Eye blinks - Eyebrow raising - Gaze - Head Rotation

20 Results Accurate and realistic Accurate and realistic No discrepancies for new face models No discrepancies for new face models Automated results need slight manual intervention Automated results need slight manual interventionhttp://

21 Conclusion Contributions Contributions 1. own system to define Chinese visemes - adds to the current research works of Chinese talking heads (currently limited). Chinese talking heads (currently limited). 2. an automatic Chinese lip-synchronization system - saves an animator much time and effort

22 Conclusion System Limitations System Limitations 1. TTS limitation 2. FAPs conflicts 3. Not integrated – inconvenient for untrained users

23 Conclusion Future Works Future Works 1. Automatic audio signal processing 2. Resolve FAPs values conflicts 3. Automatic feature points assignment system 4. Integration of system