Digital Interactive Entertainment Dr. Yangsheng Wang Professor of Institute of Automation Chinese Academy of Sciences

Overview of Digital Entertainment Industry

Digital Entertainment Industry Year 2008 the market size will be about $100B and major products are TV games 、 PC games and mobile games. Microsoft, Sony and Nintendo provide most digital entertainment products. China has a very big potential market for digital entertainment products.

Internet and Game in China The total amount of Chinese Internet users is up to 87 millions in 2004, about 25% are online game(OG) users. OGs are one of major parts of digital entertainment contents in China. Almost half of the OGs on the domestic market mainly come from Korea. Some Chinese game companiers are beginning to develop their own OGs. PC Stand-alone games are losing their market. Mobile games are developed rapidly, but the income is on the low level at the present stage. Overseas game companies enter Chinese market via Joint-venture.

source ： CGPA&IDC 2004 Internet Users in China

Online Game Market in China source ： CGPA&IDC 2004

Stand-alone Game Market in China source ： CGPA&IDC 2004

New Research & Applications of Natural Interactive Technologies

Traditional games mainly depend on mouse, keyboard, joystick and special input device. Game players desire more natural and intelligent interaction methods, including voice, gesture, expression, gaze, and intelligent sensor, etc. User-oriented multi-model HCI (Human Computer Interaction) technologies has become the hotspot of research and application in many domains. Natural Interactive Technologies (NIT)

Computer Vision for Computer Games and Graphics Camera is used as a powerful interface device for games by sensing user’s body position, head orientation, direction of gaze, pointing commands, and gestures. Developer: Mitsubishi Electric Research Laboratories. Applications: Sega games NIT Research and Application-I Vision device

Vision-based Human-Computer Interface for Games By a camera, the player's body movements are projected right into the game as an unencumbered input and control device. Developer: Sony Computer Entertainment America Applications: Sony PlayStation2 EyeToy games NIT Research and Application-II Camera

Voice Control for Games and Simulations Many of today's games have a lot of keyboard commands that can be hard to remember. Speech recognition technologies are used in this area for game control. Product: Game Commander (Mindmaker) Product: Game Voice (Microsoft SideWinder) NIT Research and Application-III

Interactive Entertainment Research Group in CASIA

Auto-Construction of a Game Object; Player-Oriented Game Multi-model Human & Computer Interaction; Natural Interactive Entertainment Platform. The Outline of Our Research

Photo-stereo Based Model Reconstruction  Images capture  Normal calculation  Surface reconstruction flashlight camera bracket control card

Images capture Images of the surface under different lighting conditions is captured in a very short period, that the register is avoid.

Normal calculation A photo-stereo method is used to retrieve the surface normal based on the reflect property of the surface.

Surface reconstruction The relevant height between the points can be integrated from the derivative field. A pyramid integration method is used to optimize the results

Orthogonal images Based Model Reconstruction  Generic Model and its feature points selection  Two images feature points editing  Generic model deformation  Texture merging and mapping

Generic model and its feature points selection 79 feature points according MPEG4

Images feature points editing and texture merging frontal faceside face texture merging result

Generic model deformation and texture mapping texture generic model model reconstruction result deforming

Individual game characters Individual faces give players more fun and immersion. player A player B

 Face detection  Deformation & Texture mapping  Expression recognition  Expression retargeting & Animation Video-Based Model Reconstruction and Animation Capture user’s frontal face with a camera, then deform an existent face model and map the face image to create an individual face model. Five kind of user’s expressions are recognized real-time to drive the relevant animations.

Auto-Extraction of key points of a face An RealBoost-based statistical learning is adopted to build a model for each facial point. During extraction, the models are used to locate facial points for each face image.

Deformation & Texture mapping Origin imagefilling image generic model merging texture reconstructed individual models deforming

Expression recognition and retargeting Expression recognition Source expression models Individual expression results surpriseanger sad happyfear A novel approach to real time facial expression recognition using an embedded HMM which uses observation vectors that are composed of DCT results.

Getting into a Game - Change face  Player’s frontal face is captured by a camera.  Player’s face feature points are detected to deform character’s face model.  Player’s face image is used change character’s face texture.

Head detection and pose estimation Fist detection and motion analysis Other motion detection and analysis Video-Based Motion Detection and Analysis

Head detection and pose estimation frontleftrightupdownclosefar  locating the position of a head  recognizing the pose of the head The novel scheme based on Markov model fusing appearance, color, and motion information.

Fist detection and motion analysis prototype DEMO training samples

Driving a car using your hands and voice  Player’s hands control the car’s direction.  Player’s voice controls the car’s speed and status.

Motion Detection and Analysis  Project video into game  Motion detection  Parameters calculation  Game objects control

Playing a game with your body  Player’s body movement is projected right into a game  The motion detection and analysis results are used to activate objects in a game.

Online-Game Strategic Research Group

A meeting of the group The group includes major digital entertainment companies, universities, research institutes, and digital entertainment association in China.

Write strategic report of digital entertainment for Chinese government. Establish “China National Center for Digital Entertainment”. The goal of the Strategic Research Group

Thank you!