Intelligent MultiMedia Storytelling System (IMSS) - Automatic Generation of Animation From Natural Language Input By Eunice Ma Supervisor: Prof. Paul Mc Kevitt School of Computing and Intelligent Systems Faculty of Informatics University of Ulster, Magee

Project Goals To interpret natural language story and movie (drama) script input and to extract concepts from the input To generate 3D animation and virtual world automatically, with speech and non-speech audio To integrate the above components to form an intelligent multimedia storytelling system for presenting multimodal stories

Previous Research Schank’s Conceptual Dependency (CD) Theory Primitives SAM Automatic Text-to-Graphics Systems WIP/COMET/APT WordsEye ‘Micons’ and CD-based language animation O Nuallain and Smith’s Spoken Image (SI) system

Previous Research (continued) Multimedia Storytelling Larsen’s Interactive Storytelling AesopWorld Multimedia Interactive Systems AIMI AlFresco KidsRoom Animated Intelligent Agents Cassell’s REA and SAM

Software Analysis Natural language processing tools 3D Animation Tools Gate CPK NLP Suite WordNet 3D Animation Tools Poser 4.0 3D Studio Max VRML 2.0/97 Microsoft DirectX The Actors – Using Conversational Agents COLLAGEN Microsoft Agent BEAT

Context Diagram IMSS Story in natural language Speech (dialogue) Movie/drama script IMSS animation Tailored menu for script input non-speech audio I/O Media of IMSS

General Architecture Natural language story Script writer Script parser Prefabricated objects Basic vocabulary Natural Language Parser Text To Speech Sound effects Concept database mapping 3D authoring tools Animation generation Primitives library Code combination 3D world with audio

Data Flow Diagram Primitives library Natural language parser labelled script Animation director VRML without sound nodes Scene&Actor descriptions Media coordination TTS Synthesized animation Script parser dialogues script Non-speech audio Sound effect driver script Script writer story Music library

DFD of Animation Director Primitives library Actor managers marked script Script distributor World builder VRML without sound nodes Scene generator VRML codes Props provider coordinator

Project Schedule

Conclusions The objectives of IMSS meet the challenging problems in language animation. Mapping language primitives with visual primitives Sophisticated spatial relations between spatial cognition and prepositions in English Visualization requires a gigantic ‘common senses’ knowledge base Representing stories by temporal multimedia requires high coordination Prospective practical applications Children education Multimedia presentation Movie/drama production Script writing Computer games, Virtual Reality