1. A Multi-agent Approach for the Integration of the Graphical and Intelligent Components of a Virtual Environment Rui Prada rui.prada@gaips.inesc-id.pt INESC-ID IST – Technical University of Lisbon, Portugal

2. Introduction  Report on ELVIS Project achievements  Combine efforts from two different research communities  Artificial Intelligence  Computer Graphics  Open Source Framework for easy development of Intelligent Virtual Environments populated with Synthetic Characters (AgentLib)

3. Motivation (1)  Entertainment and Education applications often make use of virtual realities (environments)  Often inhabited by Synthetic Characters  Interact with the user  However the interactions should be believable in order to give a good interaction experience to the user  Two different dimensions:  Visual believability  Behaviour believability

4. Motivation (2)  Visual believability  Believable appearance  Natural body and facial animations (gestures)  Believable interaction with the virtual world  Emotional expression  Behaviour believability  Action planning  Knowledge representation  Synthetic emotions  Social skills  Nature language

5. Motivation (3)  Both dimensions are important  But they have very distinct problems, development approaches and tools  Additionally, the integration of the intelligent and the visual components is often not clean  Dependencies between the intelligent and the visual components  Problems for reusability

6. Motivation (4)  To build a framework that supports the development of virtual environments and synthetic characters  And present a library with ready to use components  We need to have reusable components  Goal: Create a well defined interface between the graphic and the intelligent components

7. The Framework (1)  Inspiration from the Model-Controller-View paradigm  Create an abstract layer between the two different kind of components  Interconnect the intelligent and visual component without creating dependencies between them Abstract Model Intelligent Components Visual Components Behaviour Layer Presentation LayerAbstract Layer

8. The Framework (2)  The presentation layer that contains the components that present the simulated world to the user (e.g. 3D visualization and animation)  The abstract layer that contains an abstract representation of the simulated world and the synthetic characters (e.g. an abstract model)  The behaviour layer that contains the components that implement the intelligent behaviour of the synthetic characters. (e.g. planning algorithms)

9. Abstract Model (1)  Based on the Entity-Relationship knowledge representation scheme  Contains the abstract representation of the world's entities, their properties and relations EntityProperty 1N Relation N N N 1

10.  Entities  block A, block B, and block C  Properties  colour(A, blue)  colour(B, green)  colour(C, yellow)  Relations  OnTopOf(A, B)  OnTopOf(Table, A)  OnTopOf(Table, C) Abstract Model (2) Table A B C

11. Abstract Model (3)  Agent Based Approach  Entity Hierarchy  Agent  Behaviours EntityProperty Relation 1N N N N 1 1 N Agent Behaviour 1

12. Behaviours  Define the abilities of the agents in the world  Change the abstract model  E.g. set new properties or relations, add or remove entities  Execution cycle  Start procedure  Arguments  Stop procedure  Termination test

13. Update Cycle  Every Entity (including the agents) has two update methods  Start Update  Finished Update  Two methods to guarantee that the update process is independent from the order of update  One entity updates its children  Calls the update methods of its child entities  Calls the update methods of its behaviours  There is one Root Entity for each world

14. Events  Changes in the abstract model and the execution of behaviours generate events  Property Changes Events  Relation Changes Events  Behaviour Execution Events (start, stop)  Add/remove Entity Event  Add/remove Behaviour Event  User defined events (raised by the behaviours)  Components register into the Entities that they want to observe

15. Interface with the Intelligent Components ● Agents can be controlled by Intelligent components external to the Abstract Model 1.These components register into the entities that they want to monitor 2.Request the execution of actions in the world through the execution of behaviours (a)Start a behaviour (b)Stop a behaviour 3.Receive Perceptions from the World through the Event mechanism

16. View (1)  Establishes the interface with the presentation layer  Encapsulates the representation engines and devices  2D animated world  3D virtual reality  Augmented reality

17. View (2)  View  Represents the device that displays the world to the user (e.g. the drawing window)  Characterized by several Attributes (e.g. window size)  Composed by several Layers  Layer  x,y coordinates  Width, height  Depth level  Attributes ViewAttributes Layers 1N N N 1 1

18. View (3)  Layers are placeholders to attach Layer Elements  These Elements define the contents of the layers and the representation object that they display  Bitmap  3D Camera  Text Area ViewAttributes Layers 1N N N 1 1 Layer Element 11 1 N

19. View (4) View Layer (Bitmap) Layer (Text Field) Layer (Button)

20. View (5)  Scenes  Are characterized by several Attributes  Contain several Scene Elements  Attached to a Layer through Layer Elements (e.g. a Camera)  Scene Elements  Attributes  Feature Points  Where other Scene Elements may be attached  Plug-ins  Manipulate the Scene Elements  Examples of Scene Elements: 3D Model, Light Source

21. View (6) ViewAttributes Layers 1N N N 1 1 Layer Element 11 1 N Scene Element N 1 Plug-in N 1 Feature Points N 1 1 N N 1 1 1

22. View (7)  Plug-ins  Started  Stopped  Examples: Animations, Path Interpolators, Sound Sources  Observers  Registered in the Scene Elements  Receive events  Collision events  User Input  Keyboards events

23. The Whole Picture  The Intelligent Components interact with the entities defined on the Abstract Model  Receive perception through events  Act through agents' behaviours  The behaviours may change the Abstract Model and the View Elements  For example: one may define a behaviour that plays an animation on a 3D character and then, when it finishes, changes the a property in the Abstract Model  The View interface defines and abstract language that is implemented and realized by the Components in the presentation layer (e.g. The 3D Graphics Engine)

24. Test Case Example: FearNot! episode Abstract Model User Luke John Books View / Interaction Manager Stage Manager Contr ol Luke’s Mind Contr ol John’s Mind 3D Luke 3D John 3D Books 3D Classroom User Interface

25. Test Case (2)  Video

26. Conclusions (1)  Multi-agent technology may is used to define an abstract layer that promotes the interdependency of the intelligent and visual components of a virtual environment  This independence allows the components to be reused and therefore to build a toolkit for rapid development of virtual environments, namely with the presence of synthetic characters

27. Conclusions (2)  Using an abstract common model is one way to share research efforts  The Abstract Model was used to establish the interface of the Intelligent Components (e.g. planning algorithms) that implement the behaviour of some synthetic characters in some of our projects: e.g. the VICTEC project  The View abstraction is currently being implemented and will be tested with a simple example

28. Questions?  rui.prada@gaips.inesc-id.pt rui.prada@gaips.inesc-id.pt  The AgentLib is on SourceForge  sourceforge.net/projects/agent-sg/  Elvis Workshop in Lisbon  6 th to 8 th of April