10 september 2002 A.Broersen Developing a Virtual Piano Playing Environment By combining distributed functionality among independent Agents

10 september 2002 Overview Introduction to the project Backgrounds of the concepts Design of a multi-agent platform Playing with the system Conclusions and remarks Work in progress towards the aid for a real piano teacher

10 september 2002 Introduction: Parlevink ( A group on Language, Knowledge and Interaction doing research on: Human-Computer Interaction Dialogue systems Autonomous agents Virtual worlds Overview Introduction Design Playing Conclusions Work in progress Demonstration

10 september 2002 Introduction: Field Exploring possibilities using VR and Human-Computer Interaction in distributed learning environments. Case: Piano Playing interactions using a real keyboard, VR, dialogues and dataglove playing with more people at the same time creating musical tools and a teacher in a dynamic learning environment Overview Introduction Design Playing Conclusions Work in progress Demonstration

10 september 2002 Introduction: ADRI (Artificial Didactic Recital Instructor) Goal Building a framework for a piano teacher in a virtual piano playing environment With a real keyboard connected to a piano in a 3D-world with tools for music- and piano-education in a distributed architecture Overview Introduction Design Playing Conclusions Work in progress Demonstration

10 september 2002 Introduction: Requirements ADRI should contain: different independent processes server functionality Java3D Midi for sound using Windows and Unix general knowledge/lesson structure Overview Introduction Design Playing Conclusions Work in progress Demonstration

10 september 2002 Design: Agents For flexibility different Agents are used What are Agents in ADRI? Different functions different skilled agents When can you call the processes Agents? When are agents Intelligent? Overview Introduction Design Playing Conclusions Work in progress Demonstration

10 september 2002 Design: Multi-Agent Platform Connecting the agents with each other in a platform: Simple communication (messages) between agents On each machine one agent to control the ADRI-agents One agent is manually controlled to handle the available agents and their different connections Overview Introduction Design Playing Conclusions Work in progress Demonstration

10 september 2002 Design: Communication Exchanging XML-messages: Communication messages using a simple protocol Midi messages as quickly as possible ADRI messages exchange of knowledge (graphs) using multicast functionality Is someone listening? Overview Introduction Design Playing Conclusions Work in progress Demonstration

10 september 2002 Playing: The Agents Midiator playing and capturing Midi sounds Visualizer user dialogue in a 3D World with a visualization of a piano and notes Supervisor (remotely) control the system Communicator handle agent functionality on one Java Virtual Machine Overview Introduction Design Playing Conclusions Work in progress Demonstration

10 september 2002 Playing: More Agents Expert provide songs and chord recognition Analyser analyses the notes in a song Observer tracks the users actions and progress Instructor gives assignments to a learner Overview Introduction Design Playing Conclusions Work in progress Demonstration

10 september 2002 Playing: User Interface In the Supervisor: Dynamic visualization and control of the complete agent network in a graph An agent dialogue frame with the specific options in the Agent Sending messages to agents Different combinations of agents give different environments Overview Introduction Design Playing Conclusions Work in progress Demonstration

10 september 2002 Playing: The Visualizer

10 september 2002 Conclusions: Agents In the resulting prototype: the agents are kept simple the graphical Supervisor provides a good control and overview of the system the message-parsing is fast enough for real-time exchange of Midi there is no good structure (yet) for knowledge and lesson structure Overview Introduction Design Playing Conclusions Work in progress Demonstration

10 september 2002 Conclusions: Communication Using no standard package: The communication layer is still under development according to different needs in the application Quick exchange of sound and graphical events using Midi Midi capture is a problem using only Java Overview Introduction Design Playing Conclusions Work in progress Demonstration

10 september 2002 Work in progress: Building... Creating more demo Agents: Navigator Translator Corrector Using advise and remarks of human experts and users Developing plans for lessons: Using graphs for state-transition diagrams Overview Introduction Design Playing Conclusions Work in progress Demonstration

10 september 2002 Work in progress: Where? For (more) information or questions: Parlevink, Dept. of Computer Science University of Twente, the Netherlands Overview Introduction Design Playing Conclusions Work in progress Demonstration

10 september 2002 Demonstration: Playing

10 september 2002 Demonstration: Dataglove

10 september 2002 Playing: The Supervisor

10 september 2002 Demonstration: Screen-shots Host address ADRI Agent Other Agent Route to Supervisor Remote Host

10 september 2002 Demonstration: Screen-shots Communication Group Terminator Multicast Group Multicast Member

10 september 2002 Playing: The Midiator