1. TAUCHI – Tampere Unit for Computer-Human Interaction Markku Turunen Tampere Unit for Human-Computer Interaction University of Tampere MUMIN PhD course, Tampere, 18.-22.11.2002 Speech Application Architectures

2. TAUCHI – Tampere Unit for Computer-Human Interaction Outline Topics Background Architecture types Example architectures Topics for research Jaspis architecture

3. TAUCHI – Tampere Unit for Computer-Human Interaction Software architectures 1 Definitions “software architecture defines the system in terms of components and interactions between them. Connectors are used to mediate interaction between the components” [Garlan & Shaw, 1994] several views can be used to describe different aspects of software architectures: design view, run-time view, module view, logical view, control view, class view, … human-computer interaction viewpoint: support for interaction methods and techniques

4. TAUCHI – Tampere Unit for Computer-Human Interaction Software architectures 2 Software development tools support and tools for the construction of practical applications core architecture: basic infrastructure (hub/facilitator, communication libraries, blackboard) complete architecture: technology components (ASR, TTS), dialogue manager, database, … toolkit: dialogue editor, ASR grammar builder, corpus collection tool, annotation editor, …

5. TAUCHI – Tampere Unit for Computer-Human Interaction Speech system components speech recognition natural language processing speech synthesis natural language generation database dialogue management telephone interface user

6. TAUCHI – Tampere Unit for Computer-Human Interaction Architecture types 1 Pipelines and dialogue management architectures pipeline (batch-sequence) architectures –data flow –one-way interfaces –fixed processing order dialogue manager architectures –function calls –dialogue manager as controller –relaxed processing order TTS ASR NLU DM NLG TTS ASR NLU DM UM NLG DB

7. TAUCHI – Tampere Unit for Computer-Human Interaction client-server and blackboard architectures client-server architectures –two-way messages –hub as coordinator (star topology) –free processing order blackboard (DB) architectures –data events / db operations –shared information –free processing order IS Architecture types 2 TTS ASR NLU DM UM NLG DB HUB TTS ASR NLU DM UM NLG

8. TAUCHI – Tampere Unit for Computer-Human Interaction agent architectures independent agents –independent agents –facilitator –collaborative processing compact agents –compact agents –shared knowledge –distributed processing Architecture types 3 TTS ASR NLU DM UM NLG DB Facilitator TTS ASR Facilitator IS PE DA DE PA UA DE IA DA PE PA DA IA PA NLG NLU

9. TAUCHI – Tampere Unit for Computer-Human Interaction Example architectures 1 GALAXY-II MIT / MITRE DARPA Communicator reference architecture freely available HUB and servers frames (messages) hub scripts route messages [Seneff et al., 1998]

10. TAUCHI – Tampere Unit for Computer-Human Interaction Example architectures 2 Open Agent Architecture general agent architecture Facilitator as coordinator requesters (tasks) services (solutions) Interagent Communication Language (ICL) freely available used in speech applications [Martin et al., 1999]

11. TAUCHI – Tampere Unit for Computer-Human Interaction Example architectures 3 WITAS dialogue manager agent reacts to events send by other agents dialogue manager acts as blackboard multimodal inputs are coordinated by DM based on OAA [Lemon et al., 2001]

12. TAUCHI – Tampere Unit for Computer-Human Interaction Example architectures 4 MITRE architecture dialogue manager as controller default processing order dialogue manager monitors other components dialogue manager is a kind of blackboard based on OAA [Luperfoy et al., 1998]

13. TAUCHI – Tampere Unit for Computer-Human Interaction Example architectures 5 TRIPS agents, managers and shared databases loosely coupled components no dialogue manager KQML messages facilitator does not contain control logic [Allen et al., 2001]

14. TAUCHI – Tampere Unit for Computer-Human Interaction Current vs. new application areas Traditional speech applicationsFuture speech applications single usermultiple users desktop and telephony office and home environments, mobile settings Single deterministic dialogue open-ended, dynamically constructed concurrent dialogues active user(pro)active computer centralized dialogue managementdistributed interaction management

15. TAUCHI – Tampere Unit for Computer-Human Interaction Current vs. new application areas Traditional speech applicationsFuture speech applications mostly unimodalmostly multimodal alternative / exclusive (sequential) modalities concurrent / synergistic (parallel) modalities speech, text, graphicsspeech, sensors, haptics, gestures… monolingualMultilingual ”natural” interaction methods based on human-human interaction ”innovative” interaction methods based on human-computer interaction

16. TAUCHI – Tampere Unit for Computer-Human Interaction Adaptive systems Need for adaptive applications different users: speech-based communication can differ greatly between individual users and situations –speech is language and culture dependent –preferences and needs between user groups can be large different approaches: people from different backgrounds have different solutions for same problems we need interaction methods and architectures that adapt to the different users and situations and support multiple approaches

17. TAUCHI – Tampere Unit for Computer-Human Interaction Future example U: S: welcome to the bus timetable system? How may I help you? U: I want to go to hospital. S: Which hospital do you mean? There are three hospitals? U: The northern one. S: what is your departure location? U: railway station. S: bus number 17 leaves 10:12 U: bye. time passes, no bus is coming U: S: welcome to the taxi service. How may I help you? … S: You have appointment with your doctor. You need to hurry to catch the bus. It leaves the central station 10:12. U: thanks. time passes, bus strike hits the city S: I’m really sorry, there is no bus coming. The next train leaves seven minutes from now.. U: no, I take a taxi. S: Please wait… It is coming, please wait in front of the opera building. U: thanks.

18. TAUCHI – Tampere Unit for Computer-Human Interaction Topics for research Topics for speech systems adaptivity: how to support adaptive methods? how to make systems to be adaptive? reusability: components, interaction methods, … distributed systems: communication protocols, resource sharing, ubiquitous applications distributed interaction management: centralized dialogue manager is not suitable for many tasks shared knowledge: dialogue, user etc. development and evaluation tools: WOZ, corpora, …

19. TAUCHI – Tampere Unit for Computer-Human Interaction Jaspis architecture speech application development framework implementation of core architecture with extensions designed especially for multilingual and distributed applications overall focus on system level adaptivity current focus on ubiquitous and multimodal applications Java and XML, freely available used in several projects and applications

20. TAUCHI – Tampere Unit for Computer-Human Interaction Jaspis architecture overview NGLNLU DBUM

21. TAUCHI – Tampere Unit for Computer-Human Interaction Agents, evaluators and managers agents handle various interaction situations, such as speech input interpretations, dialogue decisions and speech output presentations evaluators measure how well agents can handle current interaction situation managers are used to coordinate agents and evaluators, especially to try to choose the best possible agents to handle each interaction situation Jaspis components

22. TAUCHI – Tampere Unit for Computer-Human Interaction Jaspis interaction management

23. TAUCHI – Tampere Unit for Computer-Human Interaction information storing method is not fixed (XML, DB) information access protocol is defined (DTD) Information Managers are used to access the Information Storage – these can be implemented in any language and they can use TCP/IP, XML-RPC or method calls Information management in Jaspis

24. TAUCHI – Tampere Unit for Computer-Human Interaction presentation agents convert conceptual messages to speech outputs for every output the most suitable agent is selected by presentation evaluators multiple presentation management modules for different phases Presentation management in Jaspis

25. TAUCHI – Tampere Unit for Computer-Human Interaction different dialogue agents for different dialogue tasks alternative dialogue agents for same dialogue tasks dialogue evaluators select dialogue agents no single controller (the dialogue manager) multiple dialogue management modules Dialogue management in Jaspis

26. TAUCHI – Tampere Unit for Computer-Human Interaction i/o-agents and evaluators handle, combine and coordinate different input streams devices – clients – servers – engines run-time interpretation and multimodal fusion separate module for selection of input modalities Communication (I/O) management in Jaspis

27. TAUCHI – Tampere Unit for Computer-Human Interaction Jaspis extensions Beyond core infrastructure XML-based linguistic information (Annotation Graphs) and log formats (corpus collection, usability tests) visualization components (blackboard, interaction) speech technology interfaces for common telephony cards, synthesizer and recognizers reusable components: error handling, general tasks SMS interface, graphical components Wizard Of Oz tools

28. TAUCHI – Tampere Unit for Computer-Human Interaction Jaspis Future improvements concurrent dialogues and multiple users event-based interaction management

29. TAUCHI – Tampere Unit for Computer-Human Interaction http://www.cs.uta.fi/hci/spi/ spi@cs.uta.fi mturunen@cs.uta.fi Tampere Unit for Computer Human Interaction Department of Computer and Information Sciences