1. Arne Jönsson 1 Multimodal research in Linköping Arne Jönsson NLPLAB/HCS/IDA

2. Arne Jönsson 2 NLPLAB Upper row: Pernilla, Sonia, Lena, Annika, Micke, Magnus Bottom row: Håkan, Lars A, Robert, Mustapha, Arne, Lise, Anders, Lars D Missing: Nils, Bertil, Pontus, Gen

3. Arne Jönsson 3 Current research (Multimodal) information access -Natural language interaction -Text and document analysis Usability of multimodal interfaces Open-Source and development methods

4. Arne Jönsson 4 Dialogue system Dialogue manager Generator Interpreter Structured Information Dialogue system

5. Arne Jönsson 5 Ex TV programme guide Search for TV programs Mediaterminal Structured database Spoken input Developed from framework and method for iterative development With Nokia Home Communication

6. Arne Jönsson 6 Demo

11. Arne Jönsson 11 Text and document analysis Language analysis Structured Information Unstructured documents Q&A MT

12. Arne Jönsson 12 Ex. AskRSV Information extraction from text document Simple Q&A-system With Ida Infront

13. Arne Jönsson 13 Ex. taxation brochure 24 pages unstructured text

14. Arne Jönsson 14 Domain ontology in ASK-RSV Taxonomy of a limited set of objects, events, properties and relations in the tax domain Only IS-A and synonymy relations OBJECT THING ACTOREVENT PAPERDATETAXATIONTAX EVENTORG.PEOPLE TAX-RETURNASSET DEBT INCOME TRANSACTION COMP. PUBL

18. Arne Jönsson 18 Combined system Structured Information Unstructured documents Language analysis Dialogue system LexiconGrammarOntology

19. Arne Jönsson 19 Shared knowledge sources Ontology, lexicon and grammar -Different demands from IE and DS Design Development -Corpus analysis, user -Background system, database Automatic update

20. Arne Jönsson 20 BirdQuest Bird encyclopaedia -Tagged, templates Corpus with user questions -Analysed Based on dialogue systems framework

24. Arne Jönsson 24 Ex iNews News telegrams -Economy Read and navigate (multimodal) Synthetic face With Telia Research

25. Arne Jönsson 25

26. Arne Jönsson 26 Ötraf Time tables from local bus company Multimodal system -Dialogue -Handling of spatial and temporal expressions

28. Arne Jönsson 28 Studies of multimodal interaction Wizard of Oz-simulation Feedback study -User’s experience -Interviews, questionnaires -Various dimensions: control, cooperation, ease of use, efficiency, anxiety, hability etc -Three conditions -No spoken feedback -Limited spoken feedback -Full spoken feedback -30 subjects

29. Arne Jönsson 29 Results All or nothing More like a human in the full feedback condition -Before the study subjects wanted to have a timetable for the task, after they preferred a human No difference in experienced control and cooperation The speech synthesis better in full feedback mode -Same voice

30. Arne Jönsson 30 Open source Share research results between research groups -Stimulate exchange and co-operation Language technology readily available for industry -Increased use and insights -The companies do their own prototype development Robust code nlpFarm http://sourceforge.net/projects/nlpfarm/

31. Arne Jönsson 31 http://sourceforge.net/projects/nlpfarm/

32. Arne Jönsson 32 Research issues Open Source Software (OSS) -Re-use, design, modularization -APIs -Knowledge representation -Integrate with the communities OSS Usage -Stable public language technology resources -Forms for co-operation and to share resources -Packaging, presentation and methods for development

33. Arne Jönsson 33 A method for development of dialogue systems Two problems when developing dialogue systems: too much time is devoted to discussions on features of the system that are interesting but often rare and hard to realise it is not easy to subdivide the work with design and implementation into manageable pieces when developing a dialogue system.

34. Arne Jönsson 34 Inspiration Extreme programming -coding instead of chart diagrams Code re-use -open source Iterative development Framework and tool development

35. Arne Jönsson 35 Iterative development Design and coding a joint activity -twofold implementation iteration -design document and module code Iterative and evolutionary -facilitates problem understanding -capability steps

36. Arne Jönsson 36 Method overview DM theory DS requirements specification Other modulesDM framework DM design DM module Conceptual design Framework customisation