Arne Jönsson 1 Multimodal research in Linköping Arne Jönsson NLPLAB/HCS/IDA

Arne Jönsson 2 NLPLAB Upper row: Pernilla, Sonia, Lena, Annika, Micke, Magnus Bottom row: Håkan, Lars A, Robert, Mustapha, Arne, Lise, Anders, Lars D Missing: Nils, Bertil, Pontus, Gen

Arne Jönsson 3 Current research (Multimodal) information access -Natural language interaction -Text and document analysis Usability of multimodal interfaces Open-Source and development methods

Arne Jönsson 4 Dialogue system Dialogue manager Generator Interpreter Structured Information Dialogue system

Arne Jönsson 5 Ex TV programme guide Search for TV programs Mediaterminal Structured database Spoken input Developed from framework and method for iterative development With Nokia Home Communication

Arne Jönsson 6 Demo

Arne Jönsson 11 Text and document analysis Language analysis Structured Information Unstructured documents Q&A MT

Arne Jönsson 12 Ex. AskRSV Information extraction from text document Simple Q&A-system With Ida Infront

Arne Jönsson 13 Ex. taxation brochure 24 pages unstructured text

Arne Jönsson 14 Domain ontology in ASK-RSV Taxonomy of a limited set of objects, events, properties and relations in the tax domain Only IS-A and synonymy relations OBJECT THING ACTOREVENT PAPERDATETAXATIONTAX EVENTORG.PEOPLE TAX-RETURNASSET DEBT INCOME TRANSACTION COMP. PUBL

Arne Jönsson 18 Combined system Structured Information Unstructured documents Language analysis Dialogue system LexiconGrammarOntology

Arne Jönsson 19 Shared knowledge sources Ontology, lexicon and grammar -Different demands from IE and DS Design Development -Corpus analysis, user -Background system, database Automatic update

Arne Jönsson 20 BirdQuest Bird encyclopaedia -Tagged, templates Corpus with user questions -Analysed Based on dialogue systems framework

Arne Jönsson 24 Ex iNews News telegrams -Economy Read and navigate (multimodal) Synthetic face With Telia Research

Arne Jönsson 25

Arne Jönsson 26 Ötraf Time tables from local bus company Multimodal system -Dialogue -Handling of spatial and temporal expressions

Arne Jönsson 28 Studies of multimodal interaction Wizard of Oz-simulation Feedback study -User’s experience -Interviews, questionnaires -Various dimensions: control, cooperation, ease of use, efficiency, anxiety, hability etc -Three conditions -No spoken feedback -Limited spoken feedback -Full spoken feedback -30 subjects

Arne Jönsson 29 Results All or nothing More like a human in the full feedback condition -Before the study subjects wanted to have a timetable for the task, after they preferred a human No difference in experienced control and cooperation The speech synthesis better in full feedback mode -Same voice

Arne Jönsson 30 Open source Share research results between research groups -Stimulate exchange and co-operation Language technology readily available for industry -Increased use and insights -The companies do their own prototype development Robust code nlpFarm

Arne Jönsson 31

Arne Jönsson 32 Research issues Open Source Software (OSS) -Re-use, design, modularization -APIs -Knowledge representation -Integrate with the communities OSS Usage -Stable public language technology resources -Forms for co-operation and to share resources -Packaging, presentation and methods for development

Arne Jönsson 33 A method for development of dialogue systems Two problems when developing dialogue systems: too much time is devoted to discussions on features of the system that are interesting but often rare and hard to realise it is not easy to subdivide the work with design and implementation into manageable pieces when developing a dialogue system.

Arne Jönsson 34 Inspiration Extreme programming -coding instead of chart diagrams Code re-use -open source Iterative development Framework and tool development

Arne Jönsson 35 Iterative development Design and coding a joint activity -twofold implementation iteration -design document and module code Iterative and evolutionary -facilitates problem understanding -capability steps

Arne Jönsson 36 Method overview DM theory DS requirements specification Other modulesDM framework DM design DM module Conceptual design Framework customisation