Goteborg University Dialogue Systems Lab Motivation for using GF with GoDiS TALK meeting Edinburgh 7/

Goteborg University Dialogue Systems Lab SR and interpretation in GoDiS Started out as text-based system –Phrase spotting for interpretation; strings to dialogue moves –Templates for generation; dialogue moves to strings Added SR component –Did not have a corpus, so needed to write a speech recognition grammar by hand

Goteborg University Dialogue Systems Lab Problems with GoDiS, 1 Mismatch between SR module and interpretation module –correctly recognised speech was misinterpreted –Leads to strange and unexpected behaviour (unexpected interactions with dialogue management)

Goteborg University Dialogue Systems Lab ASR/interpretation mismatch S> Välkommen till videon! För att få hjälp, säg hjälp. S> Låt oss se. Vad kan jag hjälpa dig med? U> jag vill spela in ett program (0.6) S> Vill du lägga till en inspelning? U> ja (0.98) S> Låt oss se. Menar du från noll ett eller till noll ett? jag vill spela in ett program –Covered by SR grammar –Not covered by phrase spotting lexicon –However spela in and ett are recognized by interpretation module Correct interpretation: –jag vill spela in ett program  request(add_rec_job) Actual interpretation: –spela in  request(add_rec_job –ett  answer(time(1:00)) System assumes ett specifies either when the recording should stop or end – thus the follow-up question

Goteborg University Dialogue Systems Lab Problems with GoDiS, 2 Mismatch between interpretation module and generation module –System said something that led the user to believe they could say X –but the system does not understand X

Goteborg University Dialogue Systems Lab Interpretation/generation mismatch S> Du kan lägga till en inspelning, ta bort en planerad inspelning, och lista planerade inspelningar. För att börja om, säg börja om. För att sluta, säg sluta. S> huvudmenyn. Vad kan jag hjälpa dig med? U> lista inspelningar (0.75) S> Ockey. gå till inspelningar. S> Låt oss se. Vill du lägga till en inspelning, radera dina inspelningar eller få information om dina inspelningar? Help message suggests that the user can say lista inspelningar But help message is not up- to-date, and neither is SR grammar. Instead interpretation module finds word inspelningar The user is asked to choose between roughly the same alternatives that was presented in the help message

Goteborg University Dialogue Systems Lab Bugs can of course be fixed... –would take lots of time –no guarantee that new bugs won’t be introduced –no guarantee that all bugs have shown up These solutions are not reusable Instead, we want a general and principled solution –enable reuse and rapid prototyping of applications, including grammars

Goteborg University Dialogue Systems Lab Solution, part 1: single grammar for SR, interpretation, generation Only one grammar needs to be written for each application However: –the different modules may require different kinds of grammars, in different formats So: –We need to be able to generate all 3 grammars from a single grammar Keeps grammars in synch, thus avoiding mismatches between e.g. ASR and interpretation grammars However, we may not actually want the exact same coverage for SR, interpretation and generation grammars –We may not want to understand everything that can be generated

Goteborg University Dialogue Systems Lab Solution, part 2: use GF! GF is a powerful tool for mutilingual grammar development Can be used to generate grammars in various other formats Subgrammars can be extracted from larger grammars –SR, interpretation and generation subgrammars may have different coverage...and there are additional benefits:

Goteborg University Dialogue Systems Lab Reduced workload for GoDiS application development and adaptation GF offers resource grammars in several languages This promises to further decrease the time needed for –writing new application grammars –adapting existing applications to new languages

Goteborg University Dialogue Systems Lab Improving SR in GoDiS GF can be combined with statistical methods for robustness –Use GF to generate a corpus to use as a baseline for SLM training –Use GF-generated SR grammars in initial system and collect corpus to be used for training SLMs for speech recognition For recognized out-of-grammar string, try to find nearest in-grammar string; confirm this with user (Gorrell 2003)

Goteborg University Dialogue Systems Lab Multimodality for GoDiS So far, GoDiS systems have been text or speech only GF offers a way of dealing with multimodality and multilinguality in the same framework

Goteborg University Dialogue Systems Lab Goal: GoDiS with GF multimodal grammar and graphical I/O agent in TrindiKit 4 Straightforward to connect multiple input and output agents –Active speech input writes to input_speech:string in TIS –Active Graphical I/O Agent writes to input_gui:string –GF parser reads from both, writes to latest_moves:queue(dmove) –DME produces next moves, writes to next_moves:queue(dmove) –This triggers GF generation (linearization) –Output from GF generation triggers Graphical I/O Agent and speech output Alternatively –Both agents write to the same queue input:queue(string)

Goteborg University Dialogue Systems Lab Conclusion Keeping track of separate SR, interpretation and generation grammars in a dialogue system is time- consuming GF offers the possibility of rapid prototyping of grammars for use in dialogue system applications –Generate other grammar formats from GF –Resource grammars for several languages –Multimodal grammars GF can be combines with statistical methods for robust ASR OAA used to connect GF with TrindiKit and GoDiS

Goteborg University Dialogue Systems Lab

Comments mention Gemini, Regulus –similar arguments –except: not multimodal, multilingual, resource grammars –subgrammar extraction???