1. Goteborg University Dialogue Systems Lab Dialogue and grammar Göteborg University Dialogue Systems Lab Language Technology in Sweden 2004-05-12 Karin Cavallin, Robin Cooper David Hjelm, Staffan Larsson

2. Goteborg University Dialogue Systems Lab Speech recognition and NL interpretation & generation Options for SR: –Statistical language model –Grammar model Options for NL interpretation: –Word/phrase spotting –Grammar & parsing Options for NL generation –Template-based –Grammar-based Can be combined in various ways

3. Goteborg University Dialogue Systems Lab Are grammars useful for dialogue systems? Most working systems don’t use a grammar! Instead: –keyword or phrase-spotting techniques for interpretation –template-based generation Also, most working systems are built more or less from scratch –do not fully separate application-specific knowledge from general and reusable knowledge

4. Goteborg University Dialogue Systems Lab GoDiS A dialogue system built using TrindiKit and based on the information state approach Implements a theory of Issue-based Dialogue Management Some goals with GoDiS: –Provide a domain-independent theory covering several dialogue genres –Modularity and reusability of system components –Enable rapid prototyping of new applications Focus on dialogue management These goals have been achieved on the level of dialogue management... –... but not for NL processing

5. Goteborg University Dialogue Systems Lab SR and interpretation in GoDiS Started out as text-based system –Phrase spotting for interpretation; strings to dialogue moves –Templates for generation; dialogue moves to strings Added SR component –Did not have a corpus, so needed to write a speech recognition grammar by hand

6. Goteborg University Dialogue Systems Lab tvGoDiS GoDiS application for programming a VCR over the telephone –schedule recordings –remove planned recordings –TV card used for recording television programs as video files Small corpus collected of users calling the system –approx. 100 sessions (calls) and 1100 utterances –misrecognised utterances not yet transcribed –GUI enables immediate access to corpus

7. Goteborg University Dialogue Systems Lab Example from tvGoDIS corpus TrindiGui displaying a tvGoDiS sessiontvGoDiS session Audio files linked with utterances Access to information state at different stages of the dialogue

8. Goteborg University Dialogue Systems Lab Problems with GoDiS Mismatch between SR module and interpretation module –correctly recognised speech was misinterpreted –Leads to strange and unexpected behaviour (unexpected interactions with dialogue management)

9. Goteborg University Dialogue Systems Lab ASR/interpretation mismatch S> Välkommen till videon! För att få hjälp, säg hjälp.Välkommen till videon! För att få hjälp, säg hjälp. S> Låt oss se. Vad kan jag hjälpa dig med?Låt oss se. Vad kan jag hjälpa dig med? U> jag vill spela in ett program (0.6)jag vill spela in ett program S> Vill du lägga till en inspelning?Vill du lägga till en inspelning? U> ja (0.98)ja S> Låt oss se. Menar du från noll ett eller till noll ett?Låt oss se. Menar du från noll ett eller till noll ett? jag vill spela in ett program –Covered by SR grammar –Not covered by phrase spotting lexicon –However spela in and ett are recognized by interpretation module Correct interpretation: –jag vill spela in ett program  request(add_rec_job) Actual interpretation: –spela in  request(add_rec_job –ett  answer(time(1:00)) System assumes ett specifies either when the recording should stop or end – thus the follow-up question

10. Goteborg University Dialogue Systems Lab Problems with GoDiS Mismatch between interpretation module and generation module –System said something that led the user to believe they could say X –but the system does not understand X

11. Goteborg University Dialogue Systems Lab Interpretation/generation mismatch S> Du kan lägga till en inspelning, ta bort en planerad inspelning, och lista planerade inspelningar. För att börja om, säg börja om. För att sluta, säg sluta.Du kan lägga till en inspelning, ta bort en planerad inspelning, och lista planerade inspelningar. För att börja om, säg börja om. För att sluta, säg sluta. S> huvudmenyn. Vad kan jag hjälpa dig med?huvudmenyn. Vad kan jag hjälpa dig med? U> lista inspelningar (0.75)lista inspelningar S> Ockey. gå till inspelningar.Ockey. gå till inspelningar. S> Låt oss se. Vill du lägga till en inspelning, radera dina inspelningar eller få information om dina inspelningar?Låt oss se. Vill du lägga till en inspelning, radera dina inspelningar eller få information om dina inspelningar? Help message suggests that the user can say lista inspelningar But help message is not up- to-date, and neither is SR grammar. Instead interpretation module finds word inspelningar The user is asked to choose between roughly the same alternatives that was presented in the help message

12. Goteborg University Dialogue Systems Lab Bugs can of course be fixed... –would take lots of time –no guarantee that new bugs won’t be introduced –no guarantee that all bugs have shown up These solutions are not reusable Instead, we want a general and principled solution –enable reuse and rapid prototyping of applications, including grammars

13. Goteborg University Dialogue Systems Lab Solution, part 1: single grammar for SR, interpretation, generation No mismatches between the grammars Only one grammar needs to be written for each application However: –the different modules may require different kinds of grammars, in different formats So: –We need to be able to generate all 3 grammars from a single grammar However, we may not want the exact same coverage for SR, interpretation and generation grammars –May not want to understand everything that can be generated

14. Goteborg University Dialogue Systems Lab Solution, part 2: use GF! GF is a powerful tool for mutilingual grammar development Can be used to generate grammars in various other formats Subgrammars can be extracted from larger grammars –SR, interpretation and generation subgrammars may have different coverage In addition, GF offers resource grammars in several languages –This promises to further decrease the time needed for writing a new application grammar

15. Goteborg University Dialogue Systems Lab GF GoDiS Grammar Application grammars –User grammar –System grammar –Shared grammar, used by user grammar and system grammar Utility grammars –Weekday, Time, Channel Application grammars may use –Utility grammars –Resource grammars

16. Goteborg University Dialogue Systems Lab GF VCR grammar One abstract syntax –GoDiS dialogue moves as GF abstract trees Three concrete syntaxes –Swedish –English –GoDiS dialogue moves in prolog notation Statistics: –Abstract syntax : ~420 lines of code –Concrete syntax : Prolog: ~440 lines of code Swedish: ~440 lines of code

17. Goteborg University Dialogue Systems Lab Parsing Swedish to abstract syntax > p -cat=DMove -lang=Swe "spela in från kvart över tolv" request (vcr_add_rec_job_args1 (startTimeToStoreReq (startTimeToStore (time hour12 minute15))))

18. Goteborg University Dialogue Systems Lab Linearising abstract syntax to Prolog notation > l -lang=Prolog request (vcr_add_rec_job_args1 (startTimeToStoreReq (startTimeToStore (time hour12 minute15)))) request(add_rec_job, answer(start_time_to_store(12 : 15))

19. Goteborg University Dialogue Systems Lab Translating Prolog to Swedish > t -cat=DMove Prolog Swe "request(add_rec_job, answer(start_time_to_store(12 : 15))" spela in från tolv femton spela in från tolv och femton spela in från femton över tolv spela in från femton minuter över tolv spela in från kvart över tolv

20. Goteborg University Dialogue Systems Lab Time Grammar A multilingual grammar module Swedish, English, Prolog, started on French and Catalan.

21. Goteborg University Dialogue Systems Lab Constructing concrete syntaxes for time expressions isn’t always trivial Abstract: time hour12 Minute30 Prolog: 12 : 30 English: half past twelve / half twelve Swedish: halv ett klockan två klockan två och femton ?klockan fjorton two o’clock *two fifteen o’clock *fourteen o’clock Corpus studies needed

22. Goteborg University Dialogue Systems Lab Another problem am or pm? på morgonen på eftermiddagen på kvällen på natten

23. Goteborg University Dialogue Systems Lab Combining GF and statistical methods for robustness Use GF to generate a corpus to use as a baseline for SLM training Use GF-generated SR grammars in initial system and collect corpus to be used for training SLMs for speech recognition For recognized out-of-grammar string, try to find nearest in-grammar string; confirm this with user (Gorrell 2003)

24. Goteborg University Dialogue Systems Lab Conclusion Keeping track of separate SR, interpretation and generation grammars in a dialogue system is time-consuming GF offers the possibility of rapid prototyping of grammars for use in dialogue system applications –Generate other grammar formats from GF –Resource grammars for several languages GF can be combined with statistical methods for robustness