Joys and Pains of building a Dialog System Let’s Go!! Joys and Pains of building a Dialog System Antoine Dialogs on Dialogs Group April 18, 2003

Process Chart Task Specification Dialog Management Design Grammar Writing NLG Design Testing Evaluation Refinement TTS System Creation/Adaptation ASR Models Building

What is interesting? How do you make interaction as natural as possible? Design the initial task model Adding some disambiguation/confirmation agents Make the system sound smart (e.g. present results in a “natural” way, etc.)

What is interesting? (2) How do people actually dialogue with a machine? Use the system yourself Ask your friends to use it Design and perform experiments

What is time consuming? How do you get this f*!@?&g system to work? Install all the components Get your database up and running (~done) Get ASR to work properly (not there yet…) Get synthesis to work properly (idem) Debug pure bugs (typos in C, Perl, …)

Impressions Very rewarding: I can get real bus information from MY system!! Very frustrating: why can’t anyone else do???

Ideas, Advice Get a (very simple) FULLY running system as soon as possible (from ASR to Synthesis) Regularly fix milestones to have a (more and more complex) running system Try your system and have other people try it as much as possible Read the logs…

What about Let’s Go!! Main Problems Recognition is very poor for most users. Synthesis: Diphone voice not nice Limited domain: much better but fails (or very bad) when not in recorded data Backend: cannot interrupt queries (pb with misrecognitions that lead to super large queries e.g. “I’d like to go to Forbes”)

What about Let’s Go!! Next goal (for me): experiment to compare natives and non-natives Step 1: have the system work well enough with natives Step 2: get 10 natives and 20 non-natives use the system with scenarios Step 3: compare performance (success rate, number of misunderstandings…) Step 4: identify main sources of performance discrepancy Step 5: propose remedial measures