1. Joys and Pains of building a Dialog System
Let’s Go!!
Joys and Pains of building a Dialog System
Antoine
Dialogs on Dialogs Group
April 18, 2003

2. Process Chart Task Specification Dialog Management Design Grammar
Writing
NLG Design
Testing
Evaluation
Refinement
TTS System
Creation/Adaptation
ASR Models
Building

3. What is interesting?
How do you make interaction as natural as possible?
Design the initial task model
Adding some disambiguation/confirmation agents
Make the system sound smart (e.g. present results in a “natural” way, etc.)

4. What is interesting? (2)
How do people actually dialogue with a machine?
Use the system yourself
Ask your friends to use it
Design and perform experiments

5. What is time consuming? How do you get this f*!@?&g system to work?
Install all the components
Get your database up and running (~done)
Get ASR to work properly (not there yet…)
Get synthesis to work properly (idem)
Debug pure bugs (typos in C, Perl, …)

6. Impressions
Very rewarding: I can get real bus information from MY system!!
Very frustrating: why can’t anyone else do???

7. Ideas, Advice
Get a (very simple) FULLY running system as soon as possible (from ASR to Synthesis)
Regularly fix milestones to have a (more and more complex) running system
Try your system and have other people try it as much as possible
Read the logs…

8. What about Let’s Go!! Main Problems
Recognition is very poor for most users.
Synthesis:
Diphone voice not nice
Limited domain: much better but fails (or very bad) when not in recorded data
Backend: cannot interrupt queries (pb with misrecognitions that lead to super large queries e.g. “I’d like to go to Forbes”)

9. What about Let’s Go!!
Next goal (for me): experiment to compare natives and non-natives
Step 1: have the system work well enough with natives
Step 2: get 10 natives and 20 non-natives use the system with scenarios
Step 3: compare performance (success rate, number of misunderstandings…)
Step 4: identify main sources of performance discrepancy
Step 5: propose remedial measures