1. Engineering Dialog for Gadgets Thomas K Harris September 12, 2003

2. ???Questions??? Motivation: Is speech a useful modality for electronic devices? Hardware: How would one get speech in “other” devices? Architecture: What should the system look like? Dialog: What should/will these conversations be like?

3. Phone ClientPDA ClientComputer Client Speech Application Backend Current Speech Application Concept

4. ??? Speech Application Frontend Current Electronic Devices

5. Protocol-based Architecture Havi adapter X10 adapter Speech Graffiti Personal Universal Controller

6. James Stereo (mode) tuner (radio band) AM frequency # station WXXX FM frequency # station WXXX seek forward backward auxiliaryCD (status) playstoppause repeat offsingle tracksingle discall discs disc # track # next track last track random onoff x-bass on off volume volume up volume down off Digital camera control Play mode play stop fast fwd rewind record pause step forward backward info Device mode camera VCR Media type Digital video unknown VHS none (mode) Other devices…

7. Speech Graffiti Dialog Artificial subset language Tree-structured functions Universal conversational primitives User-directed Great for recognition Entirely declarative (and automatic)

8. Minimal Keywords hello-jamesoptions where-am-i, where-was-i go-ahead, ok statusgoodbye what-is, what-is-the how-do-imore

9. Session Management hello-james/goodbye User: blah blah blah... System: ignoring user User: hello-james System: stereo, digital camera User: stereo System: stereo here User: goodbye System: goodbye User: blah blah blah... System: ignoring user

10. Query what-is path/status User: what-is-the am frequency System: the am frequency is five hundred thirty User: what-is random System: random is off User: what-is-the stereo System: the stereo is tuner

11. help/exploration/implicit navigation how-do-i.../options/path options User: control alarm clock radio options System: alarm, clock, radio, sleep... User: more System: x10, stereo User: stereo options System: while turning stereo on: off, am, fm, auxiliary, cd... User: control alarm clock radio options System: alarm, clock, radio, sleep... User: more System: x10, stereo User: stereo options System: while turning stereo on: off, am, fm, auxiliary, cd...

12. invocation/specification/implicit exploration/navigation Path User: stereo auxiliary System: while turning the stereo on and switching to auxiliary: auxiliary User: cd System: while switching to cd mode: cd User: play System: while playing a cd: play

13. list navigation More User: radio band am options System: bracketed list [frequency, kabc, k001, k002, k003, k004, k005][fm][off, volume][alarm, clock, sleep][x10, stereo] rendered: frequency, kabc, k001, k002... User: more System: k003, k004, k005... User: more System: fm, off, volume... User: radio band am options System: bracketed list [frequency, kabc, k001, k002, k003, k004, k005][fm][off, volume][alarm, clock, sleep][x10, stereo] rendered: frequency, kabc, k001, k002... User: more System: k003, k004, k005... User: more System: fm, off, volume...

14. orientation where-am-i User: what is the disc System: the disc is three User: where am i System: stereo cd disc

15. Research Questions Is the subset language learnable? Once learned, is it efficient? Are user mistakes infrequent enough? Are system mistake infrequent enough? Can one generalize from one device to another? Is the subset language well retained?