Nigel G. Ward, Anais G. Rivera, Karen Ward, David G. Novick Some Usability Issues and Research Priorities in Spoken Dialog Applications Nigel G. Ward, Anais G. Rivera, Karen Ward, David G. Novick Interactive Systems Group, The University of Texas at El Paso Questions Why are many dialog systems poor in usability? are developers failing to use best practice? are researchers missing the important issues? Analysis Phase 1: Usability Event Analysis Three labelers identified 115 times when the user experienced poor system usability 62 times when the subject experienced deft operator actions and categorized them by severity and likely causes Recognition and Understanding +++ / +++ / +++* direct effects on task completion, satisfaction effects on user behavior (slow, considered) effects on designer behavior (more explicit prompts, leading to time cost) Time-Outs +++ / + / ++* delayed reaction when user wants a reprompt Endpointing and Responsiveness +++ / + / ++* delayed reaction after the user finishes speaking Generation and Synthesis +++ / + / +* conservative, invariant speaking rate, causing time-cost pragmatic force unclear Feedback + / / +* unclear indications of dialog activity/ status weak, ineffective dialog control Adaptation of Vocab., Speaking Rate… ++ / /* Prosodically-Marked Speech Acts + / / +* Detecting User Self-Talk + / / +* Other + / + / + * Research Issues Identified Analysis Phase 2: Time Analysis We compared the cummulative times for various dialog activities across the two conditions. Methods Each subject performed the same task twice, once with an automated system, and once with a human operator. We observed and categorized the differences. Advantages: reveals missed opportunities, not just clear errors usability-centered, not performance-centered empirical, not theory-based exploratory, not prematurely quantitative (c.f. Walker et al. 200, Dybkjaer & Bernsen 2000, Moller 2005 …) Dialog Activity system- human human- difference normal system-side utterances 32% 37% 29% normal user utterances 13% 41% -6% error-recovery related utterances 17% 4% 27% system-side silences 6% 20% user silences 14% 11% 16% simultaneous talk 1% 0% experimenter interventions 8% Totals 100% User Operator Spoken Dialog System compare Analysis Phase 3: Relating Usability Issues, System Capabilites, and Research Issues Clear, expressive, flexible synthesis Natural language generation Accurate, recognition complete Normally users behaving sensitivity Prosodic Domain and Corpus The domain was credit card billing support. Subjects were given three structured tasks in each condition. The automated system was based on Nuance Voice Builder. The operator matched the system’s vocabulary and formality level. 20 subjects. 53 minutes total. Concise prompts Adaptive speaking rate Timeouts * impact on time/task-completion/stress Endpointing feedback Informative Responsiveness Easy recovery error Summary have identified outstanding research issues: endpointing, responsiveness, appropriate feedback, etc. have shown how experiences with applications can determine research priorities Vocabulary adaptation Initial Observations With the system, versus with human: Task-completion was lower (58% vs 77%). Time to completion per task was higher (avg. 130 vs 40 sec.) Low stress and cognitive load Quick accomplishment High task completion Domain competence Attractiveness Usability Perceived value