1. Nigel G. Ward, Anais G. Rivera, Karen Ward, David G. Novick
Some Usability Issues and Research Priorities in Spoken Dialog Applications
Nigel G. Ward, Anais G. Rivera, Karen Ward, David G. Novick
Interactive Systems Group, The University of Texas at El Paso
Questions
Why are many dialog systems poor in usability?
are developers failing to use best practice?
are researchers missing the important issues?
Analysis Phase 1: Usability Event Analysis
Three labelers identified
115 times when the user experienced poor system usability
62 times when the subject experienced deft operator actions
and categorized them by severity and likely causes
Recognition and Understanding / +++ / +++*
direct effects on task completion, satisfaction
effects on user behavior (slow, considered)
effects on designer behavior (more explicit prompts, leading to time cost)
Time-Outs / + / ++*
delayed reaction when user wants a reprompt
Endpointing and Responsiveness / + / ++*
delayed reaction after the user finishes speaking
Generation and Synthesis / + / +*
conservative, invariant speaking rate, causing time-cost
pragmatic force unclear
Feedback / / +*
unclear indications of dialog activity/ status
weak, ineffective dialog control
Adaptation of Vocab., Speaking Rate… / /*
Prosodically-Marked Speech Acts / / +*
Detecting User Self-Talk / / +*
Other / + / + *
Research Issues Identified
Analysis Phase 2: Time Analysis
We compared the cummulative times for various dialog activities across the two conditions.
Methods
Each subject performed the same task twice,
once with an automated system, and
once with a human operator.
We observed and categorized the differences.
Advantages:
reveals missed opportunities, not just clear errors
usability-centered, not performance-centered
empirical, not theory-based
exploratory, not prematurely quantitative
(c.f. Walker et al. 200, Dybkjaer & Bernsen 2000, Moller 2005 …)
Dialog Activity
system-
human
human-
difference
normal system-side utterances
32%
37%
29%
normal user utterances
13%
41%
-6%
error-recovery related utterances
17% 4%
27%
system-side silences 6%
20%
user silences
14%
11%
16%
simultaneous talk 1% 0%
experimenter interventions 8%
Totals
100%
User
Operator
Spoken Dialog System
compare
Analysis Phase 3: Relating Usability Issues,
System Capabilites, and Research Issues
Clear,
expressive,
flexible
synthesis
Natural
language
generation
Accurate,
recognition
complete
Normally
users
behaving
sensitivity
Prosodic
Domain and Corpus
The domain was credit card billing support.
Subjects were given three structured tasks in each condition.
The automated system was based on Nuance Voice Builder.
The operator matched the system’s vocabulary and formality level.
20 subjects.
53 minutes total.
Concise
prompts
Adaptive
speaking
rate
Timeouts
* impact on time/task-completion/stress
Endpointing
feedback
Informative
Responsiveness
Easy
recovery
error
Summary
have identified outstanding research issues:
endpointing, responsiveness, appropriate
feedback, etc.
have shown how experiences with
applications can determine research priorities
Vocabulary
adaptation
Initial Observations
With the system, versus with human:
Task-completion was lower (58% vs 77%).
Time to completion per task was higher (avg. 130 vs 40 sec.)
Low stress and
cognitive load
Quick
accomplishment
High task
completion
Domain
competence
Attractiveness
Usability
Perceived value