The Effect of Miscommunication Rate on User Response Preferences Hua Ai, University of Pittsburgh Thomas Harris, Carolyn Penstein Rosé, Carnegie Mellon University 1 Motivation – Smart Home Security System Voice-control Air conditioner Remote Water Monitoring Smart Kitchen Garage Door Sensors Personalized Room Modes 2 Speech Modality Eyes & hands free Human-like natural interaction However, the undesirable error rate and the high cost of error resolution degrade system performance and frustrate users 3 Wizard-of-Oz Experiments However, How likely will the user follow the system prompts to resort direct manipulation? 4 Conclusions Miscommunication Rate: Low error rate (30%) Mid error rate (50%) High error rate (70%) User actions: Direct Manipulation Repeat Rephrase User Gam e Stereo While user does not always follow the system’s advice, it does have an effect on their choice regardless of misrecognition rate Significant effect of system response to miscommunication on the likelihood of user direct manipulation (P<.05) and Repeat (P<.05) Overall, users were significantly more likely to continue with speech interaction when the system ask the user to repeat (p <.01) While we would predict that users are more likely to resort to direct manipulation as miscommunication rate increased, our surprising finding was that users were most likely to resort to direct manipulation where communication success was least predictable. we found a significant effect of High versus Low Certainty on the likelihood of Direct Manipulation (p <.005) and likelihood of continuing with speech (p <.01) Prompt users to switch to direct manipulation when continuous errors occur: Direct manipulation serves as an interpretation of user’s speech input to train the speech recognizer Direct manipulation makes the user feel in control, thus increases user satisfaction rate Experimental Design: 3X2 within subjects Factorial Design explore the separate and joint effects of 3 miscommunication rate and 2 system response to miscommunications (askRepeat and showMe) on user behavior Experimental setups: 9 subjects, 4 females and 5 males, recruited on CMU campus. Each subject is required to complete 10 tasks with a stereo while staying “alive” in an eyes & hands busy video game. Sample task: turn the stereo on, play a CD, skip over a song, etc. Subjects are told that the stereo is an intelligent stereo that can accept commands either via direct manipulation or through speech commands. No vocabulary list is given. Subjects are randomly assigned to the 3 miscommunication rate conditions, 3 people per group. A wizard plays the pre-recorded system prompts to interact with the subjects. The wizard decides the faked miscommunication points according to a pre-defined random sequence. Then either askRepeat or showMe prompts are given. The subject can response by Direct Manipulation or speech (Repeat or Rephrase). System prompts: Miscommunication condition: [askRepeat] “Could you repeat that?” [showMe] “Could you show me what do you mean?” Normal condition: [ok] “OK, I’ll do that for you.” [Done] “I’m done!” “Show & Tell”