“Show me what you meant”: Mode-switching prompts in a multi-modal dialog system with distractions Thomas Harris & Hua Ai October 25, 2005

Motivation A main benefit of multi-modal systems: flexibility. –Hands or eyes busy etc: use speech. –Poor recognition etc: use direct manipulation. The user makes predictions about the interaction and chooses modes accordingly. We would like to inform the user’s predictions/decisions for even more efficient communication. We would also like to dialog with the user to better understand and learn from multi-modal communication.

Questions Under what circumstances does a user switch between speech and direct manipulation? How influential would prompting a modality switch be? Is it useful in some conditions to prompt a modality switch? Can we reliably characterize some modality switches as grounding mechanisms?

Hypotheses Based on (Oviatt 1996), we believe that people switch from speech to direct manipulation only after successive serial speech errors. We believe that strategic mode switching prompts will improve task efficiency as well as subjective measures of the task experience. We also believe that mode switching, when conditioned by system prompts, will lead to reliable parallel signals for multi-modal learning/grounding.

Experimental Setup A common task – manipulating a stereo system while driving. We have –a driving simulator that runs on a pc –an actual shelf stereo system that you can control manually –A dialog system with headset microphone that controls the same stereo system

Potential Dependant Variables How distracting a task. How errorful the recognition. At what point to introduce mode-switching prompts.