1 Speech User Interfaces

2 Outline Review Review Motivation for speech UIs Motivation for speech UIs Speech recognition Speech recognition UI problems with speech UIs UI problems with speech UIs SpeechActs: Guidelines for speech UIs SpeechActs: Guidelines for speech UIs Speech UI design tools Speech UI design tools Multimodal UIs Multimodal UIs

3 Review Why do we prototype? Why do we prototype? get feedback on our design from customers – faster & cheaperget feedback on our design from customers – faster & cheaper Why use low-fi prototypes? Why use low-fi prototypes? traditional methods take too long & focus designers & customers on the wrong (visual) issuestraditional methods take too long & focus designers & customers on the wrong (visual) issues What is the Wizard of Oz technique? What is the Wizard of Oz technique? faking the interactionfaking the interaction What is the advantage of using informal tools like SILK, DENIM, & SUEDE? What is the advantage of using informal tools like SILK, DENIM, & SUEDE? advantages of electronic medium (editing, reuse, distribution, etc.)advantages of electronic medium (editing, reuse, distribution, etc.) faster than traditional UI toolsfaster than traditional UI tools do not focus designers/customers on the wrong issuesdo not focus designers/customers on the wrong issues ability to support testing & analysis of resulting dataability to support testing & analysis of resulting data

4 Motivation for Speech UIs: Pervasive Information Access Information & Services I-Land vision by Streitz, et. al.

5 UIs in the Pervasive Computing Era Future computing devices won’t have the same UI as current PCs Future computing devices won’t have the same UI as current PCs wide range of deviceswide range of devices small or embedded in environment small or embedded in environment often w/ “alternative” I/O & w/o screens often w/ “alternative” I/O & w/o screens information appliances information appliances I-Land vision by Streitz, et. al.

6 Information Access via Speech Read my important

7 Industry Leaders Nuance Corporation Nuance Corporation Nuance Applications: TellMe, … Applications: TellMe, …TellMe Users: Government, Computers- Microsoft, IBM, Users: Government, Computers- Microsoft, IBM,

8 Speech UI Motivation Smaller devices -> difficult I/O Smaller devices -> difficult I/O people can talk at ~ 90 wpm -> high speedpeople can talk at ~ 90 wpm -> high speed “Virtually unlimited” set of commands “Virtually unlimited” set of commands Freedom for other body parts Freedom for other body parts imagine you are working on your car & need to know something from the manualimagine you are working on your car & need to know something from the manual Natural Natural evolutionarily selected forevolutionarily selected for reading, writing, & typing are not (too new) reading, writing, & typing are not (too new)

9 Why are Speech UIs Hard to Get Right? Speech recognition far from perfect Speech recognition far from perfect imagine inputting commands w/ the mouse & getting the wrong result 5- 20% of the timeimagine inputting commands w/ the mouse & getting the wrong result 5- 20% of the time Speech UIs have no visible state Speech UIs have no visible state can’t see what you have done before or what affect your commands have hadcan’t see what you have done before or what affect your commands have had Speech UIs are hard to learn Speech UIs are hard to learn how do you explore the interface? how do you find out what you can say?how do you explore the interface? how do you find out what you can say?

10 Speech recognition Speech recognition the computer understanding what the customer is sayingthe computer understanding what the customer is saying Speech production (or synthesis) Speech production (or synthesis) the computer talking to the customerthe computer talking to the customer Speech UIs Require

11 Speech Recognition Continuous vs. non-continuous Continuous vs. non-continuous Speaker independent vs. dependent Speaker independent vs. dependent Speech often misunderstood by people Speech often misunderstood by people feedback via speech, facial expressions, & gesturefeedback via speech, facial expressions, & gesture Recognizers trained with real samples Recognizers trained with real samples often get gender-based problemsoften get gender-based problems Based on probabilities (HMMs - Bayes) Based on probabilities (HMMs - Bayes) trigrams of sounds or wordstrigrams of sounds or words Several popular recognizers Several popular recognizers Nuance, SpeechWorks, IBM ViaVoiceNuance, SpeechWorks, IBM ViaVoice

12 Speech Production Three frequency regions of great intensity visible on oscilloscope Three frequency regions of great intensity visible on oscilloscope come from larynx, throat, mouthcome from larynx, throat, mouth Two needed for recognition but “tinny” Two needed for recognition but “tinny” Can generate emotion affect in speech Can generate emotion affect in speech DemoDemo anger, disgust, gladness, sadness, fear, & surprise n/emot-speech.html anger, disgust, gladness, sadness, fear, & surprise n/emot-speech.html n/emot-speech.html n/emot-speech.html

13 Recognition Problems Good recognition Good recognition humans < 1% error rate on dictationhumans < 1% error rate on dictation top recognition systems get <1-X% error ratestop recognition systems get <1-X% error rates computers don’t use much context computers don’t use much context Key is to be application specific for lower error rates Key is to be application specific for lower error rates Background noise Background noise even worse recognition rates (20-40% error)even worse recognition rates (20-40% error) Speed Speed Better as hardware getting fasterBetter as hardware getting faster in 10 years gone from 5 high-end workstations required to some speech systems running on laptops or even PDAs in 10 years gone from 5 high-end workstations required to some speech systems running on laptops or even PDAs

14 More Recognition Problems Isolated, short words difficult Isolated, short words difficult common words become shortcommon words become short Segmentation Segmentation silly versus sill leasilly versus sill lea Spelling Spelling mail vs. male -> need to understand languag vs. male -> need to understand language

15 Speech UI Problems Speech UI no-nos Speech UI no-nos modes (no feedback)modes (no feedback) certain commands only work when in specific states certain commands only work when in specific states deep hierarchies (aka voice mail hell)deep hierarchies (aka voice mail hell) Verbose feedback wastes time/patience Verbose feedback wastes time/patience only confirm consequential thingsonly confirm consequential things use meaningful, short cuesuse meaningful, short cues Interruption Interruption half-duplex communication (i.e., no barge-in support)half-duplex communication (i.e., no barge-in support) Too much speech on the part of customer is tiring Too much speech on the part of customer is tiring Speech takes up space in working memory Speech takes up space in working memory can cause problems when problem solvingcan cause problems when problem solving

16 SpeechActs: Guidelines for Speech UIs Speech interface to computer tools Speech interface to computer tools , calendar, weather, stock quotes , calendar, weather, stock quotes Establish common ground & shared context Establish common ground & shared context make sure people know where they are in the conversationmake sure people know where they are in the conversation Pacing Pacing recog. delays are unnatural, make it clear when this occursrecog. delays are unnatural, make it clear when this occurs barge-in lets user interrupt like in real conversationsbarge-in lets user interrupt like in real conversations tapering of promptstapering of prompts progressive assistance: short errors messages at first, longer when user needs more helpprogressive assistance: short errors messages at first, longer when user needs more help implicit confirmation: include confirm in next commandimplicit confirmation: include confirm in next command

SpeechActs Video

18 Announcements Task analysis / Contextual inquiry HW Task analysis / Contextual inquiry HW average = 79/100, stdev. 8.4average = 79/100, stdev. 8.4 Low-fi user test due Monday Low-fi user test due Monday questionsquestions If you haven’t gotten a laptop yet, check with Wai-ling after class If you haven’t gotten a laptop yet, check with Wai-ling after class

19 SUEDE: Low-fi Prototyping for Speech-based UIs Supports design practice Supports design practice example scripts Wizard of Oz error simulation iterative design ( design-test- analysis ) Informal user interface Informal user interface no speech recognition/synthesis need not be programming expert fast & fluid design

machine prompt user response

21

22

23 SUEDE Summary SUEDE supports speech-based UI design SUEDE supports speech-based UI design moving from concrete examples to abstractionsmoving from concrete examples to abstractions allows designer to accept responses that aren’t exactly what they originally had in mindallows designer to accept responses that aren’t exactly what they originally had in mind embeds iterative design w/ design-test- analyzeembeds iterative design w/ design-test- analyze Designers using SUEDE need not be experts in speech recognition technology Designers using SUEDE need not be experts in speech recognition technology

24 One Vision of Future User Interfaces Star Trek style UI Star Trek style UI verbally ask the computer for informationverbally ask the computer for information may be common in mobile/hands-busy situationsmay be common in mobile/hands-busy situations problem: hard to design, build, & use!problem: hard to design, build, & use! requires perfect speech recognition & language understanding requires perfect speech recognition & language understanding

25 Our Vision of Future User Interfaces Multimodal, Context-aware UIs Multimodal, Context-aware UIs multimodalmultimodal uses multiple input modalities (speech & gesture) to disambiguate uses multiple input modalities (speech & gesture) to disambiguate user says “move it to this screen” while pointing user says “move it to this screen” while pointing context-awarecontext-aware apps can be aware of location, user, what they are doing, … apps can be aware of location, user, what they are doing, … people are talking -> don’t rely on speech I/O people are talking -> don’t rely on speech I/O Problem: how to prototype & test new ideas? Problem: how to prototype & test new ideas? Informal UI Design Tools!Informal UI Design Tools! combine Wizard of Oz & informal storyboarding combine Wizard of Oz & informal storyboarding

26 Multimodal Error Correction Dictation error correction study Dictation error correction study found users are better at correcting recognition errors with a different input modalityfound users are better at correcting recognition errors with a different input modality recognizer got it wrong the first time -> it will get it wrong the second timerecognizer got it wrong the first time -> it will get it wrong the second time hyperarticulating aggravates hyperarticulating aggravates Correct dictation errors with Correct dictation errors with vocal spelling, writing, typing, etcvocal spelling, writing, typing, etc

27 Summary Speech UIs Speech UIs may permit more natural computer accessmay permit more natural computer access allow us to use computers in more situationsallow us to use computers in more situations are hard to get to work wellare hard to get to work well lack of visible state, tax working memory, recognition problems, etc. lack of visible state, tax working memory, recognition problems, etc. UI tools are needed for speech UI design UI tools are needed for speech UI design Multimodal UIs address some of the problems with pure speech UIs Multimodal UIs address some of the problems with pure speech UIs help disambiguatehelp disambiguate help w/ correctionhelp w/ correction