Hands-free Eyes-free Text Messaging Derek Woodman, Jeffrey Gehring, Abdullah Alshalan Mobile Computing Fall 2010

Introduction Our goal is to make an android application that makes it save to drive while texting Common place to have a hand-free system for placing phone calls We are going to deliver a system that does this for texting The system will be hands-free and eyes-free so the driver still focus on driving 11/24/2010

How? We are going to use several APIs available in the android operating system TTS – text to speech STT – speech to text Messaging API Android provides the tools. We are going to put them together and test it. 11/24/2010

Testing Most papers we have seen are concerned with how fast or accurate the speech system works We are going to focus on how well it improves the user’s attention – the main goal! We are using 2 different experiments to measure the user’s attention 11/24/2010

Experiment 1- Stop Light Simulator Shown picture of a stoplight The light will change periodically User will be asked to react to the light changing 3 groups No texting Regular texting Voice texting Measure the reaction time 11/24/2010

Experiment 2 – Video Recall We have an 10-minute video from within a car driving in Tempe. We aim to show this video in three situations: no texting, regular text and voice texting, we then ask the participant to answer question of things seen in the video. Our goal is to see if the group that does the voice texting will have a closer score to the no-texting group than the group that does the regular texting 11/24/2010

Tlk or txt? Using voice input for SMS composition Anna L. Cox, Paul A. Cairns, Alison Walton, and Sasha Lee. 2008. Personal Ubiquitous Comput. 12, 8 (November 2008), 567-588. Presented by : Derek Woodman

Introduction Mobile phone text messaging as become a ubiquitous form of communication Key-press method of text entry inherently restrictive as it ties the user both visually and physically to phone This paper explores the use of speech recognition as an alternative method of text entry Two experiments: Different combinations of voice and key-press input for both navigating menus and actual text entry Explore text entry while user is unable to see the screen

Current Text Entry Mutli-tap Predictive Enter letter by pushing number multiple times to cycle through the letter represented by it HELLO 4-4, 3-3, 5-5-5, 5-5-5, 6-6-6 Predictive Enter word by pressing each letter once and then selecting between words that map to the same key sequence GONE / GOOD 4-6-6-3 / 4-6-6-3 11/24/2010

Speech Recognition People naturally speak faster than they type Normal speech is approximately 200 WPM Multitap about 11 WPM Predictive about 26 WPM However, error rates of using speech recognition is much higher Text entry time is much faster for speech recognition Time spent detecting and correcting errors longer

Experiment Setup Emulator of a mobile phone on a computer Number pad of a keyboard was used as the keypad of the mobile phone Speech recognition and key-press versions used same emulator Dragon Natural Speaking 7 was used to speech recognition – 99% accurate when trained

Experiment 1 Navigate through the phone menu manually for with voice to start entering a new text message Key-press (K) – hit menu, find text message option, hit select, find option for new message, hit select, search address book for user to send to, hit select Speech (S) – say create message, say person’s name to send to, say select if correct Enter text message with multitap (M), predictive (P) or speech (S) All combinations tested – KM, KP, KS, SM, SP, SS

Experiment 1 36 participants selected Mixture of multitap and predictive text experience, but none with speech input experience Trained on the system before experimentation occurred All tested individually All tested with the same messages Displayed the message to send (6-10 word phrases) Ex – ‘Meet me outside in 5 minutes’ Asked to review until could remember it Then asked to start experiment – message still displayed

Experiment 1 Results  time taken to complete test When asked, 50% preferred the all speech input method although keypress/speec h was better NASA TLX task workload 

Experiment 2 Text entry with reduced feedback Only focused on text entry, no menu navigation 3 conditions tested Normal view of keypad and screen (control) No view of keypad (NK) No view of keypad or display (NKND) More aligned with how a user would use a phone when they are trying to multitask

Experiment 2 Each test user placed in a mutlitap, predictive, or speech input group Users entered their own desired messages under each of the 3 conditions (control, NK, NKND) – not same message each time The order of conditions were randomly assigned to eliminate the affect of ordering on the test

Experiment 2 Results Words per minute Error Rates

Conclusion Results show that the addition of a speech modality could enhance the text messaging interface Spoken text-entry produced the fastest task completion times Majority of the participants preferred spoken interaction Spoken text-entry yielded a much higher performance than other methods, but the number of errors with speech increased This study did not focus on the effects of errors or error correction strategies

Eyes-free Text Entry on a Touchscreen Phone Hussain Tinwala and I. Scott MacKenzie Proceedings of the IEEE Toronto International Conference – Science and Technology for Humanity – TIC-STH 2009, pp. 83-89. New York: IEEE Presented by: Abdullah Alshalan

Their work in a nutshell An eyes-free text entry technique for touchscreen mobile phones. Their method uses Graffiti strokes entered using a finger on a touchscreen. Although visual feedback is present, eyes-free entry is possible using auditory and tactile stimuli. In eyes-free mode, entry is guided by speech and non- speech sounds, and by vibrations. A study with 12 participants was conducted using an Apple iPhone. Entry speed, accuracy, and stroke formations were compared between eyes-free and eyes- on modes. Entry speeds reached 7.00 wpm in the eyes-on mode and 7.60 wpm in the eyes-free mode. Text was entered with an overall accuracy of 99.6%. KSPC was 9% higher in eyes-free mode.

Introduction Phones with physical buttons are constrained since all interaction involves pre-configured hardware. Touchscreen phones use software interfaces making them highly customizable and multipurpose. HOWEVER, user's ability to engage the tactile and proprioceptive sensory channels during interaction is reduced. THEREFORE, the demand on the visual channel is increased and this compromises the "mobile" in "mobile phone"

Introduction (Cont’d) With physical buttons, the tactile and proprioceptive feedback enables users to build a spatial motor-memory map, which allows them to carry out basic tasks eyes- free. Touchscreen phones are more visually demanding making it difficult to use them when engaged in a secondary task, such as walking, attending a meeting, or shopping, and very difficult for the visually-impaired individuals.

So the idea is to bridge the GAP

Eyes-Free Text Entry On A Touchscreen Main requirement: to support text entry without the need to visually monitor or verify input Used Text-entry technique: stroke-based alphabets (Graffiti)

Graffiti Input Using Fingers (The interface)

Graffiti Input Using Fingers (Text Entry Interaction) The user draws a stroke on the screen. Digitized ink follows the user's finger during a stroke. At the end of a stroke, the application attempts to recognize the stroke and identify the intended character, and gives one the following feedback: Recognized stroke: character is spoken Double-tap for SPACE: soft beep Unrecognized stroke: vibration Delete stroke (←): erasing sound

Implementation Software Architecture Hardware Infrastructure: Apple iPhone and an Apple MacBook host system. The host was used for data collection and processing. The two devices communicated via a wireless link over a private, encrypted, ad-hoc network. Software Architecture The host listened for incoming connections from the iPhone. Upon receiving a request and establishing a connection, the program reads a series of 500 phrases ranging from 16 to 43 characters. The host randomly selected a phrase and presented it to the participant for input.

Implementation

Experiment 12 Participants Three phases: training, eyes-on, eyes-free. Participants were instructed to enter text "as quickly and accurately as possible”. The software recorded time stamps for each stroke, per-character data, per-phrase data, ink trails of each stroke, and some other statistics.

Results (Speed)

Results (Accuracy [MSD, KSPC])

Results (Accuracy [MSD, KSPC])

Results (Stroke Analysis)

Conclusion A finger-based text entry technique for touchscreen devices combining single-stroke text (Graffiti) with auditory and vibro-tactile feedback was presented. An overall entry speed of 7.30 wpm with speeds being 9% higher (7.60 wpm) in the eyes-free mode. Error rates in the transcribed text were not significant between entry modes. Participants entered text with an accuracy of 99.6%. KSPC analyses revealed that eyes-free text entry required an average of 9% more strokes per phrase.

Speech-Based Interaction with In-Vehicle Computers: The Effect of Speech-Based E-Mail on Drivers' Attention to the Roadway Lee, John D., Brent Caven, Steven Haake, and Timothy L. Brown. "Speech- Based Interaction with In-Vehicle Computers: The Effect of Speech-Based E- Mail on Drivers' Attention to the Roadway." Human Factors: The Journal of the Human Factors and Ergonomics Society 43.4 (2001): 631-640. Presented by: Jeffery Gehring

Watch Jeffery’s Video 11/24/2010

Q&A Thank you! Any questions? 11/24/2010

Speech-Based Interaction with In-Vehicle Computers: The Effect of Speech-Based E-Mail on Drivers' Attention to the Roadway Lee, John D., Brent Caven, Steven Haake, and Timothy L. Brown. "Speech- Based Interaction with In-Vehicle Computers: The Effect of Speech-Based E- Mail on Drivers' Attention to the Roadway." Human Factors: The Journal of the Human Factors and Ergonomics Society 43.4 (2001): 631-640. Presented by: Jeffery Gehring

Introduction Many cars have computer systems in them, such as a blue-tooth system for hands-free phone calls Hands free, eyes free systems seem like they should decrease driver distraction These systems can cause driver distraction Paper studied amount of distraction caused by a hands-free, eyes-free e-mail system

Method Created two e-mail systems, one complex and one simple Complexity of the e-mail system was determined by the number of options at a menu Simple system – 3 menu levels, 2 options per menu, Complex system – 4 to 7 options Used a person to simulate Speech-to-Text portion of e-mail system to eliminate distraction due to errors 24 participates in a series of 5-7 minute scenarios, participates used either the complex or simple e-mail system

Method Ran scenarios in a Hyperion Technologies Vection Research driving simulator In simulator, participates followed a lead vehicle that slowed down periodically Time between when lead vehicle slow down and when participate slow down used to measure reaction time After running a scenario, users filled out a questionnaire asking a series of situation awareness probe questions

Results – Average Reaction Times No e-mail system: 1.01 seconds Simple e-mail system: 1.23 seconds Complex e-mail system: 1.41 seconds Users rated distraction level was higher with complex e-mail system then with simple e- mail system Distraction Ratings from Questionnaire: No e-mail system: 27.1 Simple e-mail system: 40.7 Complex e-mail system: 53.3

Discussion Using the e-mail system cause slower reaction times, and increased driver distraction Users realizing they are distracted is good, can stop using system in situations that require more focus Complexity of system is important, the more complexity causes slower reaction times and more distraction

Discussion Using an actual Speech-to-Text system could increase driver distraction Users knew they were being observed, causing them to focus on driving more then normal With more use of the system, users could learn how to focus more on driving while using the system Other complexities that weren't changed in this study could also affect distraction levels System not measured verse using a hand-held device