Ahmed Sabbir Arif York University, Toronto, Canada a.s.arif@gmail.com Text Entry Nomadicity Ambient Awareness, Handedness, and Error Adaptation Ahmed Sabbir Arif York University, Toronto, Canada a.s.arif@gmail.com

Character-Based Text Entry One character at a time: Non-ambiguous: Qwerty, ... Ambiguous: Multi-tap, ... We also have: Word-based text entry: handwriting, ... Phrase-based text entry: predictive, ...

Techniques: Timeline 1714 Mechanical Transcribing Machine 1830 Stenotype Machine 1870s Qwerty 1880s Typewriters 1936 Dvorak Personal Computers 1970s Mobile Keypads 1990s 1993 handwriting Apple Newton

Typewriters & Computers Qwerty Dvorak Qwerty vs. Dvorak: Path dependence?

Chording Keyboards Chording keyboards: Twiddler, ... Chorded keyers: Septambic Keyer, ... Was never widely accepted Stenotype machine was built by Karl Drais in 1830.

12-Key Mobile Keypad Multi-tap: T9 Text Entry Other techniques: Time-out, kill button T9 Text Entry Predictive Other techniques: TiltText, LetterWise, ... Other keypads: Less-Tap, Reduced Qwerty, ...

Reduced Sized Keyboards Mini-Qwerty or thumb keyboards Virtual or soft keyboards and keypads: Usually soft versions of physical keyboards A few use different methods: RollPad, ...

Projection Keyboards The concept immerged from IBM in 1995 Failed to get its anticipated attention Business decision rather than usability issues

Touchscreens Touchscreen devices are in demand Many replace physical keyboards Difficult to input text with virtual keyboards: No synthetic tactile feedback: vibration, ... More error prone: Error prevention techniques: character replacements, key-target resizing, ...

Nomadic Text Entry Non-stationary text entry Facts: Walking, driving, or commuting Facts: Slower and more erroneous [Hillman][Lin][Mustonen] Perfect task-parallelism is not possible [Meyer] Involves a limited peripheral resource – our eyes Creates competition for the attention between the device and the ambient environment from gettyimages.ca

Nomadic Text Entry Techniques Eyes-free: Gesture – performs well only when guided by auditory feedback [Brewster][Lumsden] Voice – error prone, heavyweight, performance drops when noisy, not realistic [Mankoff] [Brewster] Other: Chorded – takes time to master [Yatani] Wearable – not convenient, erroneous [Chamberlain] Synthetic tactile feedback improves touchscreen performance [Hoggan]

Nomadic Text Entry Ambient Awareness

Our Approach Reduce the competition for focus: Real-time, because Increase users’ awareness of ambient environment By providing real-time feedback on their surroundings Users already swap focus regularly between the text entry area and the keyboard [Arif] Real-time, because Users mostly occupied with instant spatial factors Human navigation system is a dynamic, egocentric representation [Wang]

Four Feedback Techniques Textual Visual Textual & visual Textual & visual via translucent keyboard

Textual Feedback In textual or written form Like turn-by-turn directional information by a GPS We used the WOz method during the experiment pre-set list containing messages, i.e. go straight, left turn ahead, etc.

(Textual &) Visual Feedback Live video using the embedded camera Textual feedback: Translucent (alpha = 0.5) in textual & visual Visual feedback area is not compromised Background doesn’t obscure text Users hold devices in 10–40° angles: Shows the next few metres of the path Allows short-term navigation Highly beneficial

Textual & Visual via Translucent KB A translucent virtual KB (alpha = 0.35) to show the visual feedback behind the keys Less focus swap within the interface Solid textual feedback area Background doesn’t obscure text

Similar Techniques No empirical study Video feedback obscures text input and the keys Input background keeps changing Causes confusion and irritation [Pilot study] Road SMS Type n Walk Walk and Text

User Study Apple iPhone 4 Textual feedback simulated by the Wizard Inputted the presented text phrases [Soukoreff] Textual feedback simulated by the Wizard Sent directly to the iPhone using a web app Initial walking speed Text entry Stationary and nomadic

Obstacle Path Mimics realistic walking environments: Forces users’ attention to the obstacles placed along the path Similar to [Barnard] Approx. 7.5×6 metres One lap 24 metres 13 turns, 3 intersections

Design 12 participants + Initial text entry and walking performance * 5 techniques * 15 phrases = 900 phrases + Initial text entry and walking performance Record: WPM, Total ER, Tfix automatic Lap time, total laps, wrong turns, bumps manual Wizard’s mistakes manual

Results: WPM, Total ER Significant effect No Significant effect Significant effect Textual, textual & translucent significantly faster

Discussion: WPM, Total ER Improved entry speed: Textual 14% Visual 8% Textual & visual, 6% Textual & translucent 11% compared to the baseline Textual & Textual & translucent significantly faster

Results: Walking Speed 159% more time to finish a lap while nomadic No significant effect Considered walking a secondary task

Results: Wrong Turns, Collisions No Significant effect Significant effect

User Feedback No significant effect Most felt “neutral” Wanted to use in challenging scenarios i.e. busy street Wanted to acquire the textual feedback system

Overall Performance Textual and textual & visual via translucent keyboard had better overall performance Improved entry speed by 14% and 11% Reduced error rates by 13% Textual – fastest walking speed (51.10 sec. per lap) Collision count was high (8 in total) Translucent – Low collision count (4 in total) Highest lap time (57.11 sec. per lap)

Nomadic Text Entry Handedness

Research Questions Do users input text while nomadic? [YES] do they use Both hands, The dominant hand, or The non-dominant hand? While nomadic & only one hand is available? [YES] do they use:

Survey Design Online – forums, e-mailing lists, ... Voluntary sampling method – users self-selected Screened for: Adult – 18+ Owns a handheld device Fluent in English English is their primary mobile OS language

User Demographics 133 users after pre-screening From 20 countries (4 continents) 46% female 71% touch-typists Avg. usage – 4hrs/day Avg. texts – 26/day Handedness: 90% right-handed 7% left-handed 3% ambidextrous

Devices & Keyboards 89% owned a smartphone All of them use physical or virtual Qwerty keyboard 11% use regular mobile devices Use physical or virtual 12-key keypad

While Walking 48% input text [almost everyday] Gender – no significant effect Male 51% Female 49% Age – significant effect 18–25 84.2% 26–35 51.0% 36–45 29.2% 45+ 25.0%

While Walking: Handedness Mobile handedness Handedness – no significant effect Both 54.7% Dominant 36.0% Non-dominant 9.3% Gender – no significant effect Age – no significant effect

While Walking: Hand Availability 88.7% input text Gender – no significant effect Age – significant effect 18-25 years old younger users are more committed Mobile handedness Handedness – no significant effect 78.7% dominant & 21.3% non-dominant Gender – significant effect 65% male & 92% female users prefer using dominant hand Age – no significant effect

While Commuting 90% input text [significantly higher] Gender – no significant effect Age – no significant effect Mobile handedness Handedness – no significant effect 53.5% both, 46.5% dominant, & 0% non-dominant Age – no significant effect

While Commuting: Hand Availability 85.8% input text [significantly higher] Gender – no significant effect Age – no significant effect Mobile handedness Handedness – no significant effect 85.9% dominant & 14.1% non-dominant Gender – significant effect 77% male & 95% female users prefer using dominant hand

While Driving 75% drive: 58% input text [more than walking!] Gender – no significant effect Age – no significant effect Mobile handedness Handedness – no significant effect 91.7% dominant, & 8.3% non-dominant Dominant hand use [significantly higher than walking/commuting] Age – no significant effect

Key Findings A large number of users input text while nomadic Commuting – significantly more Walking & driving – similar 96% drivers texts while driving: while doing such is illegal!! Age and gender do not influence the decision of texting + commuting or driving Age influence the decision of texting + walking No effect of handedness, age, or gender on mobile handedness ~50% use both & ~50% the dominant hand

Key Findings Almost all users continue typing while while the other hand is occupied [Commute] no effect of age or gender on this choice [Walk] usually younger users Most female users prefer using the dominant hand Most drivers prefer using the dominant hand

Conclusion & Recommendations Nomadic techniques must be properly investigated In order to meet users’ need Stationary handedness do not apply Mobile handedness change based on: Whether walking/commuting or driving Hand availability Gender Must develop/explore driver-safe techniques

Adapting to a Faulty Text Entry Technique Adaptation

Research Questions Do users adapt to a faulty system? Do system errors influence adaptation process? Is there a threshold for identification as error-prone? How do users adapt to an error-prone system?

Selection of the 7 Letters To guarantee uniformity all letters must appear the same number of times Humans can remember 7 chunks ±2 in short-term memory tasks Require relatively similar effort to draw with Graffiti and Unistrokes

Procedure Input letters with pen on tablet » Primary method: Graffiti (large above) » Alternate method: Unistrokes (small above) » Suggested usage of alternate if unreliable $1 Recognizer Injected 10%, 30%, 50% errors » Into three out of seven random Graffiti letters
» Different letters for each session

Results of the Pilot Studies Pilot Study 1 I) No reliable switching behaviour for < 10% errors Pilot Study 2 II) Error-prone letters not reliably identified Instead: Global switch to alternate method III) Extra care if error-prone letters were identified

Final Study Within-subject, 12 participants Initial session × 1 block × 280 letters + Final session × 3 blocks × 280 letters = 1120 letters/user, total 13440 letters

The Usage of Alternate Method Used Unistrokes instead of Graffiti - significant

Extra Care while Inputting More time than average to draw a letter - significant

For Both Measures Significant effect for extra care per letter with more time than average to draw a letter error rates Significant learning effect

Summary Users do gradually adapt to a faulty system Adaptation is proportional to error rate Error rate has to be >10% to be perceived as error-prone Users learn to avoid frequently occurring errors faster

Thank You! Questions?