Designing a Virtual Keyboard with Multi-Modal Access for People with Disabilities Vijit Prabhu Computer Science & Engineering Indian School of Mines Dhanbad, Jharkhand, India Girijesh Prasad Intelligent Systems Research Centre (ISRC) University of Ulster, Magee Campus, Derry, N. Ireland, United Kingdom 1

Outline I.I NTRODUCTION II.P ROPOSED VIRTUAL KEYBOARD A. The working of the VK B. Modes of input accesss C. The layout design III.E VALUATION AND TESTING A. Evaluation of the layout design B. User trials 1. Types of errors 2. Testing methodology IV. R ESULTS AND DISCUSSIONS V. C ONCLUSION 2 Vijit Prabhu & Girijesh Prasad Designing a Virtual Keyboard with Multi-Modal Access for People with Disabilities

Persons with speech and motor disorders face problems in expressing themselves in an easy and intelligible way. Computer based augmentative and alternate communication (AAC) systems are developed to assist them. A Virtual Keyboard (VK), also called as On Screen Assistive Keyboard is a commonly used AAC system. A VK is characterized by the input modalities and the layout. 3 Vijit Prabhu & Girijesh Prasad Designing a Virtual Keyboard with Multi-Modal Access for People with Disabilities Introduction

Introduction … Continued. Scanning and access switches require lesser motor control. Brain-computer interface (BCI) translates the intent of a subject measured from brain signals directly into control command. Eye tracking which uses fixed gaze or blinking to activate the switch or triggers. “Hex-o-Spell” [1] virtual keyboard is an EEG based Brain Computer Interface that uses machine learning techniques to identify brain signals. Dasher[2] is an information-efficient text-entry interface, driven by natural continuous pointing gestures. 4 Vijit Prabhu & Girijesh Prasad Designing a Virtual Keyboard with Multi-Modal Access for People with Disabilities

The Proposed Virtual Keyboard: The Working Fig. 1(a): Virtual Keyboard: pointer pointing at the sub circle 5 Vijit Prabhu & Girijesh Prasad Designing a Virtual Keyboard with Multi-Modal Access for People with Disabilities

The Proposed Virtual Keyboard: The Working Fig. 1(b) Expanded Sub Circle: pointer points at the character F 6 Vijit Prabhu & Girijesh Prasad Designing a Virtual Keyboard with Multi-Modal Access for People with Disabilities

The Proposed Virtual Keyboard: Modes of Input Any one or the combination of three incorporated technologies can be used for access. The Brain Computer Interfacing (BCI) based on EEG uses ‘vivid imagination’ of a motor activity as a trigger. The Eye Tracker technology uses prolonged gaze and/or eye blink as a trigger. Access switches use any active body part such as hand, foot, mouth or head as a trigger 7 Vijit Prabhu & Girijesh Prasad Designing a Virtual Keyboard with Multi-Modal Access for People with Disabilities

The Proposed Virtual Keyboard: Layout Design Different alphabets have different frequency of occurrence in English Text. [3] [4] T ABLE I. R ELATIVE FREQUENCY OF THE ALPHABETS IN E NGLISH T EXT 8 Vijit Prabhu & Girijesh Prasad Designing a Virtual Keyboard with Multi-Modal Access for People with Disabilities CharacterFrequencyCharacterFrequency A8.167%N6.749% B1.492%O7.507% C2.782%P1.929% D4.253%Q0.095% E12.702%R5.987% F2.228%S6.327% G2.015%T9.056% H6.094%U2.758% I6.966%V0.978% J0.153%W2.360% K0.772%X0.150% L4.025%Y1.974% M2.406%Z0.074%

The Proposed Virtual Keyboard: Layout Design Different positions of characters in the layout, require different amount of access activity. Design Principle- higher the frequency of occurrence of character, lower should be the amount of activity required to access it. Clashes in positioning of character were resolved based on the probability of the character being the first letter of the word. [5] 9 Vijit Prabhu & Girijesh Prasad Designing a Virtual Keyboard with Multi-Modal Access for People with Disabilities

The Proposed Virtual Keyboard: Layout Design More probable of being the first letter of word - assigned the lower sub-circle. T ABLE II: R ELATIVE FREQUENCIES OF THE FIRST LETTERS OF A WORD IN THE E NGLISH LANGUAGE 10 Vijit Prabhu & Girijesh Prasad Designing a Virtual Keyboard with Multi-Modal Access for People with Disabilities CharacterFrequencyCharacterFrequency A11.602%N2.365% B4.702%O6.264% C3.511%P2.545% D2.670%Q0.173% E2.000%R1.653% F3.779%S7.755% G1.950%T16.671% H7.232%U1.487% I6.286%V0.619% J0.631%W6.661% K0.690%X0.005% L2.705%Y1.620% M4.374%Z0.050%

The Proposed Virtual Keyboard: Layout Design 11 Vijit Prabhu & Girijesh Prasad Designing a Virtual Keyboard with Multi-Modal Access for People with Disabilities Position Access Activity Layout Alphab- etical Position Access Activity Layout Alphab- etical 1,12spacea3,14io 1,23eb3,25rp 1,34ac3,36uq 1,45nd3,47yr 1,56de3,58bs 1,67mf3,69xt 1,78gg3,710ju 2,13th4,15hv 2,24oi4,26cw 2,35sj4,37wx 2,46lk4,48vy 2,57fl4,59qz 2,68pm4,610zspace 2,79kn4,711del

Evaluation & Testing: Evaluation of Layout Design Given a random variable x, with the probability of occurrence p(x), the expected (mean) value of x: E(x) = Σ i x i p(x i ) x -‘total activity’ associated with a character p(x) -probability of the occurrence of the character in the text based on their relative frequencies Alphabetical layout, the expected amount of activity required is computed to be For the proposed layout, the same activity drops down Vijit Prabhu & Girijesh Prasad Designing a Virtual Keyboard with Multi-Modal Access for People with Disabilities

Evaluation & Testing: User Trials Types of Errors : Wrong Character Selection (WCS): user types a wrong character-needs to use backspace to remove the incorrect character -then type in the correct character. Wrong Sub Circle Selection (WSCS): user selects the wrong sub-circle-needs to use the ‘Go Back’ option provided in the sub circle Failed Selection (FS): user realizes that the pointer (arrow) has already moved beyond the correct sub circle /character 13 Vijit Prabhu & Girijesh Prasad Designing a Virtual Keyboard with Multi-Modal Access for People with Disabilities

Evaluation & Testing: User Trials Testing Methodology Users given the layout design in advance in print Explained the working of the VK One sentence to type using the soft switch at four different scan periods 2100ms, 1800ms, 1500ms and 1200ms. Just soft switch to select button as the pointer rotated and no prediction help was allowed. Only error free words and sentences with full-stops were considered for evaluation At the end of each typing session an informal chat THE QUICK BROWN FOX JUMPS OVER A LAZY DOG. 14 Vijit Prabhu & Girijesh Prasad Designing a Virtual Keyboard with Multi-Modal Access for People with Disabilities

Results and Discussions: TABLE IV: R ESULT OF USER TRIALS. 15 Vijit Prabhu & Girijesh Prasad Designing a Virtual Keyboard with Multi-Modal Access for People with Disabilities SubjectS1S2S3S4S5S6S7 Scan time 2100ms Wrong Sub Circle Wrong Character Failed Selection Total Errors Total Time Taken (min:s) 10:3814:5810:1910:0412:1411:3610:04 Scan time 1800ms Wrong Sub Circle Wrong Character Failed Selection Total Errors Total Time Taken (min:s) 09:1809:5108:5709:1508:4908:5809:04 Scan time 1500ms Wrong Sub Circle Wrong Character Failed Selection Total Errors Total Time Taken (min:s) 07:4807:5607:228:0809:5307:2207:47 Scan time 1200ms Wrong Sub Circle Wrong Character Failed Selection Total Errors Total Time Taken (min:s) 06:3007:4806:0106:3906:2506:0206:08

Conclusion: The VK developed has multiple modes of access, hence people suffering from different levels of disability may be able to use one or more of the access methods Evaluation of the layout design of the proposed VK clearly establishes the superior performance measured in terms of average per character activity required for composing text The user trials not only helped prove the robustness of the VK but also helped in performance evaluation of the layout 16 Vijit Prabhu & Girijesh Prasad Designing a Virtual Keyboard with Multi-Modal Access for People with Disabilities

Conclusion: Continued The various scan periods were also evaluated and an attempt was made to determine the most suitable scan times. User trials with the actual intended users making use of all three modalities is an utmost requirement. Comparative study with other existing VK like the flat VK will help in better evaluation of design Work is ongoing to combine BCI and eye-tracker modalities together to create a hybrid modality and evaluate its performance. 17 Vijit Prabhu & Girijesh Prasad Designing a Virtual Keyboard with Multi-Modal Access for People with Disabilities

References [1] Blankertz et al., The Berlin Brain-Computer Interface Presents The Novel Mental Typewriter Hex-O-Spell, In: Proc. of the 3rd International Brain-Computer Interface Workshop and Training Course 2006, September , Verlag der Technischen Universität Graz.. [2] Dasher-Information-Efficient Text Entry-Hanna Wallach, University of Cambridge/University of Pennsylvania. alternative/ alternative/ [3] Lewand, Robert (2000). Cryptological Mathematics. The Mathematical Association of America. p. 36. ISBN [4] Lee, E. Stewart; Essays about Computer Security; University of Cambridge Computer Laboratory, p. 181 [5] Letter Frequency, Wikipedia, Vijit Prabhu & Girijesh Prasad Designing a Virtual Keyboard with Multi-Modal Access for People with Disabilities

19