An Open-Source Based Speech Recognition Android Application for Helping Handicapped Students Writing Programs Tong Lai Yu, Santhrushna Gande School of CSE, California State University, San Bernardino, California Ronald Yu Department of Computer Science, University of Southern California, Los Angeles, California

Overview Application allows students with difficulty typing because of physical disabilities to write Java code using voice recognition Application consists of a server and a client Student speaks into an android phone running the client Client sends data over to server via Wi-Fi Server processes data and displays the Java code on a workstation

Thread Synchronization on Android Client Main activity thread interacts with user, calls voice recognition engine, and puts voice rec results in a shared queue Communication thread reads data from the shared queue and sends it to the server Producer-Consumer paradigm used to make sure threads don’t interfere with each other

Workstation Server Threads Read thread reads data from network sent over by the android client and puts string in a shared buffer Processing thread reads data from shared buffer and processes it Synchronize threads with condition variables so threads wait for and signal to each other Simpler to implement than producer-consumer paradigm used for android client thread synchronization Only one thread can access the shared queue at a time

Keywords and Commands Voice Recognition engine commonly makes mistakes Server only processes keywords that occur frequently in Java (import, public, int, etc.) and program commands (compile, execute, etc.) Use a hashtable to map similar-sounding words to a keyword or command

Hashing Keywords Similar-sounding words are mapped to keywords in a hash table O(1) lookup Hash table stored in a text file which is read every time the program is started

Hashing Keywords (contd.) Hashing is not reversible Two different texts may map to the same hash index, leading to collision Very unlikely to happen Preprocessing to ensure collision-free If collision occurs between two texts, one text needs to be deleted or modified Ex. If Chava and Impact collide...

Hashing Commands Commands are stored in a similar hashing scheme Examples of commands: capital- Capitalizes the next word variable- Creates a new keyword to be stored in the hashing table space- Toggles whether a space is inserted after a word number- Displays line numbers undo- Undo the last keyword execute- Executes the program

Results Tested in a standard classroom at CSUSB Works well if only one or two students use the application in a class session

Conclusions Have developed an Android application using open-source tools Helps handicapped students write programs in Java using voice recognition Producer-consumer paradigm and condition variables are used to synchronize events Hashing is used to map ambiguous texts to specific keywords or commands Hash collisions are handled by preprocessing Can be extended to other languages such as C/C++

Demo