Speech Recognition ECE5526 Wilson Burgos

Outline Introduction Objective Existing Solutions Implementation Test and Result Conclusion

Introduction Lots of $$$ are spent annually to improve language skills for non native speakers. Classes for ESOL (English Speakers of other languages) Lack of effective tools Speech recognition can help us in some areas

Objective Create a tool to help people learn to speak English correctly in an effective way. Engage people using new technology (Smartphone's) Using pocketsphinx, android and Text-to-speech technology Simple and intuitive to use Fun

Existing Solutions EyeSpeak - Pros Uses Native Speakers Pronunciation, pitch, timing & loudness Cons Difficult to use Runs only on windows

Concept of Operation The user from the main menu can start the game The game screen must lead the user through a series of words and log the number of positive responses (the score). Each word has a corresponding graphic to display. For example, the game might show the user a picture of a mountain The user has at most 30 seconds to respond

Development Environment Eclipse IDE with Android plugin Cygwin Emulator QEMU-based ARM emulator Runs the same image as the device Limitations No Camera support

Development Environment Actual Device

Implementation Using Java with the Android SDK Pocketsphinx Lightweight speech recognition decoder library Implemented in C

Android Architecture

Application Building Blocks Activity IntentReceiver Service ContentProvider

Application Architecture

Implementation QuizGameActivity The screen at the heart of the application—the game play screen. This screen prompts the user to answer a series of trivia questions and stores the resulting score information Uset Text-to-Speech technology to speak word if in simple mode

Implementation RecognizerTaskAudioTask PocketSphinx VUMeter

Implementation RecognizerTask Interfaces directly to the pocketsphinx library using JNI calls The Java Native Interface (JNI) enables the integration of code written in the Java programming language with code written in other languages such as C and C Consumes data from the audio queue, produced by the AudioTask Calls process_raw Scoring Based on positive detection of the utterance

Implementation AudioTask Interfaces directly to the audio peripherals to gather data Format Sample Rate 8000Hz, 16Bit PCM, 8192k buffer

PocketSphinx Very limited documentation Packaged the pocketsphinx into a shared library Created java shared library counterpart (jar) To be added to the android application Compiled using cygwin Initialized with custom dictionary and language model Speak2me.dic Speak2me.lm Loaded at startup from java code

Limitations Hardware memory In the sphinx4 demos the recognizer was active all the time gathering data. When running in the device the AudioRecord buffer fills up preventing the recognizer to be active all the time. Game needs to be responsive, how to solve this problem?

Limitations Hardware memory The VUMeter class calculates the energy of the sampled data, removing the DC offset with a filter. Detection logic was added to trigger end of utterance automatically with configurable lock/unlock thresholds The game timer automatically starts the recognizer after every given word Device Speed To improve detection the application uses the partial results to determine if a match has been found, doesn’t penalize if partial is incorrect.

Screenshots

Test and Results The cmu07a.dic recognized very poorly hub4_wsj_sc_3s_8k.cd_semi_5000 TOTAL Words: 91 Correct: 56 Errors: 46 TOTAL Percent correct = 61.54% Error = 50.55% Accuracy = 49.45% TOTAL Insertions: 11 Deletions: 3 Substitutions: 32 hub4_wsj_sc_3s_8k.cd_semi_5000adapt TOTAL Words: 91 Correct: 71 Errors: 25 TOTAL Percent correct = 78.02% Error = 27.47% Accuracy = 72.53% T TOTAL Insertions: 5 Deletions: 9 Substitutions: 11

Test and Results Using the custom corpus and creating custom language model the tool accurately detects speech in a timely fashion ~2s.

Installation Need to install custom lexical and language modeling files

Future Additions Adapt scoring based on pitch and phoneme recognition. Add different levels of difficulty Show progress reports

References Sams Teach Yourself Android Application Development, Lauren Darcey & Shane Conder (2010)