How Spread Works

Spread Spread stands for Speech and Phoneme Recognition as Educational Aid for the Deaf and Hearing Impaired Children It is a game used to visually motivate deaf and hearing impaired children to learn to speak

CLIENT How does Spread work? Record Selection SERVER Sphinx.wav file + current word Transcribe Scoring result Feedback

CLIENT Selection SERVER

Selection The user is presented with a screen showing the word to pronounce

Selection The user is presented with a screen showing the word to pronounce

Selection The user is presented with a screen showing the word to pronounce

CLIENT Recording Record Selection SERVER

Recording Recording begins once the user clicks the record button.

CLIENT Transmission Record Selection SERVER.wav file + current word

Transmission Transmission begins once the stop button is pressed. The wav file, the current word and the training phoneme are sent to the server for processing. transmission CLIENT K AA R SERVER Training Phoneme

CLIENT Transcribing & Sphinx Record Selection SERVER Sphinx.wav file + current word Transcribe

Transcribing Once the wav file arrives at the server, it is inputted into Sphinx in order to recognize what the user said Sphinx

Sphinx is a Java-based Hidden Markov Model speech recognition system developed by Carnegie Mellon University Sphinx

To decode the wav file, Sphinx needs three data sets – Acoustic Model – Dictionary – Language Model Sphinx Acoustic Model Dictionary Language Model

Acoustic Model The Acoustic Model maps sound features to units of speech called phonemes Derived through the sampling of a large data set of spoken words called a speech corpus K AA R

Dictionary The dictionary maps words into phonemes... CAN K AE N CAR K AA R CAT K AE T T...

Language Model The language model indicates the probability of a particular word appearing given the previous words – Not used since Spread only needs to recognize individual words

Decoding Sphinx in Spread is configured to detect what phonemes were pronounced by the user SPHINX K K AA R R

Increasing Accuracy To increase accuracy, Sphinx in Spread is only made to recognize a limited number of phonemes per level 7 levels means 7 individually configured Sphinxes Sphinx Level1 CAR, JAR, STAR… Sphinx Level2 BED, NET, TENT… Sphinx Level3 PLAY, PARTY, CIRCLE…

CLIENT Scoring Record Selection SERVER Sphinx.wav file + current word Transcribe Scoring

The server compares the decoded result against the expected result, taking note of the training phoneme Sphinx You said: K AA R You said: K AA R Expected: Training Phoneme

CLIENT Final result Record Selection SERVER Sphinx.wav file + current word Transcribe Scoring result Feedback

The result is sent over to the client to give feedback to the user

Preliminary results Tested with adult members of the hearing impaired community – Very positive. – "I wish I had this when I was learning speech" Problems: Too enthusiastic – Loud cheering noises reduced recognition rates

Preliminary results SPREAD was tested with hearing impaired students of the SPED division of the Batino Elementary School in Proj. 3, Quezon City – Accuracy testing and software evaluation

Working with the children Of the 40 students, only 5 volunteered to test the software – The children were generally shy and hesitant to perform the speech

Working with the children The children only knew very few words – They knew how to sign some of the words but not to vocalize them General mood was as if they were taking an exam that they were not prepared for

Working with the children Surprisingly, children were very good at conversational phrases – “Good morning” – “Good bye and thank you!”

Working with the teachers Teachers still need to help the students vocalize some words – System at yet cannot be left unsupervised with the students

Working with the teachers Noisy screen distracts students – Need to have a simpler screen to focus on

Recognition Rates Sphinx recognition rates were low – Hampered by noisy environment

Conclusion Need to work closely with SPED teachers on speech curriculum – Test on just recently learned words Conversational phrases – Hearing impaired children use simple phrases rather than words. – Conversational phrases spoken, other words signed UI improvements, simple is better Accuracy improvements urgently needed

The Spread Team

Image Sources Microphone - Crystal Project - Wave form -

Extra slides follow…

Scoring There are three possible outcomes – EXCELLENT – Good – Sorry

Scoring Getting the training phoneme correctly as well as the correct length of the phoneme gets an EXCELLENT score K AA R Expected: Sphinx You said: K AA R You said: K AA R 3 Phonemes Long Got the Training Phoneme

Scoring Note that Spread is only looking for the correct pronunciation of the training vowel K AA R Expected: Sphinx You said: K AA T You said: K AA T 3 Phonemes Long Got the Training Phoneme

Scoring Not getting the correct word length gets a Good score K AA R Expected: Sphinx You said: K AA R T You said: K AA R T 3 Phonemes Long Got the Training Phoneme

Scoring Not getting the training vowel means the user will have to try again – Length is no longer checked K AA R Expected: Sphinx You said: K AE R You said: K AE R 3 Phonemes Long Got the Training Phoneme Sorry =(

Updates SPREAD has undergone BETA testing with a group of hearing impaired adults – Testing of original (pass/fail) algorithm Results – Low recognition rates even for recognizable speech – Puzzling due to high recognition rates with lab speech

Recognition Rate WordRateClose word Apple60%Apple (60%) Art6%Bat (66%) Banana13%Apple(73%) Bat66%Bat (66%) Car0%Hand (46%) Fan0%Hand (53%) Hand20%Bat (33%) Jar0%Hand (60%) Lamb0%Apple (33%) Sofa0%Hand (46%) Star0%Fan (26%) Table0%Apple (46%) Van0%Art (26%) Wallet0%Hand (60%)

Analysis Microphone Lab test data Live data

Recommendations Better microphone/setup – Sphinx has preprocessing modules for less noise Per word recognition – Use creative word combinations to isolate training phoneme w/o having to go into per phoneme recognition Check out phoneme recognizers

Per phoneme recognition Per phoneme recognition is worse – Spread is highly dependent on full words for increased recognition rates Recognizing: Lamb2.wav I heard: ae ah m Recognizing: Lamb3.wav I heard: ae m Recognizing: Sofa1.wav I heard: s ow l ow Recognizing: Sofa2.wav I heard: s ae Recognizing: Sofa3.wav I heard: s ow hh aa Recognizing: Star1.wav I heard: ao t Recognizing: Star2.wav I heard: s d aa r Recognizing: Star3.wav I heard: s d aa r Recognizing: Table1.wav I heard: ah d l Recognizing: Table2.wav I heard: ae ah Recognizing: Table3.wav I heard: ae ah Recognizing: Van1.wav I heard: m ae Recognizing: Van2.wav I heard: m ae