MTA SZTAKI An assistive interpreter tool using glove-based hand gesture recognition Department of Distributed Systems Péter Mátételki Máté Pataki Sándor Turbucz László Kovács My name is Peter Mátételki and let me present you our assistive interpreter tool that uses glove-based hand gesture recognition to translate sign language to speech.

pioneers of web technology in Hungary main fields MTA SZTAKI DSD MTA founded in 1827 SZTAKI founded in 1968 DSD founded in 1994 pioneers of web technology in Hungary main fields collaboration systems digital libraries web accessibility IoT crisis management plagiarism detection I am a research associate at the Hungarian Academy of Sciences, (MTA stands for it in Hungarian) Research Institute for Computer Science and Control, (abbreviated here as SZTAKI) Department of Distributed Systems.

Helen Keller Let me start with a quote from Helen Keller, the first deafblind person who earned a bachelor of arts degree, and who was also an activist supporting education for disabled people. Referring to her condition, she said: “Blindness separates us from things, but deafness separates us from people.” This very sentence leads us to the main problem of the deaf disabled group, and to the focus of our development. „Blindness separates us from things, but deafness separates us from people.”

hearing and speech impaired people Challenges Target groups hearing and speech impaired people deaf people (native language: sign language) Social problems problematic social integration isolated group To overcome the language barriers human sign language interpreter or: a good assistive tool Unlike for example the phisically disabled, hearing imparied people can not communicate with us. They speak another language, namely the sign language. For those who are deaf from birth, sign language is their native language. So when we try to speak with a deaf person, it’s the same situation as if we were trying to speak to a foreigner: we won’t understand each other because of the language barriers. It’s also a similar situation when you try to talk to your friend on a sidewalk of a busy road. You can’t hear each other so can not communicate, it’s very frustrating, right? This leads to serious social problems of the deaf, as they can not really integrate into the society, they form an isolated group. Today, their only chance to speak to the nondisabled is by being helped by a sign language interpreter. What our project aims is to give them a new assistive tool to let them communicate without any human assistance.

Silent Speech Translation (interACT) First, let’s see some other experiments in this field. The interACT project senses facial muscle movements with sensors attached to the skin and assigns text by matching the mimics to pre-recorded samples This solves the problem but probably most of you don’t want to walk on the street with these sensors glued to your face. Silent Speech Translation (interACT)

Eyes-Free Keypad Input Here you can see gesture controled keyboard. It is very simple: wherever you tap the display you get a 5. If you drag upwards you get a 2, if you drag downwards you get an 8 and so on. This is a great solution for numeric input but won’t work with a full keyboard to type letters. Eyes-Free Keypad Input

Keyboard Glove – University of Alabama The Keyboard Glove is made of a glove and micro-switches stitched under the glove. You can type by pushing the switches with your thumb. Here my problem is that this solution needs quite a bit of learning and I suspect that after a few hours of use your hand muscles will - most probably – be stiff, in pain. I think we need a more intuitive solution. Keyboard Glove – University of Alabama

is controlled by sign language (intuitive) talks for the deaf Talking Hands Our assistive tool is controlled by sign language (intuitive) talks for the deaf everyday use Features sign language to speech real time seamless Having seen the above projects our conclusion is that a good assistive tool for the deaf should be suitable for everyday use and follow their existing communication behavior. This means, that it should be controlled by sign language. So we suggest a solution for a real-time sign-language interpreter that translates the gestures to text and speech, realized in a way that can be used in everyday life. I hope that by now you became interested in our solution, so here it is:

TalkingHands We call it TalkingHands. Here you can see photos of our prototype showing sign language gestures. The solutions consists of a glove and a software component doing all the calculations as a mobile application.

custom gestures and text motion capture with the glove How does it work? sign language custom gestures and text motion capture with the glove gesture descriptor stream signal processing language processing text to speech Here is an overview of how it works: users can enter letters by showing hand gestures of the international fingerspelling alphabet AND can enter any text using custom gestures The glove captures the hand states transforms the hand state into gesture descriptors transmits the descriptors to the user’s mobile device The mobile application processes the gesture descriptors creates understandable text and reads it out loud On the following slides let me show you the glove and the processing algorithms with some more details.

Glove: gesture capturing 9DOF sensors signal fusion absolute position relative angles virtual hand gesture descriptors 30 Hagdil descriptor/sec Bluetooth For gesture capturing we use 3-axis accelerometers, gyroscopes and magnetometers on each major bones. We placed 2 sensors on each finger, 1 on the back of the hand and 1 on the wrist, a total of 12 sensors. We calculate the absolute position and the relative angles of the sensors, this results in a virtual hand. The virtual Hand is transformed to a gesture descriptor. For this we came up with our custom hand gesture descriptor language, that is called Hagdil. Each second, 30 Hagdil descriptors are transmitted to the mobile application. http://upload.wikimedia.org/wikipedia/commons/8/8c/Skeletal-hand_.jpg

Mobile application: processing Segmentation algorithm simple similarity repetition unkown descriptors sliding window kinetics based algorithm Context-sensitive auto-correction modified Levenshtein n-gram (1- & 2-grams) confusion matrix descriptors raw text text The Hagdil gesture descriptors are the input for the app. We need to pick the best ones from the stream, that’s the duty of the segmentation algorithm. We experimented with many approaches and found that the sliding window and the kinetics based algorithms produced the best results. The sliding window calculates an average in a sliding window. The kinetics based algorithm works by detecting the speed of the hand movements. We call the result of the segmentation raw text, as this text usually contains errors and typos. We tried existing spell-checkers to correct the raw text, but because of the different error characteristics they were not capable to do so. This is because the glove and segmentation errors are very different from the typos when typing on a keyboard. So we built a custom context-sensitive text-correction algorithm to transform the raw text into understandable text. Evaluation of these algorithms can be found in detail in the paper. (What we found the most interesting about the above algorithms, is that although the numeric results (so the distance of the original text and the raw text) resulting from the segmented stream are very similar, the context-sensitive correction algorithm can perform much better on the output of the kinetics based algorithm. This is caused by the two algorithm’s different correction- and error characteristics, so in the prototype we picked the more expensive segmentation algorithm.) speech

Scenarios, situations Can you guess, where can our TalkingHands help the deaf people?

work banking shopping public services healthcare education free time … Scenarios, situations work banking shopping public services healthcare education free time … Here are some scenarios that are, in general, problematic for the deaf people today. TalkingHands can enable them to handle normal everyday situations. They simply put on the glove, the phone in the pocket and signing can begin. (Lip reading Most hearing disabled people can lipread very well) Work: TalkingHands could improve employment opportunities for hearing disabled people, enhancing their social integration It does not only substitute a human signer (sign lang. Interpreter) but has a very positive impact on the user life (as the glove is at hand, they are not dependant on others) Grant for education: 120 hours + special curricula of 60 hours per semester

TalkingHands as a product two-handed gesture recognition Future plans TalkingHands as a product two-handed gesture recognition capture motion dynamics research in assistive technologies and IoT robotics remote manipulation Evolve Improve Enable

The project The project is executed as a consortium by two partners.

Péter Mátételki MTA SZTAKI DSD matetelki@sztaki. hu http://dsd. sztaki In case you are interested and need further information on TalkingHands, please contact me on the above address. I would be happy to answer all inquiries. Thank you for your attention!

www.youtube.com/watch?v=NhTUeZ16ZTw

Dactyl fingerspelling alphabet

Hagdil gesture descriptor

Architecture