1. MTA SZTAKI
An assistive interpreter tool using glove-based hand gesture recognition
Department of Distributed Systems
Péter Mátételki
Máté Pataki
Sándor Turbucz
László Kovács
My name is Peter Mátételki and let me present you our assistive interpreter tool that uses glove-based hand gesture recognition to translate sign language to speech.

2. pioneers of web technology in Hungary main fields
MTA SZTAKI DSD
MTA founded in 1827
SZTAKI founded in 1968
DSD founded in 1994
pioneers of web technology in Hungary
main fields
collaboration systems
digital libraries
web accessibility
IoT
crisis management
plagiarism detection
I am a research associate at the
Hungarian Academy of Sciences, (MTA stands for it in Hungarian)
Research Institute for Computer Science and Control, (abbreviated here as SZTAKI)
Department of Distributed Systems.

3. Helen Keller
Let me start with a quote from Helen Keller, the first deafblind person who earned a bachelor of arts degree, and who was also an activist supporting education for disabled people. Referring to her condition, she said: “Blindness separates us from things, but deafness separates us from people.”
This very sentence leads us to the main problem of the deaf disabled group, and to the focus of our development.
„Blindness separates us from things, but deafness separates us from people.”

4. hearing and speech impaired people
Challenges
Target groups
hearing and speech impaired people
deaf people (native language: sign language)
Social problems
problematic social integration
isolated group
To overcome the language barriers
human sign language interpreter
or: a good assistive tool
Unlike for example the phisically disabled, hearing imparied people can not communicate with us. They speak another language, namely the sign language. For those who are deaf from birth, sign language is their native language. So when we try to speak with a deaf person, it’s the same situation as if we were trying to speak to a foreigner: we won’t understand each other because of the language barriers.
It’s also a similar situation when you try to talk to your friend on a sidewalk of a busy road. You can’t hear each other so can not communicate, it’s very frustrating, right?
This leads to serious social problems of the deaf, as they can not really integrate into the society, they form an isolated group.
Today, their only chance to speak to the nondisabled is by being helped by a sign language interpreter.
What our project aims is to give them a new assistive tool to let them communicate without any human assistance.

5. Silent Speech Translation (interACT)
First, let’s see some other experiments in this field.
The interACT project
senses facial muscle movements with sensors attached to the skin and
assigns text by matching the mimics to pre-recorded samples
This solves the problem but probably most of you don’t want to walk on the street with these sensors glued to your face.
Silent Speech Translation (interACT)

6. Eyes-Free Keypad Input
Here you can see gesture controled keyboard.
It is very simple: wherever you tap the display you get a 5. If you drag upwards you get a 2, if you drag downwards you get an 8 and so on.
This is a great solution for numeric input but won’t work with a full keyboard to type letters.
Eyes-Free Keypad Input

7. Keyboard Glove – University of Alabama
The Keyboard Glove is made of a glove and micro-switches stitched under the glove.
You can type by pushing the switches with your thumb.
Here my problem is that this solution needs quite a bit of learning and I suspect that after a few hours of use your hand muscles will - most probably – be stiff, in pain.
I think we need a more intuitive solution.
Keyboard Glove – University of Alabama

8. is controlled by sign language (intuitive) talks for the deaf
Talking Hands
Our assistive tool
is controlled by sign language (intuitive)
talks for the deaf
everyday use
Features
sign language to speech
real time
seamless
Having seen the above projects our conclusion is that a good assistive tool for the deaf should be suitable for everyday use and follow their existing communication behavior.
This means, that it should be controlled by sign language.
So we suggest a solution for a real-time sign-language interpreter that translates the gestures to text and speech,
realized in a way that can be used in everyday life.
I hope that by now you became interested in our solution, so here it is:

9. TalkingHands We call it TalkingHands.
Here you can see photos of our prototype showing sign language gestures.
The solutions consists of a glove and a software component doing all the calculations as a mobile application.

10. custom gestures and text motion capture with the glove
How does it work?
sign language
custom gestures and text
motion capture with the glove
gesture descriptor stream
signal processing
language processing
text to speech
Here is an overview of how it works:
users can enter letters by showing hand gestures of the international fingerspelling alphabet
AND can enter any text using custom gestures
The glove
captures the hand states
transforms the hand state into gesture descriptors
transmits the descriptors to the user’s mobile device
The mobile application
processes the gesture descriptors
creates understandable text
and reads it out loud
On the following slides let me show you the glove and the processing algorithms with some more details.

11. Glove: gesture capturing
9DOF sensors
signal fusion
absolute position
relative angles
virtual hand
gesture descriptors
30 Hagdil descriptor/sec
Bluetooth
For gesture capturing we use 3-axis accelerometers, gyroscopes and magnetometers on each major bones.
We placed 2 sensors on each finger, 1 on the back of the hand and 1 on the wrist, a total of 12 sensors.
We calculate the absolute position and the relative angles of the sensors, this results in a virtual hand.
The virtual Hand is transformed to a gesture descriptor.
For this we came up with our custom hand gesture descriptor language, that is called Hagdil.
Each second, 30 Hagdil descriptors are transmitted to the mobile application.

12. Mobile application: processing
Segmentation algorithm
simple similarity
repetition
unkown descriptors
sliding window
kinetics based algorithm
Context-sensitive auto-correction
modified Levenshtein
n-gram (1- & 2-grams)
confusion matrix
descriptors
raw text
text
The Hagdil gesture descriptors are the input for the app.
We need to pick the best ones from the stream, that’s the duty of the segmentation algorithm.
We experimented with many approaches and found that the sliding window and the kinetics based algorithms produced the best results.
The sliding window calculates an average in a sliding window.
The kinetics based algorithm works by detecting the speed of the hand movements.
We call the result of the segmentation raw text, as this text usually contains errors and typos.
We tried existing spell-checkers to correct the raw text, but because of the different error characteristics they were not capable to do so. This is because the glove and segmentation errors are very different from the typos when typing on a keyboard.
So we built a custom context-sensitive text-correction algorithm to transform the raw text into understandable text.
Evaluation of these algorithms can be found in detail in the paper.
(What we found the most interesting about the above algorithms, is that although the numeric results (so the distance of the original text and the raw text) resulting from the segmented stream are very similar, the context-sensitive correction algorithm can perform much better on the output of the kinetics based algorithm. This is caused by the two algorithm’s different correction- and error characteristics, so in the prototype we picked the more expensive segmentation algorithm.)
speech

13. Scenarios, situations
Can you guess, where can our TalkingHands help the deaf people?

14. work banking shopping public services healthcare education free time …
Scenarios, situations
work
banking
shopping
public services
healthcare
education
free time …
Here are some scenarios that are, in general, problematic for the deaf people today.
TalkingHands can enable them to handle normal everyday situations.
They simply put on the glove, the phone in the pocket and signing can begin.
(Lip reading
Most hearing disabled people can lipread very well)
Work: TalkingHands could improve employment opportunities for hearing disabled people, enhancing their social integration
It does not only substitute a human signer (sign lang. Interpreter) but has a very positive impact on the user life (as the glove is at hand, they are not dependant on others)
Grant for education: 120 hours + special curricula of 60 hours per semester

15. TalkingHands as a product two-handed gesture recognition
Future plans
TalkingHands as a product
two-handed gesture recognition
capture motion dynamics
research in assistive technologies and IoT
robotics
remote manipulation
Evolve
Improve
Enable

16. The project
The project is executed as a consortium by two partners.

17. Péter Mátételki MTA SZTAKI DSD matetelki@sztaki. hu http://dsd. sztaki
In case you are interested and need further information on TalkingHands, please contact me on the above address. I would be happy to answer all inquiries.
Thank you for your attention!

19. Dactyl fingerspelling alphabet

20. Hagdil gesture descriptor

21. Architecture