1. SigninGlove SDP 2016 October 24, 2015 Preliminary Design Review
Advisor: Professor Jackson

2. SigninGlove Team
John Gontowicz
Mathew Lau
Aaron Gilbert
Kacey Looney

3. Translating Sign Language
No way to translate sign language without an interpreter
Makes it difficult for the severely deaf to communicate

4. Significance of the Project
70+ million people use sign language as their first language
ASL is said to be the 4th most used language in the US
The severely deaf only know Sign Language
Cannot communicate outside the deaf community

5. Current Solutions
Carry around a pen and paper to write what they are saying
Type message into phone and show it
An Interpreter is needed
No commercial product

6. Design Alternatives Smart Glove
Prototype by Northeastern
Used Hall Effect
Never completed
Kinect Sensor to watch user’s hand motions
Highly accurate
Non-Portable
MotionSavvy Uni
Uses tablet camera for interpreter
Requires interpreter to use

7. Our Solution: SigninGlove
Wearable Glove that translates sign language directly to user’s phone
Multiple sensors
Flex
Contact
Accelerometer
Gyroscope
Outputs to Pi which then sends a rough translation to the App
App shows the words that are being said
Ability to record new gestures for “home signs”

8. Our Solution: Block Diagram
Glove
Raspberry Pi 2
Android App
Sensor Data Transmission
Sensor Inputs
Sign Lookup Table Storage
Power Supply
GUI
Sensor filtering/ waveshaping
Processed Data Transmission
Data Collection
Sensors
Flex Sensor
Sensor Data Processing
Incoming Data Processing
Palm/Finger contact sensors
Flex, Contact sensor output processing
Sign lookup
Gyroscope
Accelerometer, Gyro Signal Analysis
String building
Accelerometer

9. Glove and Sensor Construction
Raspberry Pi 2
Android App
Sensor Data Transmission
Sensor Inputs
Sign Lookup Table Storage
Power Supply
GUI
Sensor filtering/ waveshaping
Processed Data Transmission
Data Collection
Sensors
Flex Sensor
Sensor Data Processing
Incoming Data Processing
Palm/Finger contact sensors
Flex, Contact sensor output processing
Sign lookup
Gyroscope
Accelerometer, Gyro Signal Analysis
String building
Accelerometer

10. Glove Construction Made of 4 main sensors
Flex (Yellow)
Contact (Red)
Gyroscope (Blue)
Accelerometer (Blue)
Wrist will hold an enclosure for the Raspberry PI and the power supply (Purple)

11. Flex Sensor Used on each finger Tells if finger is bent or straight
Resistance increases as sensor is flexed
Conductive Ink-based sensor
Only need to measure deflection in one direction
Durable sensors

12. Contact Sensors Used on fingertips, palm, and in between fingers
Used to tell if finger is making contact and where
Resistance is lowered as pressure is added
Force Sensitive Resistor
Simple setup, good pressure sensor
Low accuracy in terms of pressure amount

13. Gyroscope On backside of the hand Used to tell hand position
Used in both static and dynamic signs
Paired with Accelerometer to save space

14. Accelerometer Only used in dynamic signs
Will record both wrist movement and arm movement
Little movements will be filtered out, only concerned with big movements

15. Digitizing and Transmitting Sensor Data
Glove
Raspberry Pi 2
Android App
Sensor Data Transmission
Sensor Inputs
Sign Lookup Table Storage
Power Supply
GUI
Sensor filtering/ waveshaping
Processed Data Transmission
Data Collection
Sensors
Flex Sensor
Sensor Data Processing
Incoming Data Processing
Palm/Finger contact sensors
Flex, Contact sensor output processing
Sign lookup
Gyroscope
Accelerometer, Gyro Signal Analysis
String building
Accelerometer

16. Digitizing and Transmitting Sensor Data
Filter sensor output with active band pass filters
Use comparator or Schmitt Trigger circuit to make output of flex and pressure sensors binary
Extract data from accelerometer/gyroscope data register
Raspberry pi will receive high or low from flex and pressure sensors and acceleration data bytes from gyroscope/accelerometer
Lower delays as much as possible

17. Powering the Glove Portable USB PSU Flex sensors - 5V, 0.5W-1W
Accelerometer/Gyroscope V, ~5mA in normal mode
Raspberry Pi - 5V, 1.2A max current on 5V supply
Pressure sensors - 5V, 1mA max current
Total max current~ A
Total max power~11-12W

18. Powering the Glove cont.
Patriot Fuel+ 9000mAh PSU
45Wh on 5V~4 hours of continuous usage
1A and 2.5A ports
Smaller than Samsung Note 3 and Nokia lumia

19. Sensor Data Processing and Transmission
Glove
Raspberry Pi 2
Android App
Sensor Data Transmission
Sensor Inputs
Sign Lookup Table Storage
Power Supply
GUI
Sensor filtering/ waveshaping
Processed Data Transmission
Data Collection
Sensors
Flex Sensor
Sensor Data Processing
Incoming Data Processing
Palm/Finger contact sensors
Flex, Contact sensor output processing
Sign lookup
Gyroscope
Accelerometer, Gyro Signal Analysis
String building
Accelerometer

20. Sensor Data Processing and Transmission - High level overview
Incoming Signal Management Thread
Signal data processing Thread
Network packet construction and transmission thread
Retrieve next sign entry
Retrieve next processed entry
Sensor Input
Record incoming Signal
Process sign data
Create sign packet
new sign button Input
Store finished signal
Store processed sign
Transmit to App
Processing Queue
Transmission Queue
Memory
Raspberry Pi 2

21. Sensor Data Processing and Transmission- Raspberry Pi Input
Binary input for each flex and contact sensor
Filtered Gyro/Accelerometer sensor signals
Button or other sensor indicating switching signs
Incoming Signal Management Thread
Sensor Input
Record incoming Signal
new sign button Input
Store finished signal

22. Sensor Data Processing
Accelerometer/Gyroscope
Separate into 3 signals corresponding to x,y,z values
Break smoothed signal apart at direction changes
Create movement vectors <x, y z> from signal segments
Flex/Contact sensor
Create n-bit values, 1 bit per sensor
y(t)
y-position
y’(t)
y-velocity
up, down, up, down
y-accel
y’’(t)

23. Transmission Creates packets from transmission queue
Each sign is one packet
Packet contains segments of the sign signal
Packet Headers
Movement involved flag
Total segments/vectors
Possible other data

24. Android Application Android App Glove Raspberry Pi 2 Sensor Inputs
Sensor Data Transmission
Sensor Inputs
Sign Lookup Table Storage
Power Supply
GUI
Sensor filtering/ waveshaping
Processed Data Transmission
Data Collection
Sensors
Flex Sensor
Sensor Data Processing
Incoming Data Processing
Palm/Finger contact sensors
Flex, Contact sensor output processing
Sign lookup
Gyroscope
Accelerometer, Gyro Signal Analysis
String building
Accelerometer

25. Android App - High Level Overview
Receive UDP packets from Raspberry Pi with one packet per gesture
Protocol for gesture verification so no skipped gestures and no corrupt data
Lookup gesture object in Hash Table
Hash Table returns a word/String that’s used as text
Android API will be used for text to speech
Android app will also allow the user to access other features such as save new gesture 1 2 3 4 5 6

26. Android App - Flowchart
RasPi sends UDP packets
RasPi sends UDP packets
Protocol (ACK/NAK) gesture
Protocol (ACK/NAK) gesture
Hash Table Lookup for Gestures
Hash Table Lookup for Gestures
Speech
Android API
Gesture translated to word from Hash
Text

27. Android App - Lookup Algorithm
Since the movement vectors will not match perfectly, the lookup algorithm will calculate the difference between saved vectors and the given vector 1
The lookup algorithm will try to match a gesture based on binary value of pressure/flex sensors and then match similar movement vectors 2
Then the most similar gesture based on movement vector with the same pressure/flex sensor values will be picked and the associated String returned 3

28. Android App - Features Features enabled from Android App:
Easy to use and simple GUI
Enable new gesture mode that will save a new gesture (next gesture sent from the glove) into the Hash Table
Enable new sentence mode that will save a sentence (saved associated gestures together as one sentence)
Real-Time translation mode (gesture translated to speech in Real-Time)
Ability to correct a misinterpreted translation 1 2 3 4 5

29. Alternative to Android App
wtopic.php?f=38&t=68693
Alternative to Android App
Offline Text to speech for android OS usable on Raspberry Pi
Would bypass need for smartphone altogether and output sign language through raspberry pi speaker
Requires Raspberry Pi to do signal processing, translating, and storing word database
Android app would only be used to configure glove

30. Proposed MDR Deliverables
Demonstration of Glove Sensors
Flex and Contact sensors return binary values for flexed/unflexed and open/closed
Accelerometer and Gyroscope showing data for position of hand and arm movement
Demonstration of Pi transmitting to app
Using simulated data Pi sends a “sign” to the app
App displays the translated “sign”
Flex: 0 1 1
Contact: Gyro: -12dps 9 dps 15dps Accelerometer: -.15g .33g .27g

31. Thank You
Questions?