SigninGlove SDP 2016 February 29, 2016 Cumulative Design Review Department of Electrical and Computer Engineering Advisor: Professor Jackson

SigninGlove Team John Gontowicz Mathew Lau Aaron Gilbert Kacey Looney

Sign Language Translation Currently need translator Difficult to perform everyday activities Ordering food, paying at shops, etc. SigninGlove is a portable translator Especially useful when a translator is not available Kacey: Currently a deaf person needs a translator to accompany them when they are trying to interact with the hearing world. However in everyday situations, such as going grocery shopping or eating at a restaurant, translators are not available. In these situations there is usually a lot of confusion and misunderstandings occurring between the deaf individual and the hearing person they are trying to communicate with. The SigninGlove would help alleviate these misunderstandings and make it easier for the deaf person to be understood. This would be a great help in situations where a translator is not available.

Our Solution: Block Diagram Glove Raspberry Pi 2 Android App Sensor Data Transmission Sensor Inputs Sign Lookup Table Storage Connect to RPi GPIO Power Supply GUI Sensor filtering/ waveshaping Processed Data Transmission Data Collection Sensors Send over wireless connection Flex Sensor Sensor Data Processing Incoming Data Processing Palm/Finger contact sensors Flex, Contact sensor output processing Sign lookup Gyroscope Accelerometer, Gyro Signal Analysis String building Accelerometer

MDR Review Demonstrated end to end data transmission Used 3 flex sensors and 3 pressure sensors Sent sensor data to Raspberry Pi Raspberry Pi sent translated value to Android App Android App looked up value and displayed corresponding sign Kacey

Sensors Conductive Ink-Based Flex Sensors Force Sensitive Resistor 0 if straight, 1 if bent Force Sensitive Resistor 0 if no pressure, 1 if pressure is applied Kacey

Accelerometer/Gyroscope Berry IMU LSM9DS0 3 acceleration channels 2/4/6/8/16 g linear acceleration 3 angular rate channels 245/500/2000 dps angular rate Kacey

Battery Selection 5V, 2600 mAh=13Wh Powers Raspberry Pi and Sensors Majority of power consumed by Raspberry Pi Slightly more than 5W, battery would last about 5 hours Easily replaceable Mat

Raspberry Pi - Gyroscope Connects to RPi via I2C Bus Reads X, Y angles of IMU Using Complementary Filter Accounts for Gyro settling RPi Records min and max of XY angles (0 neg. angle, 1 positive) Example: Gyro Y angle Aaron max angle min angle

Raspberry Pi - Accelerometer Reads force acting upon IMU in g’s (set to 0-2g range) Gyro angle used to cancel gravity Records max/min acceleration in y, z direction → possible 0, 1 or 2 Example: Accel Y force Aaron max → 2 min → 0

Android App - New Table Currently searches through lookup table for flex/contact sensors Signs with same flex/contact but different gyro/accelerometer data then have a linked table Tree will be implemented since similar behavior Input Letter or Table Input→ Letter Table Key Value 11110111110001 a 00001000000001 b 11111000000000 k table John ... ...

Android App - Auto Update The app now updates automatically when a new packet is received; enabled using Handler class This will allow the user to use left hand to control when a new sign is starting/ending GUI Thread Waits idle until Handler’s message Updates GUI with Handler’s value Network Thread Continuously waiting for packets Sends packet value to Handler Class UDP Packet via WIFI John Handler

Demo - “G” and “Q”

Demo - “K” & “P”

Demo - “Z”

Future Work

Turing Test List of sentences commonly used in real world Test our glove side by side with an interpreter See how often the glove is understood compared to interpreter Test both pre-programmed sentences and on the fly sentences Kacey

FPR/Demo Day Presentation Entirely Portable Form sentences using pre-programmed words Add words/sentences in real time Button is part of the app App shows current and previous signs All

Adding of Words and Sentences The next step will be to add the functionality of adding new words to the app Holding the button will record the data on the RasPi and then send binary values to app App will then insert the values in the proper place within the tree for future use Pi Pi records data until button is released Turns values into binary values Sends to app as packet APP Holding button records data on Pi Pi sends back binary values App inserts data into tree Command to record Send packet Insert into tree

Android button The app will also have a button that will tell the Pi when the user is done with that sign This will eliminate the physical button and allow smoother signing Pi Pi records data until button is released Turns values into binary values Sends to app as packet APP Holding button samples data on Pi Pi sends back binary values App searches tree for correct sign Command to sample Search tree Send packet Show sign on GUI

Current Challenges/ Solutions Challenge: All the wires complicate signing right now Solution: PCB will eliminate wires and make portable along with battery Challenge: We can get some corrupt data through UDP Solution: TCP and error checking will eliminate any corrupt data Challenge: Accelerometer/Gyro data is not 100% consistent Solution: Calibrate threshold and digitization of data; implement most similar algorithm on app

Thank You Questions?

Portability of Glove Back Front Pi PCB Battery

Sample Turing Test Sentences Where’s the restroom? My name is _______. Yes No Can I have a _______? (Ordering) Hello I agree Please Thank you

Sensor Outputs Flex Sensor Pressure Sensor (Circle) Straight (3.2mV): <1.1V Flexed (3V): >1.1V Pressure Sensor (Circle) No Pressure (2.9mV): <2.4V Pressure (2.9V): >2.4V Pressure Sensor (Square) No Pressure (3.1mV): <3.2V Pressure (3V): >3.2V

Sensor Power Consumption Resistor Resistance Power (W) R1 (+) 15k 7.13E-04 R1 (G) 5.6k 3.26E-04 R2 27k 1.23E-04 R3 20k 9.94E-05 RM 100k 1.35E-04 Sensor 25k-150k 4.49E-05

Raspberry Pi - Inputs Flex Sensors Contact Sensors Berry IMU Sensor Power Push Button