SigninGlove SDP 2016 October 24, 2015 Preliminary Design Review Advisor: Professor Jackson

SigninGlove Team John Gontowicz Mathew Lau Aaron Gilbert Kacey Looney

Translating Sign Language No way to translate sign language without an interpreter Makes it difficult for the severely deaf to communicate

Significance of the Project 70+ million people use sign language as their first language ASL is said to be the 4th most used language in the US The severely deaf only know Sign Language Cannot communicate outside the deaf community

Current Solutions Carry around a pen and paper to write what they are saying Type message into phone and show it An Interpreter is needed No commercial product

Design Alternatives Smart Glove Prototype by Northeastern Used Hall Effect Never completed Kinect Sensor to watch user’s hand motions Highly accurate Non-Portable MotionSavvy Uni Uses tablet camera for interpreter Requires interpreter to use

Our Solution: SigninGlove Wearable Glove that translates sign language directly to user’s phone Multiple sensors Flex Contact Accelerometer Gyroscope Outputs to Pi which then sends a rough translation to the App App shows the words that are being said Ability to record new gestures for “home signs”

Our Solution: Block Diagram Glove Raspberry Pi 2 Android App Sensor Data Transmission Sensor Inputs Sign Lookup Table Storage Power Supply GUI Sensor filtering/ waveshaping Processed Data Transmission Data Collection Sensors Flex Sensor Sensor Data Processing Incoming Data Processing Palm/Finger contact sensors Flex, Contact sensor output processing Sign lookup Gyroscope Accelerometer, Gyro Signal Analysis String building Accelerometer

Glove and Sensor Construction Raspberry Pi 2 Android App Sensor Data Transmission Sensor Inputs Sign Lookup Table Storage Power Supply GUI Sensor filtering/ waveshaping Processed Data Transmission Data Collection Sensors Flex Sensor Sensor Data Processing Incoming Data Processing Palm/Finger contact sensors Flex, Contact sensor output processing Sign lookup Gyroscope Accelerometer, Gyro Signal Analysis String building Accelerometer

Glove Construction Made of 4 main sensors Flex (Yellow) Contact (Red) Gyroscope (Blue) Accelerometer (Blue) Wrist will hold an enclosure for the Raspberry PI and the power supply (Purple)

Flex Sensor Used on each finger Tells if finger is bent or straight Resistance increases as sensor is flexed Conductive Ink-based sensor Only need to measure deflection in one direction Durable sensors

Contact Sensors Used on fingertips, palm, and in between fingers Used to tell if finger is making contact and where Resistance is lowered as pressure is added Force Sensitive Resistor Simple setup, good pressure sensor Low accuracy in terms of pressure amount

Gyroscope On backside of the hand Used to tell hand position Used in both static and dynamic signs Paired with Accelerometer to save space

Accelerometer Only used in dynamic signs Will record both wrist movement and arm movement Little movements will be filtered out, only concerned with big movements

Digitizing and Transmitting Sensor Data Glove Raspberry Pi 2 Android App Sensor Data Transmission Sensor Inputs Sign Lookup Table Storage Power Supply GUI Sensor filtering/ waveshaping Processed Data Transmission Data Collection Sensors Flex Sensor Sensor Data Processing Incoming Data Processing Palm/Finger contact sensors Flex, Contact sensor output processing Sign lookup Gyroscope Accelerometer, Gyro Signal Analysis String building Accelerometer

Digitizing and Transmitting Sensor Data Filter sensor output with active band pass filters Use comparator or Schmitt Trigger circuit to make output of flex and pressure sensors binary Extract data from accelerometer/gyroscope data register Raspberry pi will receive high or low from flex and pressure sensors and acceleration data bytes from gyroscope/accelerometer Lower delays as much as possible

Powering the Glove Portable USB PSU Flex sensors - 5V, 0.5W-1W Accelerometer/Gyroscope - 1.9-3.6V, ~5mA in normal mode Raspberry Pi - 5V, 1.2A max current on 5V supply Pressure sensors - 5V, 1mA max current Total max current~2.4-2.5A Total max power~11-12W

Powering the Glove cont. Patriot Fuel+ 9000mAh PSU 45Wh on 5V~4 hours of continuous usage 1A and 2.5A ports Smaller than Samsung Note 3 and Nokia lumia

Sensor Data Processing and Transmission Glove Raspberry Pi 2 Android App Sensor Data Transmission Sensor Inputs Sign Lookup Table Storage Power Supply GUI Sensor filtering/ waveshaping Processed Data Transmission Data Collection Sensors Flex Sensor Sensor Data Processing Incoming Data Processing Palm/Finger contact sensors Flex, Contact sensor output processing Sign lookup Gyroscope Accelerometer, Gyro Signal Analysis String building Accelerometer

Sensor Data Processing and Transmission - High level overview Incoming Signal Management Thread Signal data processing Thread Network packet construction and transmission thread Retrieve next sign entry Retrieve next processed entry Sensor Input Record incoming Signal Process sign data Create sign packet new sign button Input Store finished signal Store processed sign Transmit to App Processing Queue Transmission Queue Memory Raspberry Pi 2

Sensor Data Processing and Transmission- Raspberry Pi Input Binary input for each flex and contact sensor Filtered Gyro/Accelerometer sensor signals Button or other sensor indicating switching signs Incoming Signal Management Thread Sensor Input Record incoming Signal new sign button Input Store finished signal

Sensor Data Processing Accelerometer/Gyroscope Separate into 3 signals corresponding to x,y,z values Break smoothed signal apart at direction changes Create movement vectors <x, y z> from signal segments Flex/Contact sensor Create n-bit values, 1 bit per sensor y(t) y-position y’(t) y-velocity up, down, up, down y-accel y’’(t)

Transmission Creates packets from transmission queue Each sign is one packet Packet contains segments of the sign signal Packet Headers Movement involved flag Total segments/vectors Possible other data

Android Application Android App Glove Raspberry Pi 2 Sensor Inputs Sensor Data Transmission Sensor Inputs Sign Lookup Table Storage Power Supply GUI Sensor filtering/ waveshaping Processed Data Transmission Data Collection Sensors Flex Sensor Sensor Data Processing Incoming Data Processing Palm/Finger contact sensors Flex, Contact sensor output processing Sign lookup Gyroscope Accelerometer, Gyro Signal Analysis String building Accelerometer

Android App - High Level Overview Receive UDP packets from Raspberry Pi with one packet per gesture Protocol for gesture verification so no skipped gestures and no corrupt data Lookup gesture object in Hash Table Hash Table returns a word/String that’s used as text Android API will be used for text to speech Android app will also allow the user to access other features such as save new gesture 1 2 3 4 5 6

Android App - Flowchart RasPi sends UDP packets RasPi sends UDP packets Protocol (ACK/NAK) gesture Protocol (ACK/NAK) gesture Hash Table Lookup for Gestures Hash Table Lookup for Gestures Speech Android API Gesture translated to word from Hash Text

Android App - Lookup Algorithm Since the movement vectors will not match perfectly, the lookup algorithm will calculate the difference between saved vectors and the given vector 1 The lookup algorithm will try to match a gesture based on binary value of pressure/flex sensors and then match similar movement vectors 2 Then the most similar gesture based on movement vector with the same pressure/flex sensor values will be picked and the associated String returned 3

Android App - Features Features enabled from Android App: Easy to use and simple GUI Enable new gesture mode that will save a new gesture (next gesture sent from the glove) into the Hash Table Enable new sentence mode that will save a sentence (saved associated gestures together as one sentence) Real-Time translation mode (gesture translated to speech in Real-Time) Ability to correct a misinterpreted translation 1 2 3 4 5

Alternative to Android App https://www.raspberrypi.org/forums/vie wtopic.php?f=38&t=68693 Alternative to Android App Offline Text to speech for android OS usable on Raspberry Pi Would bypass need for smartphone altogether and output sign language through raspberry pi speaker Requires Raspberry Pi to do signal processing, translating, and storing word database Android app would only be used to configure glove

Proposed MDR Deliverables Demonstration of Glove Sensors Flex and Contact sensors return binary values for flexed/unflexed and open/closed Accelerometer and Gyroscope showing data for position of hand and arm movement Demonstration of Pi transmitting to app Using simulated data Pi sends a “sign” to the app App displays the translated “sign” Flex: 0 1 1 Contact: 0 1 1 0 0 Gyro: -12dps 9 dps 15dps Accelerometer: -.15g .33g .27g

Thank You Questions?