University of Pennsylvania Moore School of Electrical Engineering ABSTRACT: The ability to communicate is essential for surviving in today’s world, but for many people this must be accomplished through non-verbal means, such as sign language. The glove system developed will help bridge the barrier between users of sign language and those using spoken language. It performs automatic translation and displaying of a specific subset of American Sign Language (ASL) signs made by the user’s hand. The translated signs are then transmitted to a remote display device to be viewed by the intended audience. The design of the overall system targets deaf/mute people who will be able to use the translating glove for a variety of situations in which communication with people unfamiliar with ASL is desired, such as conversations or in presentations. AUTHORS : Alex Chang ‘05 Thomas Malec ‘05 ADVISOR : Dr Daniel Lee DEMO TIMES : Thursday, April 22 th, AM, 10:30AM, 2PM and 2:30PM GROUP 3 Hand Motion Recognition for Sign Language Translation 1)The software starts in an initial waiting state. 2)Upon the capture button being pushed an interrupt is called to capture the data from the flex and tilt sensors. 3)A least distance algorithm is performed on the flex sensor data to determine the closest possible match of the letter the user was trying to make. 4)a) If only one match is found then that character is added to an array containing all of the letters entered since the previous msg was last sent to the serial port. b) If multiple matches are found then the tilt sensor data is used to find the orientation of the hand and compare it to the orientation of possible characters the user is attempting to make. 5) a) If the last character entered is the termination character then the string is sent out the serial port b) Otherwise the system goes back to the initial waiting state to continue adding characters MCU Subsystem Software Flowchart Piezo-Electric Switch   Tilt Sensor System Resistive Flex Sensors Glove Subsystem Component Summary: Resistive Flex Sensors:  Varies in resistance from 10 k  to 40 k  with bends from 0 to 180 degrees, respectively  One flex sensor laid over each finger of the glove to determine amount of bending of each finger  Analog voltage over sensor taken as output  Output used to determine # of 45º units in the sum of the bend angles,  + , from the two major joints of the finger Tilt Sensors:  Consists of 5 tilt switches  Each tilt switch uses a mechanical ball which allows conduction between its terminals when switch is “on”  Each switch is “on” when oriented upright w.r.t the plane of the ground and “off” when turned upside down (cylinder end pointing towards the floor)  Tilt sensor system allows for unambiguous determination of orientation of the palm of the hand (palm facing left/right, forward/backward, up/down) Piezo-Electric Switch:  Acts as a push button to indicate when to “capture”, decode, and transmit the hand sign currently being formed by the user  Sends a pulse to the MCU’s input capture pin, when pressed, to begin process illustrated below Glove Subsystem Hardware Output Data Lines (5) Bluetooth Tx/Rx Module with Serial I/O (MCU Side) Bluetooth Tx/Rx Module with Serial I/O (PC Side) RS-232 Wireless Tx/Rx Communication/Display Subsystem Laptop with Msg Display GUI Application  After the msg, to be transmitted by the MCU, is received by the Bluetooth Module, it is transmitted wirelessly to the recieving Bluetooth module at the PC end. Then the msg string is sent serially to the PC and displayed on a GUI application for the user’s viewing  Two identical TDK Bluetooth Intelligent Serial Modules are used with a maximum range of 100 meters in free space for Tx/Rx  Msg Display GUI App can be applied to any device using Windows operating system (eg. PDA, PC, Laptop) for flexibility in daily use Plane of Palm Press Here Finger Joints