Craig Newman Curtin University of Technology Translating Australian Sign Language into Speech

Introduction To Auslan Elements Of Auslan Signs Hardware Used Hardware Control Algorithm Elements Of Sign Recognition Algorithm Future Intentions Presentation Overview

British and Irish origins More than 4000 commonly used signs 0.1% of Australians use Auslan Introduction to Auslan

Two-handed fingerspelling alphabet

Handshapes Location Orientation Movement Expression Elements of Auslan signs

Handshapes

First Attribute: Degree to which fingers are bent

Handshapes Second Attribute: Combination of fingers touching

Handshapes Third Attribute: Pitch of the hand

Neutral Space Primary Locations Secondary Locations Location

Orientation

Roll required to differentiate Yaw required to differentiate

Large Scale Straight Line Series of Straight Lines Arcs Circles Small Scale Changes in Orientation Changes in Fingers New Handshape Movement

Head Eyebrows Eyes Mouth Cheeks Expression

P5 glove from Essential Reality

Known through the P5 Degree of Finger Bend Position Primary Locations, Movement Orientation Required Contacting Secondary Locations What was Needed?

Acquiring Secondary Locations

M16C/62 Single-chip Microcontroller

The Prototype

Collect Data Every 17ms 60 Hertz Synchronisation with P5 Test Pins Not Logged Write Pin High, Read Remaining Pins Log Connected Pins Debounce: Test For Consistency Encapsulate for Transmission Transmit When Required Contact Sensor Algorithm

Working Example 1

Working Example 2 INSERT CLIP: IM

3 Samples Stored in stop_buffer Position Orientation Finger Bend Contacts Check Last 2 Samples within Range When was the hand considered Stopped?

Wait Until Hands Stop Take First Sample From stop_buffer Filter Through if/else Structure to find Correct Combination Return Handshape Identifying Handshapes

Working Example 1 INSERT CLIP: GUN

Working Example 2 INSERT CLIP: MOTHER

Record Position of Head Shoulder Bicep Forearm Extrapolate Primary Locations Else is Neutral Space Problem Requires Dynamic Position Tracking Identifying Primary Locations

Wait Until “not stopped” While “not stopped” copy stop_buffer Allow for Slight Pauses Filter Data By Testing For Inherent Geometric Structure Output Movement Type Identifying Movement Type

Identify Initial Handshape, Location, Orientation Filter Through if/else Structure If Sign Recognised Output Sign Else Identify Movement Identify Final Handshape, Location, Orientation Filter through if/else Structure Output Sign Putting it all together: Recognizing Signs

Working Example INSERT CLIP: WRAPUP2

Design Composite Glove Dynamic Tracking Of Primary Locations Introduce Learning Algorithms Identifying Handshapes Identifying Movement Translate Signing Context Into English Text Integrate Text To Voice Synthesizer Future Intentions

Questions & Answers