MULTI-LINGUAL AND DEVICELESS COMPUTER ACCESS FOR DISABLED USERS C.Premnath and J.Ravikumar S.S.N. College of Engineering TamilNadu

Abstract Hand forms one of the most effective interaction tool for HCI. Currently, the only technology that satisfies the advanced requirements is glove-based sensing. We present a new tool for gesture analysis, by a simple gesture-symbol/character mapping esp. suited to the disabled users.

AIM : To use the hand gestures of the user to help them control/access the computer easily OBJECTIVE : To provide simple and cheap system of communication to people with single or multiple disabilities To overcome the language problems in communication/computer access.

INTRODUCTION Difficulties and impairments reduce computer use. Direct use of the hand as an input device is an attractive method for providing natural human-computer interaction (HCI).

Currently, the only technology that satisfies the advanced requirements of hand-based input for HCI is glove based sensing. Hinders the ease and naturalness with which the user can interact with the computer controlled environment Requires long calibration and setup procedures.

Glove-Based Approach Basic operation is to sense the gesture by electric/magnetic contact or by monitoring threshold values in chemical/electrode based sensors. The sensors are connected to a control unit to find the gesture. Cost, dexterity and flexibility.

Hinders the ease and naturalness with which the user can interact with the computer controlled environment Requires long calibration and setup procedures.

Computer vision has the potential to provide much more natural, non- contact solutions.

Gesture recognition methods Model-based approach Appearance-based approach

Model-Based Approach Model based approaches estimate the position of a hand by projecting a 3-D hand model to image space and comparing it with image features. The steps involved are: Extracting a set of features from the input images Projecting the model on the scene (or back- projecting the image features in 3D) Establishing a correspondence between groups of model and image features

Appearance-Based Approach Appearance based approaches estimate hand postures directly from the images after learning the mapping from image feature space to hand configuration space. The feature vectors obtained is compared against user templates to determine the user whose hand photograph was taken.

Require considerable research on mapping and other relevant work. Actually allow us to create simple and cost effective systems.

Systems providing computer-access for people with disabilities JavaSpeak Parse the program and "speak" the program’s structure to a blind user ViaVoice, which has a published API, is used as the speech reader.

Emacspeak Provides functions geared directly to programming. Only for someone familiar with a UNIX environment.

METHODOLOGY Novel approach of mapping the character set of the language with the possible set of hand gestures and executes the actions mapped for the particular gesture. Capture the user’s gesture Manipulate and create a 5-digit code Execute required system operation User-friendliness – providing audio request.

The phases involved IMAGE CAPTURING PRE-PROCESSING EDGE DETECTION EDGE TRACKING CODE GENERATION ACTION EXECUTION

Image Capturing Setup and capture

Pre-processing Synthetic image An arithmetic operation is performed with the different channels

a) Sample input image b) Synthetic image

Edge Detection Need for edge detection Edges Edge detection methods Fourier domain Spatial domain

GradientMagnitude operation Spatial Domain Method Performs convolution operations on the source image using kernels.

Sample Output a) Synthetic image b) Edge detection output

Edge Tracking Find critical points.

Lets us see in detail how we trace fingertip shown below. In-depth finger tip image

Tracing for finger valley shown below is done in the exact reverse manner as discussed for finger tip In-depth finger valley image

Output after edge tracking a) Critical points marked with red dot. b) Finger length using Pythagoras Theorem.

CODE GENERATION Using phalanx information

Information about the phalanxes of the right hand

Values to be assigned 1 if the finger is open. 0 if the finger is half-closed i.e., only the proximal phalanx is visible. Already have data about the full finger length information of the user During code generation, 1 assigned when approximate matches with the stored value 0 when the obtained finger length is half that of the corresponding one in the database.

5 fingers - 2 values each Overall 32 (2×2×2×2×2) action gestures.

Mulitilingualism Map the gestures currently associated with only English characters, to the characters in other languages by analyzing the phonetics and their translation to English.

For example, the words involving Tamil characters, Hindi characters, Telugu characters, Malayalam characters, Can all be mapped with the English letter ‘A’,

European languages the alphabet is almost similar. Voice engine support important Latin Non-Latin languages where we have no space between words (Hindi and Arabic), are supported by tailoring the Run-time speech engine Free TTS

ACTION EXECUTION The tree panel Acquires the path information of a file/folder whenever that particular file/folder gets selected by the user’s input. The filename is passed to the speech synthesizer unit and verification done by the user.

JMF player controls the browsing work For example, if character ‘A’ is passed to the file manager then it passes the next file/folder name starting with the letter ‘A’ to the JMF player. File operations Type of the file selected (media/text) and the user’s input gesture. Pass the file to the JMF player unit Execute appropriate operations

Features Minimized cost and user friendliness of the project. Flexibility to change the gesture mapping based upon user’s comfort Ambidexterity

Limitations Gesture mapping for languages with large character sets like Chinese and Japanese. Voice support from the speech engine

Conclusion Novel approach for providing computer access to disabled user with a multilingual method. Overcomes the problem of user’s age involved and physical measures. Support for the illiterate users.

FUTURE WORK Both the hands as input aided with touch pad technology for the computer access (2 10 values - taking 2 values for each of the ten fingers) Assume the ten bit code is associated with the word “Pause”, then the system would type the word “Pause” if the environment is a text editor and PAUSE the current music track if the environment is a music player.

Map the gestures with system commands. Other applications currently inaccessible for disabled users.

The project has proposed an ambidextrous system where the computer access is all within your 5 fingers and the proposed enhancement has the potential to bring the world in your hands.