1. Applications of Image Processing to Sign Language Recognition
Byron Hood
Computer Systems Lab Project

2. Computer Systems Lab Project 2007-08
Overview
Average person typing speed
Composing: 19 wpm
Transcribing: 33 wpm
Sign speaker
Full sign language: 200—220 wpm
Spelling: est. 50 wpm
Byron Hood
Computer Systems Lab Project

3. Computer Systems Lab Project 2007-08
Purpose and Scope
Native signers faster “typing”
Benefits:
Hearing & speaking disabled
Sign interpreters
Just letters & numbers
Byron Hood
Computer Systems Lab Project

4. Computer Systems Lab Project 2007-08
Research
Original
Related projects
Using mechanical gloves
Tracking body parts
Image techniques
Edge detection
Line detection
Byron Hood
Computer Systems Lab Project

5. Program Architecture (1)‏
Webcam interface
Using Video 4 Linux 1
Saves images to disk
Edge detection
Robert’s Cross
Line detection
Hough transform
Byron Hood
Computer Systems Lab Project

6. Program Architecture (2)‏
Line processing
Image data dropped
AI to find “best fit”
Letter matching
Load hand data from XML files
Try to match hand
Byron Hood
Computer Systems Lab Project

7. Computer Systems Lab Project 2007-08
Sample Code
From edge_detect.c :
int row,col;
// loop through the image
for(row=0; row<rows; row++) {
for(col=0; col<cols; col++) {
// avoid the problems from trying to calculate on an edge
if(row==0 || row==rows-1 || col==0 || col==cols-1) {
image2[row][col]=0; // so set value to 0 }
else {
int left = image1[row][col-1]; // some variables
int right = image1[row][col+1]; // to make the final
int top = image1[row-1][col]; // equation seem a
int bottom = image1[row+1][col]; // little bit neater
image2[row][col]=(int)(sqrt(pow(left – right , 2) +
pow(top - bottom, 2)));
Byron Hood
Computer Systems Lab Project

8. Computer Systems Lab Project 2007-08
Testing (1)‏
General testing model
~/syslab-tech $ \
> ./main images/hand.pgm in.pgm
[DEBUG] Edge detect time: 486 ms
[DEBUG] Wrote image file to `in.pgm’
Errors: Warnings: 0
Byron Hood
Computer Systems Lab Project

9. Computer Systems Lab Project 2007-08
Testing (2)‏
Results:
Of edge detection
Similar for cropping or line finding
Automated testing difficult and impractical
Original image
(800 x 703)‏
Final image
(800 x 703)‏
Byron Hood
Computer Systems Lab Project

10. Computer Systems Lab Project 2007-08
The Mysterious Future
Finish line detection
Detection and parameterization
Build AI to interpret lines
Finish camera-computer interaction
OPTIMIZE!!!
Byron Hood
Computer Systems Lab Project