1. Byron Hood | version 0.3 Computer Systems Lab Project 2007-2008 Sign Language Recognition

2. Byron Hood | version 0.3 Computer Systems Lab Project 2007-2008 Overview Average person typing speed  Composing: 19 words per minute  Transcribing: 33 words per minute Sign speaker  Full sign language: 200—220 wpm  Spelling out: est. 50-55 wpm  Faster by 1.5x to 3x

3. Byron Hood | version 0.3 Computer Systems Lab Project 2007-2008 Purpose and Scope Native signers can input faster Benefits:  Hearing & speaking disabled  Sign interpreters Just letters & numbers for now

4. Byron Hood | version 0.3 Computer Systems Lab Project 2007-2008 Research Related projects  Using mechanical gloves, colored gloves  Tracking body parts  Neural network-based application Still images: 92% accuracy Motion: less than 50% accuracy  Feature vector-based application Also about 90% accuracy on stills No motion tests

5. Byron Hood | version 0.3 Computer Systems Lab Project 2007-2008 More Research Image techniques:  Edge detection (Robert’s Cross)‏  Line detection (Hough transform)‏  Line interpretation methods Chaining groups of lines Macro-scale templates

6. Byron Hood | version 0.3 Computer Systems Lab Project 2007-2008 Program Architecture Webcam interface Edge detection Line detection Line interpretation Hand attribute matching

7. Byron Hood | version 0.3 Computer Systems Lab Project 2007-2008 Testing Model Human interaction necessary General testing model: bhood@bulusan: ~/syslab-tech $ \ >./main images/hand.pgm in.pgm [DEBUG] Edge detect time: 486 ms [DEBUG] Wrote image file to `in.pgm’ Errors: 0 Warnings: 0

8. Byron Hood | version 0.3 Computer Systems Lab Project 2007-2008 Code sample Robert’s Cross edge detection: int row,col; // loop through the image for(row=0; row<rows; row++) { for(col=0; col<cols; col++) { // avoid the problems from trying to calculate on an edge if(row==0 || row==rows-1 || col==0 || col==cols-1) { image2[row][col]=0; // so set value to 0 } else { int left = image1[row][col-1]; // some variables int right = image1[row][col+1]; // to make the final int top = image1[row-1][col]; // equation seem a int bottom = image1[row+1][col]; // little bit neater image2[row][col]=(int)(sqrt(pow(left – right, 2) + pow(top - bottom, 2))); }

9. Byron Hood | version 0.3 Computer Systems Lab Project 2007-2008 Edge Detection Results Results:  Shows the important edges but little else  Robert’s Cross method the optimal balance Original image (800 x 703)‏ Final image (800 x 703)‏

10. Byron Hood | version 0.3 Computer Systems Lab Project 2007-2008 Cropping Results Remove useless rows with no features Very large optimization  Area difference  Input/output Original (800 x 703)‏ Result (633 x 645)‏

11. Byron Hood | version 0.3 Computer Systems Lab Project 2007-2008 Line detection Mostly complete  Still some bugs to iron out  Shows nonexistent lines occasionally (see image!)‏

12. Byron Hood | version 0.3 Computer Systems Lab Project 2007-2008 Line Interpretation Chaining groups of lines Templates  Generation  Template-based comparison Hand matching

13. Byron Hood | version 0.3 Computer Systems Lab Project 2007-2008 The Mysterious Future Finish line interpretation  Building method for interpreting lines  Working on template-based recognition  Write XML template files Finish camera-computer interaction  Device control must be precise, picky