Byron Hood | version 0.4 Computer Systems Lab Project Sign Language Recognition using Webcams

Byron Hood | version 0.4 Computer Systems Lab Project Overview Average person’s typing speed  Composing: ~19 words per minute  Transcribing: ~33 words per minute Sign speaker  Full sign language: ~200 words per minute  Spelling out: estimate: 50 words per minute  Up to 3x faster

Byron Hood | version 0.4 Computer Systems Lab Project Purpose and Scope Native signers can input faster Benefits:  Hearing & speaking disabled  Sign interpreters Just letters & numbers for now  Additional complexity too much to handle  Would require smaller distinctions

Byron Hood | version 0.4 Computer Systems Lab Project Research Related projects  Using mechanical gloves, colored gloves  Tracking body parts  Neural network-based application Still images: 92% accuracy Motion: less than 50% accuracy  Feature vector-based application Also about 90% accuracy on stills No motion tests

Byron Hood | version 0.4 Computer Systems Lab Project More Research Image techniques:  Edge detection (Robert’s Cross)‏  Line detection (Hough transform)‏  Line interpretation methods Chaining groups of lines Macro-scale templates  Residual math Memory management

Byron Hood | version 0.4 Computer Systems Lab Project Testing Model Human interaction necessary General testing model: ~/syslab-tech $ \ >./main images/hand.png [DEBUG] Edge detect time: 29 ms Errors: 0 Warnings: 0

Byron Hood | version 0.4 Computer Systems Lab Project Program Architecture Webcam capture Edge detection Line detection Interpretation Attribute matching IMAGE FEATURE OUTLINE LINE LIST FINGER POSITONS SERVER PROCESS

Byron Hood | version 0.4 Computer Systems Lab Project Edge Detection Results Results:  Outlines the important edges and not much besides  Robert’s Cross balances detection of major and minor lines Original image (800 x 703)‏ Final image (800 x 703)‏

Byron Hood | version 0.4 Computer Systems Lab Project Cropping Results Original (800 x 703)‏ Result (633 x 645)‏ Remove useless rows & columns with no features Better contrast Very large optimization  Memory savings  Area difference means order n 2

Byron Hood | version 0.4 Computer Systems Lab Project Line detection Finished!  Recently finished tweaking sensitivities  Still a few potential memory issues

Byron Hood | version 0.4 Computer Systems Lab Project Line Grouping Part of line detection Large optimization  Iterate over an order of magnitude fewer items  Easier to handle, more pronounced trends Examples of line groups, called “chains”

Byron Hood | version 0.4 Computer Systems Lab Project Line Interpretation Chaining groups of lines Templates  Generation  Template-based comparison Line residuals  Use point coordinate averages  Calculate average offset from average  Easy to find height of finger

Byron Hood | version 0.4 Computer Systems Lab Project Sample Output After a typical run: 10 days, 6:12:19 until graduation!! ~/syslab-tech/src $./main hand.png Edge detection took 0.04 sec Image cropping took 0.00 sec Line detection took 0.17 sec (detected 1424 lines)‏ Line chaining took 0.25 sec (detected 130 chains)‏ Getting orientation took src (1 => ORIENTATION_FORWARD)‏ Getting pinky pos. took 0.00 sec (2 => FINGER_BENT)‏ Getting ring pos. took 0.00 sec (2 => FINGER_BENT)‏ Getting middle pos. took 0.01 sec (2 => FINGER_BENT)‏ Getting index pos. took sec (4 => FINGER_TUCKED)‏ Overall process took 0.47 sec [TOTALCOUNT] allocated: , freed: ; leaked:

Byron Hood | version 0.4 Computer Systems Lab Project Timing Timing data from runs:  To nearest hundredth of a second (1)Edge detection: 0.04 sec (2)Image cropping: 0.00 sec (3)Line detection: 0.17 sec (4)Line chaining: 0.25 sec (5)orientation: 0.08 sec (6)Pinky finger: 0.00 sec (7)Ring finger: 0.00 sec (8)Middle finger: 0.01 sec (9)Index finger: 0.00 sec (10)Overall process: 0.47 sec  A little slow considering goal of real-time

Byron Hood | version 0.4 Computer Systems Lab Project The Mysterious Future Perfect line interpretation Work on memory management  Am leaking large quantities (~50K) of memory  Aggressive profiling needed Finish camera-computer interaction  Device control must be precise, picky

Byron Hood | version 0.4 Computer Systems Lab Project The End! Code will be available to future years Contact me for a copy: