1. Vogler and Metaxas University of Toronto Computer Science CSC 2528: Handshapes and Movements: Multiple- channel ASL recognition Christian Vogler and Dimitris Metaxas (presented by Christopher Collins)

2. 2 University of Toronto Computer Science Vogler and Metaxas Overview: Part II Introduction to ASL recognition Challenges of ASL recognition Related work Modelling  Phoneme-based modelling  Independent Channels  Handshape Parallel Hidden Markov Models Experiments Conclusions and Future Work

3. 3 University of Toronto Computer Science Vogler and Metaxas ASL Recognition: Introduction Computer interaction is still mainly keyboard/mouse  requires literacy in a written language or an agreed-upon standard written form of ASL (e.g. sign-writing)  difficult for many people who are deaf

4. 4 University of Toronto Computer Science Vogler and Metaxas ASL Recognition: Challenges More difficult than speech recognition due to:  simultaneous events

5. 5 University of Toronto Computer Science Vogler and Metaxas ASL Recognition: Challenges More difficult than speech recognition due to:  simultaneous events  inflections

6. 6 University of Toronto Computer Science Vogler and Metaxas ASL Recognition: Challenges More difficult than speech recognition due to:  simultaneous events  inflections  phonology poorly understood, no agreed standard

7. 7 University of Toronto Computer Science Vogler and Metaxas Challenges of Simultaneity

8. 8 University of Toronto Computer Science Vogler and Metaxas Related Work C. Vogler and D. Metaxas. Parallel Hidden Markov Models for ASL Recognition (1999). G. Fang et al. Signer-independent continuous sign language recognition based on SRN/HMM (2001). R.-H. Liang and M. Ouhyoung. A real-time continuous gesture recognition system for sign language (1998).

9. 9 University of Toronto Computer Science Vogler and Metaxas Overview HMM-based approach to ASL recognition  parallel HMMs for different channels  channels are left and right handshape and movement  uses the movement-hold phonology

10. 10 University of Toronto Computer Science Vogler and Metaxas Movement-Hold Example

11. 11 University of Toronto Computer Science Vogler and Metaxas Handshape Modelling Most previous work uses joint and abduction angles as features (low- level) Also experiment with a measure of the openness of a finger (high level)  height and width of quadrilateral  MPJ angle  abduction angles

12. 12 University of Toronto Computer Science Vogler and Metaxas Extensions to HMM Regular HMM model one process evolving over time To model parallel, possibly interacting processes with a regular HMM, events must evolve in lockstep Earlier work by Vogler and Metaxas explains development of parallel HMM model

13. 13 University of Toronto Computer Science Vogler and Metaxas Factorial HMM

14. 14 University of Toronto Computer Science Vogler and Metaxas Coupled HMM

15. 15 University of Toronto Computer Science Vogler and Metaxas Parallel HMM

16. 16 University of Toronto Computer Science Vogler and Metaxas Combination of Processes Using independence assumption, combine path probabilities (from each channel, with states representing the same sign sequence) by multiplying them. Choose the most probable state sequence. Time is polynomial in number of states, linear in number of parallel processes More info: C. Vogler and D. Metaxas, Parallel Hidden Markov Models for ASL Recognition; Proc. Int. Conf. on Comp. Vis., Greece, 1999.

17. 17 University of Toronto Computer Science Vogler and Metaxas Experiments Compare handshape models (joint angles vs. quadrilateral) for handshape recognition task Compare PaHMM model with various channel combinations against single hand movement channel (naïve baseline?) Vocabulary of 22 signs, 400 training sentences of length 2-7 signs, and 99 test sentences Omitted left-hand handshape?

18. 18 University of Toronto Computer Science Vogler and Metaxas Choice of Handshape Model Measure correctly recognized handshape (recognizing signs with handshape alone not possible) Quadrilateral feature vector results in better (and more consistent) recognition accuracy

19. 19 University of Toronto Computer Science Vogler and Metaxas Experimental Results H=correct, D = deletion, S = substitution, I = insertion, N = number

20. 20 University of Toronto Computer Science Vogler and Metaxas Conclusions Handshape information is important in ASL recognition Parallel HMM a promising model for multi- channel data

21. 21 University of Toronto Computer Science Vogler and Metaxas Future Work Training/Test data from native signers Include facial expressions Use of relative spatial information (classifiers) Larger vocabulary Incorporation of language modelling to improve recognition, such as n-gram or parsing