Download presentation
Presentation is loading. Please wait.
Published byMarley Morden Modified over 10 years ago
1
Vogler and Metaxas University of Toronto Computer Science CSC 2528: Handshapes and Movements: Multiple- channel ASL recognition Christian Vogler and Dimitris Metaxas (presented by Christopher Collins)
2
2 University of Toronto Computer Science Vogler and Metaxas Overview: Part II Introduction to ASL recognition Challenges of ASL recognition Related work Modelling Phoneme-based modelling Independent Channels Handshape Parallel Hidden Markov Models Experiments Conclusions and Future Work
3
3 University of Toronto Computer Science Vogler and Metaxas ASL Recognition: Introduction Computer interaction is still mainly keyboard/mouse requires literacy in a written language or an agreed-upon standard written form of ASL (e.g. sign-writing) difficult for many people who are deaf
4
4 University of Toronto Computer Science Vogler and Metaxas ASL Recognition: Challenges More difficult than speech recognition due to: simultaneous events
5
5 University of Toronto Computer Science Vogler and Metaxas ASL Recognition: Challenges More difficult than speech recognition due to: simultaneous events inflections
6
6 University of Toronto Computer Science Vogler and Metaxas ASL Recognition: Challenges More difficult than speech recognition due to: simultaneous events inflections phonology poorly understood, no agreed standard
7
7 University of Toronto Computer Science Vogler and Metaxas Challenges of Simultaneity
8
8 University of Toronto Computer Science Vogler and Metaxas Related Work C. Vogler and D. Metaxas. Parallel Hidden Markov Models for ASL Recognition (1999). G. Fang et al. Signer-independent continuous sign language recognition based on SRN/HMM (2001). R.-H. Liang and M. Ouhyoung. A real-time continuous gesture recognition system for sign language (1998).
9
9 University of Toronto Computer Science Vogler and Metaxas Overview HMM-based approach to ASL recognition parallel HMMs for different channels channels are left and right handshape and movement uses the movement-hold phonology
10
10 University of Toronto Computer Science Vogler and Metaxas Movement-Hold Example
11
11 University of Toronto Computer Science Vogler and Metaxas Handshape Modelling Most previous work uses joint and abduction angles as features (low- level) Also experiment with a measure of the openness of a finger (high level) height and width of quadrilateral MPJ angle abduction angles
12
12 University of Toronto Computer Science Vogler and Metaxas Extensions to HMM Regular HMM model one process evolving over time To model parallel, possibly interacting processes with a regular HMM, events must evolve in lockstep Earlier work by Vogler and Metaxas explains development of parallel HMM model
13
13 University of Toronto Computer Science Vogler and Metaxas Factorial HMM
14
14 University of Toronto Computer Science Vogler and Metaxas Coupled HMM
15
15 University of Toronto Computer Science Vogler and Metaxas Parallel HMM
16
16 University of Toronto Computer Science Vogler and Metaxas Combination of Processes Using independence assumption, combine path probabilities (from each channel, with states representing the same sign sequence) by multiplying them. Choose the most probable state sequence. Time is polynomial in number of states, linear in number of parallel processes More info: C. Vogler and D. Metaxas, Parallel Hidden Markov Models for ASL Recognition; Proc. Int. Conf. on Comp. Vis., Greece, 1999.
17
17 University of Toronto Computer Science Vogler and Metaxas Experiments Compare handshape models (joint angles vs. quadrilateral) for handshape recognition task Compare PaHMM model with various channel combinations against single hand movement channel (naïve baseline?) Vocabulary of 22 signs, 400 training sentences of length 2-7 signs, and 99 test sentences Omitted left-hand handshape?
18
18 University of Toronto Computer Science Vogler and Metaxas Choice of Handshape Model Measure correctly recognized handshape (recognizing signs with handshape alone not possible) Quadrilateral feature vector results in better (and more consistent) recognition accuracy
19
19 University of Toronto Computer Science Vogler and Metaxas Experimental Results H=correct, D = deletion, S = substitution, I = insertion, N = number
20
20 University of Toronto Computer Science Vogler and Metaxas Conclusions Handshape information is important in ASL recognition Parallel HMM a promising model for multi- channel data
21
21 University of Toronto Computer Science Vogler and Metaxas Future Work Training/Test data from native signers Include facial expressions Use of relative spatial information (classifiers) Larger vocabulary Incorporation of language modelling to improve recognition, such as n-gram or parsing
Similar presentations
© 2025 SlidePlayer.com. Inc.
All rights reserved.