1. CVPR Workshop on RTV4HCI 7/2/2004, Washington D.C. Gesture Recognition Using 3D Appearance and Motion Features Guangqi Ye, Jason J. Corso, Gregory D. Hager Computational Interaction and Robotics Lab The Johns Hopkins University Baltimore, MD

2. CVPR Workshop on RTV4HCI 7/2/2004, Washington D.C. Analogy Between Gesture and Speech

3. CVPR Workshop on RTV4HCI 7/2/2004, Washington D.C. 4DT Platform  Previous work: J. Corso, D. Burschka, G. Hager, The 4DT: Unencumbered HCI With VICs. CVPRHCI, 2003.  Geometrically and photometrically calibrated  Known background

4. CVPR Workshop on RTV4HCI 7/2/2004, Washington D.C. Video Preprocessing Acquisition Rectification Background Subtraction Color Calibration

5. CVPR Workshop on RTV4HCI 7/2/2004, Washington D.C. System Framework Image Preprocessing Coarse Stereo Matching Appearance/Motion Extraction Feature Clustering Gesture Recognition

6. CVPR Workshop on RTV4HCI 7/2/2004, Washington D.C. Visual Feature Capturing: 3D Volume  Consider limited 3D space around object  Block-based coarse stereo matching

7. CVPR Workshop on RTV4HCI 7/2/2004, Washington D.C. Motion Computation  Motion by differencing of stereo volume

8. CVPR Workshop on RTV4HCI 7/2/2004, Washington D.C. Unsupervised Learning of Feature Set  VQ: K-means approach  Choice of cluster number based on distortion analysis

9. CVPR Workshop on RTV4HCI 7/2/2004, Washington D.C. Temporal Gesture Modeling  6-state discrete forward HMMs  Multilayer Neural Network Aligning all sequences to have equal length 3-layers, 50 hidden nodes

10. CVPR Workshop on RTV4HCI 7/2/2004, Washington D.C. Experiment: Gesture Vocabulary Push Toggle

11. CVPR Workshop on RTV4HCI 7/2/2004, Washington D.C. Gesture Vocabulary Swipe Left Swipe Right

12. CVPR Workshop on RTV4HCI 7/2/2004, Washington D.C. Gesture Vocabulary Twist Clockwise Anti-clockwise

13. CVPR Workshop on RTV4HCI 7/2/2004, Washington D.C. Different Feature Data Sets  Appearance volume  5x5x5=125  10x10x10=1000  Motion volume  Concatenation of appearance and motion e.g.,(125-appearance, 1000-d motion)  Combination of clustering result of appearance and motion  Form a 2-d vector of cluster identity e.g., (3, 2)

14. CVPR Workshop on RTV4HCI 7/2/2004, Washington D.C. Gesture Recognition  Training: >100 sequences for each gesture  Test: >70 sequences for each gesture  Combination achieves best results Feature SetClustersHMMNN Appearance 899.5 100.098.8 Motion 1598.498.197.786.3 Concatenation 1898.999.098.987.7 Combination 8*15=120100.0 96.6

15. CVPR Workshop on RTV4HCI 7/2/2004, Washington D.C. Real-time Implementation Demo

16. CVPR Workshop on RTV4HCI 7/2/2004, Washington D.C. Conclusion  Novel approach to extract 3D appearance and motion cues without tracking  VQ clustering to learn gesteme  Modeling dynamic gestures using HMM, NN  Real-time implementation on 4DT  Extensive experiments achieve high recognition accuracy

17. CVPR Workshop on RTV4HCI 7/2/2004, Washington D.C. Thanks

18. CVPR Workshop on RTV4HCI 7/2/2004, Washington D.C. 3D Appearance Volume  Comprehensive color normalization  Coarse disparity map Consider local images of m x n patches, perform pair-wise image matching between patches  Disparity search range [0, ( p-1 ) * w ]  Dimensionality 3D volume m*n*p

19. CVPR Workshop on RTV4HCI 7/2/2004, Washington D.C. Gesture Recognition  HMM modeling on collapsed sequences Raw: 6 6 6 5 5 5 5 5 4 4 4 4 1 1 Collapsed: 6 5 4 1  Without considering duration Feature SetTrainingTest Appearance 88.386.1 Motion 98.496.6 Concatenation 90.889.0 Combination 99.898.8

20. CVPR Workshop on RTV4HCI 7/2/2004, Washington D.C. 4DT Platform  Gestures in visual HCI: popular choice  Manipulative gesture modeling without tracking Difficulty of reliable tracking of hand Complexity of hand modeling  3D data acquisition Limitation of 2D cues for modeling hand Stereo matching Special sensors

21. CVPR Workshop on RTV4HCI 7/2/2004, Washington D.C. Properties of 4DT  Known background & object properties

22. CVPR Workshop on RTV4HCI 7/2/2004, Washington D.C.