CVPR Workshop on RTV4HCI 7/2/2004, Washington D.C. Gesture Recognition Using 3D Appearance and Motion Features Guangqi Ye, Jason J. Corso, Gregory D. Hager Computational Interaction and Robotics Lab The Johns Hopkins University Baltimore, MD

CVPR Workshop on RTV4HCI 7/2/2004, Washington D.C. Analogy Between Gesture and Speech

CVPR Workshop on RTV4HCI 7/2/2004, Washington D.C. 4DT Platform  Previous work: J. Corso, D. Burschka, G. Hager, The 4DT: Unencumbered HCI With VICs. CVPRHCI,  Geometrically and photometrically calibrated  Known background

CVPR Workshop on RTV4HCI 7/2/2004, Washington D.C. Video Preprocessing Acquisition Rectification Background Subtraction Color Calibration

CVPR Workshop on RTV4HCI 7/2/2004, Washington D.C. System Framework Image Preprocessing Coarse Stereo Matching Appearance/Motion Extraction Feature Clustering Gesture Recognition

CVPR Workshop on RTV4HCI 7/2/2004, Washington D.C. Visual Feature Capturing: 3D Volume  Consider limited 3D space around object  Block-based coarse stereo matching

CVPR Workshop on RTV4HCI 7/2/2004, Washington D.C. Motion Computation  Motion by differencing of stereo volume

CVPR Workshop on RTV4HCI 7/2/2004, Washington D.C. Unsupervised Learning of Feature Set  VQ: K-means approach  Choice of cluster number based on distortion analysis

CVPR Workshop on RTV4HCI 7/2/2004, Washington D.C. Temporal Gesture Modeling  6-state discrete forward HMMs  Multilayer Neural Network Aligning all sequences to have equal length 3-layers, 50 hidden nodes

CVPR Workshop on RTV4HCI 7/2/2004, Washington D.C. Experiment: Gesture Vocabulary Push Toggle

CVPR Workshop on RTV4HCI 7/2/2004, Washington D.C. Gesture Vocabulary Swipe Left Swipe Right

CVPR Workshop on RTV4HCI 7/2/2004, Washington D.C. Gesture Vocabulary Twist Clockwise Anti-clockwise

CVPR Workshop on RTV4HCI 7/2/2004, Washington D.C. Different Feature Data Sets  Appearance volume  5x5x5=125  10x10x10=1000  Motion volume  Concatenation of appearance and motion e.g.,(125-appearance, 1000-d motion)  Combination of clustering result of appearance and motion  Form a 2-d vector of cluster identity e.g., (3, 2)

CVPR Workshop on RTV4HCI 7/2/2004, Washington D.C. Gesture Recognition  Training: >100 sequences for each gesture  Test: >70 sequences for each gesture  Combination achieves best results Feature SetClustersHMMNN Appearance Motion Concatenation Combination 8*15=

CVPR Workshop on RTV4HCI 7/2/2004, Washington D.C. Real-time Implementation Demo

CVPR Workshop on RTV4HCI 7/2/2004, Washington D.C. Conclusion  Novel approach to extract 3D appearance and motion cues without tracking  VQ clustering to learn gesteme  Modeling dynamic gestures using HMM, NN  Real-time implementation on 4DT  Extensive experiments achieve high recognition accuracy

CVPR Workshop on RTV4HCI 7/2/2004, Washington D.C. Thanks

CVPR Workshop on RTV4HCI 7/2/2004, Washington D.C. 3D Appearance Volume  Comprehensive color normalization  Coarse disparity map Consider local images of m x n patches, perform pair-wise image matching between patches  Disparity search range [0, ( p-1 ) * w ]  Dimensionality 3D volume m*n*p

CVPR Workshop on RTV4HCI 7/2/2004, Washington D.C. Gesture Recognition  HMM modeling on collapsed sequences Raw: Collapsed:  Without considering duration Feature SetTrainingTest Appearance Motion Concatenation Combination

CVPR Workshop on RTV4HCI 7/2/2004, Washington D.C. 4DT Platform  Gestures in visual HCI: popular choice  Manipulative gesture modeling without tracking Difficulty of reliable tracking of hand Complexity of hand modeling  3D data acquisition Limitation of 2D cues for modeling hand Stereo matching Special sensors

CVPR Workshop on RTV4HCI 7/2/2004, Washington D.C. Properties of 4DT  Known background & object properties

CVPR Workshop on RTV4HCI 7/2/2004, Washington D.C.