1. BoVDW: Bag-of-Visual-and-Depth- Words for Gesture Recognition All rights reserved HuBPA© Human Pose Recovery and Behavior Analysis Antonio Hernández-Vela 1,2 Miguel Ángel Bautista 1,2 Xavier Perez-Sala 2,3 Victor Ponce Lopez 1,2 Xavier Baro 2,4 Oriol Pujol 1,2 Cecilio Angulo 3 Sergio Escalera 1,2 1 Dept. Applied Mathematics and Analysis, Universitat de Barcelona 2 Computer Vision Center 3 CETpD, Universitat Politècnica de Catalunya 4 EIMT, Universitat Oberta de Catalunya

2. 1.Introduction 2.Methodology 3.Results 4.Conclusion BoVDW for Gesture Recognition All rights reserved HuBPA© Human Pose Recovery and Behavior Analysis Outline

3. BoVDW for Gesture Recognition All rights reserved HuBPA© Human Pose Recovery and Behavior Analysis Bag of (Visual) Words IntroductionMethodologyResultsConclusion

4. BoVDW for Gesture Recognition All rights reserved HuBPA© Human Pose Recovery and Behavior Analysis Bag of Visual and Depth Words IntroductionMethodologyResultsConclusion In this work, we propose: Bag of Visual and Depth Words (BoVDW). A new depth descriptor. Comparison with state-of-the-art descriptors. Gesture recognition framework.

5. BoVDW for Gesture Recognition All rights reserved HuBPA© Human Pose Recovery and Behavior Analysis Standard pipeline IntroductionMethodologyResultsConclusion Point detection Point description Vocabulary & Representation Classification

6. BoVDW for Gesture Recognition All rights reserved HuBPA© Human Pose Recovery and Behavior Analysis IntroductionMethodologyResultsConclusion Point detection Point description Vocab. & Represent. Classification Spatio-Temporal Interest Points (STIPs) [1] [1] I. Laptev, "On Space-Time Interest Points", (2005) in International Journal of Computer Vision, vol 64, number 2/3, pp.107-123. Temporal extension of the Harris operator

7. BoVDW for Gesture Recognition All rights reserved HuBPA© Human Pose Recovery and Behavior Analysis IntroductionMethodologyResultsConclusion Point detection Point description Vocab. & Represent. Classification Viewpoint Feature Histogram (VFH)[2] [2] Rusu, R.B et al., "Fast 3D recognition and pose using the Viewpoint Feature Histogram", IROS, 2010 Figs. credit to [2] and Aitor Aldoma Histogram of angles between surface normals and viewpoint direction Camera Roll Histogram (CRH) Invariant to rotations in the roll axis of the camera!

8. BoVDW for Gesture Recognition All rights reserved HuBPA© Human Pose Recovery and Behavior Analysis IntroductionMethodologyResultsConclusion Point detection Point description Vocab. & Represent. Classification Concatenation of VFH and CRH

9. BoVDW for Gesture Recognition All rights reserved HuBPA© Human Pose Recovery and Behavior Analysis IntroductionMethodologyResultsConclusion Point detection Point description Vocab. & Represent. Classification Vocabulary building  K-means clustering Spatio-temporal pyramids … Final histogram: Concatenation of 8+1 histograms

10. BoVDW for Gesture Recognition All rights reserved HuBPA© Human Pose Recovery and Behavior Analysis IntroductionMethodologyResultsConclusion Point detection Point description Vocabulary Classification K-nearest neighbor classifier Distance function: Histogram intersection:

11. BoVDW for Gesture Recognition All rights reserved HuBPA© Human Pose Recovery and Behavior Analysis Chalearn dataset IntroductionMethodologyResultsConclusion RGB-D video sequences. Organised in 20 batches: 47 sequences of 1-5 gestures each. Gestures from certain lexicon. Same actor. One-shot learning problem: Just 1 training sample for gesture.

12. BoVDW for Gesture Recognition All rights reserved HuBPA© Human Pose Recovery and Behavior Analysis Results IntroductionMethodologyResultsConclusion Evaluation measurement: Levenshtein distance Depth RGB Mean Levenshtein distance (α= 1) (α= 0)

13. BoVDW for Gesture Recognition All rights reserved HuBPA© Human Pose Recovery and Behavior Analysis Results (Late fusion) IntroductionMethodologyResultsConclusion Late fusion approachMean Lev. Dist. HOGHOF/VFHCRH0.2714 HOG/HOF/VFHCRH0.2662 Batch number Mean Levenshtein dist. (α= 0.8)

14. BoVDW for Gesture Recognition All rights reserved HuBPA© Human Pose Recovery and Behavior Analysis Conclusion IntroductionMethodologyResultsConclusion We have presented: BoVDW approach for gesture recognition. VFHCRH, a new depth descriptor. Comparison of state-of-the-art descriptors. Analysis of Late fusion of RGB and Depth information. Future work: Test other methodologies for spatial coherence. Improve continuous gesture detection.

15. BoVDW: Bag-of-Visual-and-Depth-Words for Gesture Recognition All rights reserved HuBPA© Human Pose Recovery and Behavior Analysis Antonio Hernández-Vela 1,2 Miguel Ángel Bautista 1,2 Xavier Perez-Sala 2,3 Victor Ponce Lopez 1,2 Xavier Baro 2,4 Oriol Pujol 1,2 Cecilio Angulo 3 Sergio Escalera 1,2 1 Dept. Applied Mathematics and Analysis, Universitat de Barcelona 2 Computer Vision Center 3 CETpD, Universitat Politècnica de Catalunya 4 EIMT, Universitat Oberta de Catalunya Thank you! Questions?