1. One-Shot Learning Gesture Recognition Students:Itay Hubara Amit Nishry Supervisor:Maayan Harel Gal-On

2. Introduction Reduced Problem Complete Problem ConclusionOutline 2

3. Background Gesture recognition is a strong upcoming field in computer vision Gesture recognition can be seen as a way for computers to begin to understand human body language 3

4. Motivation Existing Gesture recognition demand a long configuration and training Different Gestures are been solved using different approaches 4

5. Goals Learn and understand existing Gesture recognition algorithms. Compare different approaches Design Gesture recognition algorithm which reduces training time 5

6. Data The Data is compose from several set each contains: o Gesture vocabulary (learning set) which contain only one sample per gesture. o Test set which contain one or more gestures. Each of the sets has different vocabulary features such as large/small gesture hand/legs movement etc. 6

7. Data – Train 7 Base gesture

8. Data – Test 8 Multiple base gestures Large movements

9. Data - Test 9 Multiple base gestures Small movements

10. Challenges One shot learning - only one learning sample (unlike the common approach of multi class classification) Tests videos segmentation Same gesture can have different number of frames Each set has different features (small/big gestures) 10

11. Introduction Reduced Problem Complete Problem ConclusionOutline 11

12. Reduced Problem Assume that each of the test movies has only one gesture Goal: finding features space and distance function which have good separation of the features space 12

13. Problem Approach Classic machine learning problem Select Feature One Shot – Match using similarity function 13

14. Features Motion Energy – subtracting consecutive frames – Space Quantization 14

15. Features Harris Corner Detector – Find interest point in the difference image based on corner detection Space Time Interest Points – Extend Harris to the time domain 15

16. Features Harris Corner Detector – Find interest point in the image based on corner detection 16

17. Features Space Time Interest Points 17

18. Features 18 STIPHarris

19. Features Head Relative Interest Points 19

20. Features 20 Interest pointsHead Histogram

21. Distance Functions Good features space is defined not only by the features but also by the distance (similarity) function Different features need different distance functions 21

22. Principal Motion Using PCA Using principal component analysis (PCA), to find the main motion vectors. For test set - project feature onto each of train principals and evaluate similarity 22

23. Earth Moving Distance Given two sets of distribution, EMD will measure the minimum cost to shift “dirt” from one distribution to the other. 23

24. Perturbed Variations 24 Given two sets of distribution and predefined value of permitted variations optimally perturbs the distribution to best fit each other. Transportation problem under permitted variations constrain

25. Perturbed Variations 25

26. Levenshtein Distance Measure the difference between two sequences. Consider lengths and classification. 26

27. Results 27

28. Results 28 Top 10 Top 20

29. Results 29

30. Introduction Reduced Problem Complete Problem ConclusionOutline 30

31. Complete Problem Separate problems Basic Segmentation (equal/movement) Whole problem solving approach Moving Window Dynamic Time Warping (DTW) 31

32. Problem Approach Three different method to solve the problem: – Basic Segmentation (equal/movement) – Moving Window – Dynamic Time Warping (DTW) 32

33. Moving Window Move a window along the test video. Assume each window frames has only one gesture Preform basic analysis as did before to and build the distance matrix 33

34. Moving Window 34

35. Moving Window After Sorting the Distance matrix we extract labels and cuts 35 Several other operation (such as smoothing) are done before extracting the final result

36. Dynamic Time Warping Create a state machine from train data: – Module standing position – Form standing position can move to start of base gestures – Assume we can move forward, or stay in the same sate. For a given gesture – find the best path along the sate machine 36

37. Dynamic Time Warping 37

38. Results 38

39. Results 39

40. Results 40

41. Results 41 Top 10 Top 20

42. Introduction Reduced Problem Complete Problem ConclusionOutline 42

43. Conclusions Each approach receive better results in different feature and similarity function Different algorithms has different strengths (segmentation\recognition) Segmentation require standing position model. 43

44. Conclusions 44 Pre-processing unsupervised algorithms help better representing the data. There is still allot left to do on the field

45. Future Work Try different models for the standing position to improve segmentation results Try combing DTW for segmentation and PCA for recognition. Use different unsupervised algorithms to better represent the data. 45

46. References Ivan Laptev, "On Space-Time Interest Points”, 2005 Hugo Jair Escalantea and Isabelle Guyonb, "Principal motion: PCA-based reconstruction of motion histograms” M.Harel, S.Manor, "The Perturbed Variation”, NIPS 2012 Elizaveta Levina, Peter Bickel Department of Statistics, “The EarthMover’s Distance is the Mallows Distance: Some Insights from Statistics”. Ofir Pele,Michael Werman, “Fast and Robust Earth Mover’s Distances”.2008 46