1. 1 Action Classification: An Integration of Randomization and Discrimination in A Dense Feature Representation Computer Science Department, Stanford University {bangpeng,aditya86,feifeili}@cs.stanford.edu Bangpeng Yao, Aditya Khosla, and Li Fei-Fei

2. 2 Action Classification & Intuition Our Method Our Results Conclusion Outline

3. Action Classification & Intuition Our Method Our Results Conclusion Outline 3

4. Action Classification 4 Object classification: Presence of parts and their spatial configurations. [Lazebnik et al, 2006] [Fergus et al, 2003] … PhoningRidingBike Running

5. Action Classification 5 All images contain humans; Object classification: Presence of parts and their spatial configurations. [Lazebnik et al, 2006] [Fergus et al, 2003] …

6. Action Classification 6 All images contain humans; Objects small or absent; Object classification: Presence of parts and their spatial configurations. [Lazebnik et al, 2006] [Fergus et al, 2003] …

7. Action Classification 7 All images contain humans; Objects small or absent; Large pose variation & occlusion; Background clutter; Challenging… Object classification: Presence of parts and their spatial configurations. [Lazebnik et al, 2006] [Fergus et al, 2003] …

8. Our Intuition 8 Focus on image regions that contain the most discriminative information.

9. Our Intuition 9 Focus on image regions that contain the most discriminative information. How to represent the features? Dense feature space Randomization & Discrimination How to explore this feature space?

10. Outline 10 Action Classification & Intuition Our Method Our Results Conclusion

11. 11... Region Height Region Width Dense Feature Space Normalized Image Size of image region Center of image region

12. 12... Region Height Region Width Normalized Image Size of image region Center of image region Dense Feature Space

13. 13... Region Height Region Width Normalized Image Size of image region Center of image region Dense Feature Space

14. 14... Region Height Region Width How can we identify the discriminative regions efficiently and effectively? Normalized Image Size of image region Center of image region Dense Feature Space Image size: N×N Image regions: O(N 6 )

15. 15... Region Height Region Width Normalized Image Size of image region Center of image region Apply randomization to sample a subset of image patches Dense Feature Space Random Forests (RF)

16. 16... Region Height Region Width This classOther classes Random Forests (RF) Normalized Image Size of image region Center of image region Dense Feature Space

17. 17... Region Height Region Width RF with discriminative classifiers Normalized Image Size of image region Center of image region This classOther classes Dense Feature Space

18. 18 Generalization Ability of RF Generalization error of a RF: : correlation between decision trees : strength of the decision trees Discriminative classifiers Better generalization Dense feature space decreases increases

19. 19 …… …… RF with Discriminative Classifiers

20. 20 …… …… Train a binary SVM RF with Discriminative Classifiers 1 2 3 4 5 0 1 1 1 0 BoW or SPM of SIFT-LLC features

21. 21 …… …… Train a binary SVM RF with Discriminative Classifiers 1 2 3 4 5 0 1 1 1 0 Biggest information gain

22. 22 …… …… Train a binary SVM RF with Discriminative Classifiers 1 2 3 4 5 0 1 1 1 0

23. 23 …… …… RF with Discriminative Classifiers We stop growing the tree if: - The maximum depth is reached; - There is only one class at the node; - The entropy of the training data at the node is low

24. Classification With RF 24 …… …… Number of trees Class Label

25. Action Classification & Intuition Our Method Our Results Conclusion Outline 25

26. Results on VOC 2011 Actions 26 Action Others’ Best Our Method Jumping71.666.0 Phoning50.741.0 Playing instrument77.560.0 Reading37.841.5 Riding bike88.890.0 Riding horse90.292.1 Running87.986.6 Taking photo25.728.8 Using computer58.962.0 Walking59.565.9 Our method ranks the first in six out of ten classes.

27. Results on VOC 2011 Actions 27 Action Others’ Best Our Method Jumping71.666.0 Phoning50.741.0 Playing instrument77.560.0 Reading37.841.5 Riding bike88.890.0 Riding horse90.292.1 Running87.986.6 Taking photo25.728.8 Using computer58.962.0 Walking59.565.9

28. Results on VOC 2011 Actions 28 Action Others’ Best Our Method Jumping71.666.0 Phoning50.741.0 Playing instrument77.560.0 Reading37.841.5 Riding bike88.890.0 Riding horse90.292.1 Running87.986.6 Taking photo25.728.8 Using computer58.962.0 Walking59.565.9

29. 29 Generalization Ability of RF Discriminative classifiers Better generalization Dense feature space Tree correlation decreases Tree strength increases dense feature(spatial pyramid) SPM feature Vs. strong classifierweak classifierVs. Train discriminative SVM classifiers Generate feature weights randomly (Results on PASCAL VOC 2010)

30. Action Classification & Intuition Our Method Our Results Conclusion Outline 30

31. Conclusion 31 Exploring dense image features can benefit action classification; Combining randomization and discrimination is an effective way to explore the dense image representation; Achieves very good performance based on only one type of image descriptor; Code will be available soon.

32. 32 …… …… Train a binary SVM 1 2 3 4 5 0 1 1 1 0 Acknowledgement Bangpeng Yao, Aditya Khosla, and Li Fei-Fei. “Combining Randomization and Discrimination for Fine-Grained Image Categorization.” CVPR 2011. Thanks to Su Hao, Olga Russakovsky, and Carsten Rother. Reference: