1 Action Classification: An Integration of Randomization and Discrimination in A Dense Feature Representation Computer Science Department, Stanford University Bangpeng Yao, Aditya Khosla, and Li Fei-Fei

2 Action Classification & Intuition Our Method Our Results Conclusion Outline

Action Classification & Intuition Our Method Our Results Conclusion Outline 3

Action Classification 4 Object classification: Presence of parts and their spatial configurations. [Lazebnik et al, 2006] [Fergus et al, 2003] … PhoningRidingBike Running

Action Classification 5 All images contain humans; Object classification: Presence of parts and their spatial configurations. [Lazebnik et al, 2006] [Fergus et al, 2003] …

Action Classification 6 All images contain humans; Objects small or absent; Object classification: Presence of parts and their spatial configurations. [Lazebnik et al, 2006] [Fergus et al, 2003] …

Action Classification 7 All images contain humans; Objects small or absent; Large pose variation & occlusion; Background clutter; Challenging… Object classification: Presence of parts and their spatial configurations. [Lazebnik et al, 2006] [Fergus et al, 2003] …

Our Intuition 8 Focus on image regions that contain the most discriminative information.

Our Intuition 9 Focus on image regions that contain the most discriminative information. How to represent the features? Dense feature space Randomization & Discrimination How to explore this feature space?

Outline 10 Action Classification & Intuition Our Method Our Results Conclusion

11... Region Height Region Width Dense Feature Space Normalized Image Size of image region Center of image region

12... Region Height Region Width Normalized Image Size of image region Center of image region Dense Feature Space

13... Region Height Region Width Normalized Image Size of image region Center of image region Dense Feature Space

14... Region Height Region Width How can we identify the discriminative regions efficiently and effectively? Normalized Image Size of image region Center of image region Dense Feature Space Image size: N×N Image regions: O(N 6 )

15... Region Height Region Width Normalized Image Size of image region Center of image region Apply randomization to sample a subset of image patches Dense Feature Space Random Forests (RF)

16... Region Height Region Width This classOther classes Random Forests (RF) Normalized Image Size of image region Center of image region Dense Feature Space

17... Region Height Region Width RF with discriminative classifiers Normalized Image Size of image region Center of image region This classOther classes Dense Feature Space

18 Generalization Ability of RF Generalization error of a RF: : correlation between decision trees : strength of the decision trees Discriminative classifiers Better generalization Dense feature space decreases increases

19 …… …… RF with Discriminative Classifiers

20 …… …… Train a binary SVM RF with Discriminative Classifiers BoW or SPM of SIFT-LLC features

21 …… …… Train a binary SVM RF with Discriminative Classifiers Biggest information gain

22 …… …… Train a binary SVM RF with Discriminative Classifiers

23 …… …… RF with Discriminative Classifiers We stop growing the tree if: - The maximum depth is reached; - There is only one class at the node; - The entropy of the training data at the node is low

Classification With RF 24 …… …… Number of trees Class Label

Action Classification & Intuition Our Method Our Results Conclusion Outline 25

Results on VOC 2011 Actions 26 Action Others’ Best Our Method Jumping Phoning Playing instrument Reading Riding bike Riding horse Running Taking photo Using computer Walking Our method ranks the first in six out of ten classes.

Results on VOC 2011 Actions 27 Action Others’ Best Our Method Jumping Phoning Playing instrument Reading Riding bike Riding horse Running Taking photo Using computer Walking

Results on VOC 2011 Actions 28 Action Others’ Best Our Method Jumping Phoning Playing instrument Reading Riding bike Riding horse Running Taking photo Using computer Walking

29 Generalization Ability of RF Discriminative classifiers Better generalization Dense feature space Tree correlation decreases Tree strength increases dense feature(spatial pyramid) SPM feature Vs. strong classifierweak classifierVs. Train discriminative SVM classifiers Generate feature weights randomly (Results on PASCAL VOC 2010)

Action Classification & Intuition Our Method Our Results Conclusion Outline 30

Conclusion 31 Exploring dense image features can benefit action classification; Combining randomization and discrimination is an effective way to explore the dense image representation; Achieves very good performance based on only one type of image descriptor; Code will be available soon.

32 …… …… Train a binary SVM Acknowledgement Bangpeng Yao, Aditya Khosla, and Li Fei-Fei. “Combining Randomization and Discrimination for Fine-Grained Image Categorization.” CVPR Thanks to Su Hao, Olga Russakovsky, and Carsten Rother. Reference: