1. Retrieving Actions in Group Contexts Tian Lan, Yang Wang, Greg Mori, Stephen Robinovitch Simon Fraser University Sept. 11, 2010

2. Outline Action Retrieval as Ranking Results and Future Work Contextual Representation of Actions

4. Nursing Home Fall analysis in nursing home surveillance videos – a system automatically rank the videos according to the relevance to fall action is expected

5. Action-Action Context Context What other people are doing ?

6. Actions in Group Context Motivation – human actions are rarely performed in isolation, the actions of individuals in a group can serve as context for each other. Goal – explore the benefit of contextual information in action retrieval in challenging real-world applications

7. Action Context Descriptor τ action τ z + Focal personContext

8. Action Context Descriptor Feature Descriptor Multi-class SVM action class score action class score … action class score max action class score e.g. HOG by Dalal & Triggs

9. Outline Action Retrieval as Ranking Results and Future Work Contextual Representation of Actions

10. Classification or Retrieval Previous Work – Most work in human action understanding focuses on action classification.

11. Classification or Retrieval Most surveillance tasks are typical retrieval tasks – retrieve a small video segment contains a particular action from thousands of hours of videos. The “action of interest” is rare event – Extremely imbalanced classes

12. Action Retrieval Rank according to the relevance to falls Query : fall

13. Learning Input: document-rank pair (x i,y i ) Optimization Joachims, KDD 06

14. Ranking SVM Ranking function h(x) h(x) Rank r1 Rank r2 Rank r3

15. Action Retrieval - training irrelevant very relevant

16. Outline Action Retrieval as Ranking Results and Future Work Contextual Representation of Actions

17. Dataset Nursing Home Dataset 5 action categories: walking, standing, sitting, bending and falling. (per person) 18 video clips. Query: fall Collective Activity Dataset (Choi et al. VS. 09) 5 action categories: crossing, waiting, queuing, walking, talking. (per person) 44 video clips. Query: each of the five actions

18. Nursing Home Dataset Dataset

19. Collective Activity Dataset

20. System Overview Person Detector Person Detector Person Descriptor Person Descriptor Video u v Rank SVM Rank SVM Pedestrian Detection by Felzenszwalb et al. Background Subtraction HOG by Dalal & Triggs LST by Loy et al. at cvpr 09

21. Baselines Context vs No Context – Action Context Descriptor – Original feature descriptors, e.g. HOG (Dalal & Triggs at CVPR 05), LST (Loy et al. at CVPR 09) RankSVM vs SVM Methods – Context + RankSVM (our method) – Context + SVM – No Context + RankSVM – No Context + SVM

23. Retrieval Results Nursing Home Dataset

24. Retrieval Results Collective Activity Dataset

25. Retrieval Results Collective Activity Dataset

26. Retrieval Results Collective Activity Dataset

27. 1 1 2 2 3 3 4 4

28. 7 7 8 8 6 6 5 5

29. Action Classification [10] Choi et al. in VS. 09 Collective Activity Dataset

30. Conclusion A new contextual feature descriptor to represent actions – action context (AC) descriptor Formulate our problem as a retrieval task.

31. Future Work Contextual Feature Descriptors – How to only encode useful context? Rank-SVM loss, optimize the NDCG score

32. Thank you!

34. 7 7 8 8 6 6 5 5