1. ACADS-SVMConclusions Introduction CMU-MMAC Unsupervised and weakly-supervised discovery of events in video (and audio) Fernando De la Torre

2. ACADS-SVMConclusions Introduction CMU-MMAC A dream

3. ACADS-SVMConclusions Introduction CMU-MMAC Outline Introduction CMU-Multimodal Activity database Unsupervised discovery of video events Aligned Cluster Analysis (ACA) Weakly-supervised discovery of video events Detection-Segmentation SVMs Conclusions

4. ACADS-SVMConclusions Introduction CMU-MMAC Quality of life technologies (QLoT)

5. ACADS-SVMConclusions Introduction CMU-MMAC Multimodal data collection 40 subjects, 5 recipes www.kitchen.cs.cmu.edu

6. ACADS-SVMConclusions Introduction CMU-MMAC Multimodal data collection 40 subjects, 5 recipes www.kitchen.cs.cmu.edu

7. ACADS-SVMConclusions Introduction CMU-MMAC Anomalous dataset

8. ACADS-SVMConclusions Introduction CMU-MMAC Time series analysis Anomalous detection formulated as detecting outliers in multimodal time series. – Supervised – Unsupervised – Semi-supervised or weakly supervised

9. ACADS-SVMConclusions Introduction CMU-MMAC Time series analysis Anomalous detection formulated as detecting outliers in multimodal time series. – Supervised – Unsupervised – Semi-supervised or weakly supervised

10. ACADS-SVMConclusions Introduction CMU-MMAC Unsupervised discovery of events in video

11. ACADS-SVMConclusions Introduction CMU-MMAC Motivation Mining facial expression for one subject

12. ACADS-SVMConclusions Introduction CMU-MMAC Mining facial expression for one subject Motivation Mining facial expression for one subject Summarization Visualization Indexing

13. ACADS-SVMConclusions Introduction CMU-MMAC Mining facial expression for one subject Looking up SleepingSmiling Looking forward Waking up Motivation Summarization Visualization Indexing

14. ACADS-SVMConclusions Introduction CMU-MMAC Mining facial expression of one subject Motivation Summarization Embedding Indexing

15. ACADS-SVMConclusions Introduction CMU-MMAC Mining facial expression for one subject Motivation Summarization Embedding Indexing

16. ACADS-SVMConclusions Introduction CMU-MMAC Related work in time series Change point detection (e.g. Page ‘54, Stephens 94’, Lai ‘95, Ge and Smyth ‘00, Steyvers & Brown ’05, Murphy et al. ‘07, Harchaoui et al. ‘08) Segmental HMMs (e.g. Ge and Smith ‘00, Kohlmoren et al. ’01, Ding & Fan ‘07) Mixtures of HMMs (e.g. Fine et al. ‘98, Murphy & Paskin ‘01, Oliver et al. ’02, Alon et al. ‘03) Switching LDS (e.g. Pavolvic et al. ‘00, Oh et al. ‘08, Turaga et al. ‘09) Hierarchical Dirichelet Process (e.g. Beal et al. ‘02, Fox et al. ‘08) Aligned Cluster Analysis (ACA)

17. ACADS-SVMConclusions Introduction CMU-MMAC Summarization with ACA

18. ACADS-SVMConclusions Introduction CMU-MMAC x y Kernel k-means and spectral clustering (Ding et al. ‘02, Dhillon et al. ‘04, Zass and Shashua ‘05, De la Torre ‘06) 1 2 3 4 5 6 7 8 9 10 x y x y x y

19. ACADS-SVMConclusions Introduction CMU-MMAC Problem formulation for ACA Labels (G) Start and end of the segments (h) Dynamic Time Alignment Kernel (Shimodaira et al. 01)

20. ACADS-SVMConclusions Introduction CMU-MMAC Dynamic Time Alignment Kernel (Shimodaira et al. 01) X [S i, S i+1 ) mcmc X mcmc Problem formulation for ACA

21. ACADS-SVMConclusions Introduction CMU-MMAC Matrix formulation for ACA samples segments clusters segments Dynamic Time Alignment Kernel (Shimodaira et al. 01) 23 frames, 3 clusters

22. ACADS-SVMConclusions Introduction CMU-MMAC Facial image features Appearance Active Appearance Models (Baker and Matthews ‘04) Upper face Lower face Shape Image features

23. ACADS-SVMConclusions Introduction CMU-MMAC Unsupervised facial event discovery

24. ACADS-SVMConclusions Introduction CMU-MMAC Cohn-Kanade: 30 people and five different expressions (surprise, joy, sadness, fear, anger) Facial event discovery across subjects

25. ACADS-SVMConclusions Introduction CMU-MMAC ACASpectral Clustering (SC) 0.87(.05)0.56(.04) Cohn-Kanade: 30 people and five different expressions (surprise, joy, sadness, fear, anger) Facial event discovery across subjects 10 sets of 30 people

26. ACADS-SVMConclusions Introduction CMU-MMAC Honey bee dance (Oh et al. ‘08) Seq 1Seq 2Seq 3Seq 4Seq 5Seq 6 ACA0.8450.9250.6000.9220.8780.928 PS- SLDS (Oh et al. ‘08)0.7590.9240.8310.9340.9040.910 HDP- VAR(1)-HMM (Fox et al. ‘08) 0.4650.4410.4560.8320.9320.887 Spectral Clustering0.6980.6310.5090.6710.5770.649 Three behaviors: 1-waggling 2-turning left 3-turning right

27. ACADS-SVMConclusions Introduction CMU-MMAC Clustering human motion

28. ACADS-SVMConclusions Introduction CMU-MMAC Weakly supervised discovery of events in images and video

29. ACADS-SVMConclusions Introduction CMU-MMAC Spot the differences!

30. ACADS-SVMConclusions Introduction CMU-MMAC What distinguish these images?

31. ACADS-SVMConclusions Introduction CMU-MMAC Classification of time series

32. ACADS-SVMConclusions Introduction CMU-MMAC Similarity of these problems? Global statistics are not distinctive enough! Better understanding of the discriminative regions or events

33. ACADS-SVMConclusions Introduction CMU-MMAC Image Bag of ‘regions’ At least one positive All negative

34. ACADS-SVMConclusions Introduction CMU-MMAC Support vector machines (SVMs)

35. ACADS-SVMConclusions Introduction CMU-MMAC Learning formulation Standard SVM -3 -2 0.5 3 (Andrews et. al. ’03, Felzenszwalb et al. ‘08)

36. ACADS-SVMConclusions Introduction CMU-MMAC Optimization all possible subwindows 100ms/image (480*640 pixels) (Lampert et al. CVPR08) 1) 2) 0.5 0.1 3) SVM with QP -3 -2 1 2

37. ACADS-SVMConclusions Introduction CMU-MMAC Discriminative patterns in time series At most k disjoint intervals We name it: k-segmentation Efficient search: Global optimum guaranteed! 10ms/sequence (15000 frames)

38. ACADS-SVMConclusions Introduction CMU-MMAC Representation of signals Training data Compute frame-level feature vectors IDs of visual words Visual dictionary clustering

39. ACADS-SVMConclusions Introduction CMU-MMAC K-segmentation Original signal IDs of visual words Histogram of visual words We need:

40. ACADS-SVMConclusions Introduction CMU-MMAC What is ? SVM parameters Original signal (x) IDs of visual words m-segmentation  (m+1)-segmentation Consider m-segmentation: Situation 1: Situation 2:

41. ACADS-SVMConclusions Introduction CMU-MMAC Experiment 1 – glasses vs. no-glasses 624 images, 20 people under different expression/pose 8 people training (126 sunglasses, 128 no glasses), 12 testing (185 sunglasses and 185 no glasses)

42. ACADS-SVMConclusions Introduction CMU-MMAC Localization result

43. ACADS-SVMConclusions Introduction CMU-MMAC Experiment 2 – car vs. no car 400 images, half contains cars and other half no cars. Each image 10,000 SIFT descriptors and a vocabulary of 1,000 visual words.

44. ACADS-SVMConclusions Introduction CMU-MMAC Localization result

45. ACADS-SVMConclusions Introduction CMU-MMAC Bad localization cases

46. ACADS-SVMConclusions Introduction CMU-MMAC Classification performance Human labels Our method outperforms SVM with human labels!!! whole image discriminative regions

47. ACADS-SVMConclusions Introduction CMU-MMAC Experiment 3 – synthetic data Positive class Negative class Result k: maximum number of disjoint intervals. Accuracy

48. ACADS-SVMConclusions Introduction CMU-MMAC Experiment 4 – mouse activity Mouse activities: – Drinking, eating, exploring, grooming, sleeping

49. ACADS-SVMConclusions Introduction CMU-MMAC Result – F1 scores

50. ACADS-SVMConclusions Introduction CMU-MMAC Conclusions CMU Multimodal Activity database Unsupervised discovery of events in time-series – Aligned Cluster Analysis for summarization, indexing and visualization of time-series – Code online (www.humansensing.cs.cmu.edu)www.humansensing.cs.cmu.edu – Open problems: automatic selection of number of clusters Weakly-supervised discovery of events in time-series – DS-SVM – Novel & efficient algorithm for time series – Outperform methods with human labeled data Kernel methods a fundamental framework for multimodal data fusion.

51. ACADS-SVMConclusions Introduction CMU-MMAC Thanks Questions?