ACADS-SVMConclusions Introduction CMU-MMAC Unsupervised and weakly-supervised discovery of events in video (and audio) Fernando De la Torre

ACADS-SVMConclusions Introduction CMU-MMAC A dream

ACADS-SVMConclusions Introduction CMU-MMAC Outline Introduction CMU-Multimodal Activity database Unsupervised discovery of video events Aligned Cluster Analysis (ACA) Weakly-supervised discovery of video events Detection-Segmentation SVMs Conclusions

ACADS-SVMConclusions Introduction CMU-MMAC Quality of life technologies (QLoT)

ACADS-SVMConclusions Introduction CMU-MMAC Multimodal data collection 40 subjects, 5 recipes

ACADS-SVMConclusions Introduction CMU-MMAC Multimodal data collection 40 subjects, 5 recipes

ACADS-SVMConclusions Introduction CMU-MMAC Anomalous dataset

ACADS-SVMConclusions Introduction CMU-MMAC Time series analysis Anomalous detection formulated as detecting outliers in multimodal time series. – Supervised – Unsupervised – Semi-supervised or weakly supervised

ACADS-SVMConclusions Introduction CMU-MMAC Time series analysis Anomalous detection formulated as detecting outliers in multimodal time series. – Supervised – Unsupervised – Semi-supervised or weakly supervised

ACADS-SVMConclusions Introduction CMU-MMAC Unsupervised discovery of events in video

ACADS-SVMConclusions Introduction CMU-MMAC Motivation Mining facial expression for one subject

ACADS-SVMConclusions Introduction CMU-MMAC Mining facial expression for one subject Motivation Mining facial expression for one subject Summarization Visualization Indexing

ACADS-SVMConclusions Introduction CMU-MMAC Mining facial expression for one subject Looking up SleepingSmiling Looking forward Waking up Motivation Summarization Visualization Indexing

ACADS-SVMConclusions Introduction CMU-MMAC Mining facial expression of one subject Motivation Summarization Embedding Indexing

ACADS-SVMConclusions Introduction CMU-MMAC Mining facial expression for one subject Motivation Summarization Embedding Indexing

ACADS-SVMConclusions Introduction CMU-MMAC Related work in time series Change point detection (e.g. Page ‘54, Stephens 94’, Lai ‘95, Ge and Smyth ‘00, Steyvers & Brown ’05, Murphy et al. ‘07, Harchaoui et al. ‘08) Segmental HMMs (e.g. Ge and Smith ‘00, Kohlmoren et al. ’01, Ding & Fan ‘07) Mixtures of HMMs (e.g. Fine et al. ‘98, Murphy & Paskin ‘01, Oliver et al. ’02, Alon et al. ‘03) Switching LDS (e.g. Pavolvic et al. ‘00, Oh et al. ‘08, Turaga et al. ‘09) Hierarchical Dirichelet Process (e.g. Beal et al. ‘02, Fox et al. ‘08) Aligned Cluster Analysis (ACA)

ACADS-SVMConclusions Introduction CMU-MMAC Summarization with ACA

ACADS-SVMConclusions Introduction CMU-MMAC x y Kernel k-means and spectral clustering (Ding et al. ‘02, Dhillon et al. ‘04, Zass and Shashua ‘05, De la Torre ‘06) x y x y x y

ACADS-SVMConclusions Introduction CMU-MMAC Problem formulation for ACA Labels (G) Start and end of the segments (h) Dynamic Time Alignment Kernel (Shimodaira et al. 01)

ACADS-SVMConclusions Introduction CMU-MMAC Dynamic Time Alignment Kernel (Shimodaira et al. 01) X [S i, S i+1 ) mcmc X mcmc Problem formulation for ACA

ACADS-SVMConclusions Introduction CMU-MMAC Matrix formulation for ACA samples segments clusters segments Dynamic Time Alignment Kernel (Shimodaira et al. 01) 23 frames, 3 clusters

ACADS-SVMConclusions Introduction CMU-MMAC Facial image features Appearance Active Appearance Models (Baker and Matthews ‘04) Upper face Lower face Shape Image features

ACADS-SVMConclusions Introduction CMU-MMAC Unsupervised facial event discovery

ACADS-SVMConclusions Introduction CMU-MMAC Cohn-Kanade: 30 people and five different expressions (surprise, joy, sadness, fear, anger) Facial event discovery across subjects

ACADS-SVMConclusions Introduction CMU-MMAC ACASpectral Clustering (SC) 0.87(.05)0.56(.04) Cohn-Kanade: 30 people and five different expressions (surprise, joy, sadness, fear, anger) Facial event discovery across subjects 10 sets of 30 people

ACADS-SVMConclusions Introduction CMU-MMAC Honey bee dance (Oh et al. ‘08) Seq 1Seq 2Seq 3Seq 4Seq 5Seq 6 ACA PS- SLDS (Oh et al. ‘08) HDP- VAR(1)-HMM (Fox et al. ‘08) Spectral Clustering Three behaviors: 1-waggling 2-turning left 3-turning right

ACADS-SVMConclusions Introduction CMU-MMAC Clustering human motion

ACADS-SVMConclusions Introduction CMU-MMAC Weakly supervised discovery of events in images and video

ACADS-SVMConclusions Introduction CMU-MMAC Spot the differences!

ACADS-SVMConclusions Introduction CMU-MMAC What distinguish these images?

ACADS-SVMConclusions Introduction CMU-MMAC Classification of time series

ACADS-SVMConclusions Introduction CMU-MMAC Similarity of these problems? Global statistics are not distinctive enough! Better understanding of the discriminative regions or events

ACADS-SVMConclusions Introduction CMU-MMAC Image Bag of ‘regions’ At least one positive All negative

ACADS-SVMConclusions Introduction CMU-MMAC Support vector machines (SVMs)

ACADS-SVMConclusions Introduction CMU-MMAC Learning formulation Standard SVM (Andrews et. al. ’03, Felzenszwalb et al. ‘08)

ACADS-SVMConclusions Introduction CMU-MMAC Optimization all possible subwindows 100ms/image (480*640 pixels) (Lampert et al. CVPR08) 1) 2) ) SVM with QP

ACADS-SVMConclusions Introduction CMU-MMAC Discriminative patterns in time series At most k disjoint intervals We name it: k-segmentation Efficient search: Global optimum guaranteed! 10ms/sequence (15000 frames)

ACADS-SVMConclusions Introduction CMU-MMAC Representation of signals Training data Compute frame-level feature vectors IDs of visual words Visual dictionary clustering

ACADS-SVMConclusions Introduction CMU-MMAC K-segmentation Original signal IDs of visual words Histogram of visual words We need:

ACADS-SVMConclusions Introduction CMU-MMAC What is ? SVM parameters Original signal (x) IDs of visual words m-segmentation  (m+1)-segmentation Consider m-segmentation: Situation 1: Situation 2:

ACADS-SVMConclusions Introduction CMU-MMAC Experiment 1 – glasses vs. no-glasses 624 images, 20 people under different expression/pose 8 people training (126 sunglasses, 128 no glasses), 12 testing (185 sunglasses and 185 no glasses)

ACADS-SVMConclusions Introduction CMU-MMAC Localization result

ACADS-SVMConclusions Introduction CMU-MMAC Experiment 2 – car vs. no car 400 images, half contains cars and other half no cars. Each image 10,000 SIFT descriptors and a vocabulary of 1,000 visual words.

ACADS-SVMConclusions Introduction CMU-MMAC Localization result

ACADS-SVMConclusions Introduction CMU-MMAC Bad localization cases

ACADS-SVMConclusions Introduction CMU-MMAC Classification performance Human labels Our method outperforms SVM with human labels!!! whole image discriminative regions

ACADS-SVMConclusions Introduction CMU-MMAC Experiment 3 – synthetic data Positive class Negative class Result k: maximum number of disjoint intervals. Accuracy

ACADS-SVMConclusions Introduction CMU-MMAC Experiment 4 – mouse activity Mouse activities: – Drinking, eating, exploring, grooming, sleeping

ACADS-SVMConclusions Introduction CMU-MMAC Result – F1 scores

ACADS-SVMConclusions Introduction CMU-MMAC Conclusions CMU Multimodal Activity database Unsupervised discovery of events in time-series – Aligned Cluster Analysis for summarization, indexing and visualization of time-series – Code online ( – Open problems: automatic selection of number of clusters Weakly-supervised discovery of events in time-series – DS-SVM – Novel & efficient algorithm for time series – Outperform methods with human labeled data Kernel methods a fundamental framework for multimodal data fusion.

ACADS-SVMConclusions Introduction CMU-MMAC Thanks Questions?