Download presentation
Presentation is loading. Please wait.
Published byMarian Horton Modified over 9 years ago
1
ACADS-SVMConclusions Introduction CMU-MMAC Unsupervised and weakly-supervised discovery of events in video (and audio) Fernando De la Torre
2
ACADS-SVMConclusions Introduction CMU-MMAC A dream
3
ACADS-SVMConclusions Introduction CMU-MMAC Outline Introduction CMU-Multimodal Activity database Unsupervised discovery of video events Aligned Cluster Analysis (ACA) Weakly-supervised discovery of video events Detection-Segmentation SVMs Conclusions
4
ACADS-SVMConclusions Introduction CMU-MMAC Quality of life technologies (QLoT)
5
ACADS-SVMConclusions Introduction CMU-MMAC Multimodal data collection 40 subjects, 5 recipes www.kitchen.cs.cmu.edu
6
ACADS-SVMConclusions Introduction CMU-MMAC Multimodal data collection 40 subjects, 5 recipes www.kitchen.cs.cmu.edu
7
ACADS-SVMConclusions Introduction CMU-MMAC Anomalous dataset
8
ACADS-SVMConclusions Introduction CMU-MMAC Time series analysis Anomalous detection formulated as detecting outliers in multimodal time series. – Supervised – Unsupervised – Semi-supervised or weakly supervised
9
ACADS-SVMConclusions Introduction CMU-MMAC Time series analysis Anomalous detection formulated as detecting outliers in multimodal time series. – Supervised – Unsupervised – Semi-supervised or weakly supervised
10
ACADS-SVMConclusions Introduction CMU-MMAC Unsupervised discovery of events in video
11
ACADS-SVMConclusions Introduction CMU-MMAC Motivation Mining facial expression for one subject
12
ACADS-SVMConclusions Introduction CMU-MMAC Mining facial expression for one subject Motivation Mining facial expression for one subject Summarization Visualization Indexing
13
ACADS-SVMConclusions Introduction CMU-MMAC Mining facial expression for one subject Looking up SleepingSmiling Looking forward Waking up Motivation Summarization Visualization Indexing
14
ACADS-SVMConclusions Introduction CMU-MMAC Mining facial expression of one subject Motivation Summarization Embedding Indexing
15
ACADS-SVMConclusions Introduction CMU-MMAC Mining facial expression for one subject Motivation Summarization Embedding Indexing
16
ACADS-SVMConclusions Introduction CMU-MMAC Related work in time series Change point detection (e.g. Page ‘54, Stephens 94’, Lai ‘95, Ge and Smyth ‘00, Steyvers & Brown ’05, Murphy et al. ‘07, Harchaoui et al. ‘08) Segmental HMMs (e.g. Ge and Smith ‘00, Kohlmoren et al. ’01, Ding & Fan ‘07) Mixtures of HMMs (e.g. Fine et al. ‘98, Murphy & Paskin ‘01, Oliver et al. ’02, Alon et al. ‘03) Switching LDS (e.g. Pavolvic et al. ‘00, Oh et al. ‘08, Turaga et al. ‘09) Hierarchical Dirichelet Process (e.g. Beal et al. ‘02, Fox et al. ‘08) Aligned Cluster Analysis (ACA)
17
ACADS-SVMConclusions Introduction CMU-MMAC Summarization with ACA
18
ACADS-SVMConclusions Introduction CMU-MMAC x y Kernel k-means and spectral clustering (Ding et al. ‘02, Dhillon et al. ‘04, Zass and Shashua ‘05, De la Torre ‘06) 1 2 3 4 5 6 7 8 9 10 x y x y x y
19
ACADS-SVMConclusions Introduction CMU-MMAC Problem formulation for ACA Labels (G) Start and end of the segments (h) Dynamic Time Alignment Kernel (Shimodaira et al. 01)
20
ACADS-SVMConclusions Introduction CMU-MMAC Dynamic Time Alignment Kernel (Shimodaira et al. 01) X [S i, S i+1 ) mcmc X mcmc Problem formulation for ACA
21
ACADS-SVMConclusions Introduction CMU-MMAC Matrix formulation for ACA samples segments clusters segments Dynamic Time Alignment Kernel (Shimodaira et al. 01) 23 frames, 3 clusters
22
ACADS-SVMConclusions Introduction CMU-MMAC Facial image features Appearance Active Appearance Models (Baker and Matthews ‘04) Upper face Lower face Shape Image features
23
ACADS-SVMConclusions Introduction CMU-MMAC Unsupervised facial event discovery
24
ACADS-SVMConclusions Introduction CMU-MMAC Cohn-Kanade: 30 people and five different expressions (surprise, joy, sadness, fear, anger) Facial event discovery across subjects
25
ACADS-SVMConclusions Introduction CMU-MMAC ACASpectral Clustering (SC) 0.87(.05)0.56(.04) Cohn-Kanade: 30 people and five different expressions (surprise, joy, sadness, fear, anger) Facial event discovery across subjects 10 sets of 30 people
26
ACADS-SVMConclusions Introduction CMU-MMAC Honey bee dance (Oh et al. ‘08) Seq 1Seq 2Seq 3Seq 4Seq 5Seq 6 ACA0.8450.9250.6000.9220.8780.928 PS- SLDS (Oh et al. ‘08)0.7590.9240.8310.9340.9040.910 HDP- VAR(1)-HMM (Fox et al. ‘08) 0.4650.4410.4560.8320.9320.887 Spectral Clustering0.6980.6310.5090.6710.5770.649 Three behaviors: 1-waggling 2-turning left 3-turning right
27
ACADS-SVMConclusions Introduction CMU-MMAC Clustering human motion
28
ACADS-SVMConclusions Introduction CMU-MMAC Weakly supervised discovery of events in images and video
29
ACADS-SVMConclusions Introduction CMU-MMAC Spot the differences!
30
ACADS-SVMConclusions Introduction CMU-MMAC What distinguish these images?
31
ACADS-SVMConclusions Introduction CMU-MMAC Classification of time series
32
ACADS-SVMConclusions Introduction CMU-MMAC Similarity of these problems? Global statistics are not distinctive enough! Better understanding of the discriminative regions or events
33
ACADS-SVMConclusions Introduction CMU-MMAC Image Bag of ‘regions’ At least one positive All negative
34
ACADS-SVMConclusions Introduction CMU-MMAC Support vector machines (SVMs)
35
ACADS-SVMConclusions Introduction CMU-MMAC Learning formulation Standard SVM -3 -2 0.5 3 (Andrews et. al. ’03, Felzenszwalb et al. ‘08)
36
ACADS-SVMConclusions Introduction CMU-MMAC Optimization all possible subwindows 100ms/image (480*640 pixels) (Lampert et al. CVPR08) 1) 2) 0.5 0.1 3) SVM with QP -3 -2 1 2
37
ACADS-SVMConclusions Introduction CMU-MMAC Discriminative patterns in time series At most k disjoint intervals We name it: k-segmentation Efficient search: Global optimum guaranteed! 10ms/sequence (15000 frames)
38
ACADS-SVMConclusions Introduction CMU-MMAC Representation of signals Training data Compute frame-level feature vectors IDs of visual words Visual dictionary clustering
39
ACADS-SVMConclusions Introduction CMU-MMAC K-segmentation Original signal IDs of visual words Histogram of visual words We need:
40
ACADS-SVMConclusions Introduction CMU-MMAC What is ? SVM parameters Original signal (x) IDs of visual words m-segmentation (m+1)-segmentation Consider m-segmentation: Situation 1: Situation 2:
41
ACADS-SVMConclusions Introduction CMU-MMAC Experiment 1 – glasses vs. no-glasses 624 images, 20 people under different expression/pose 8 people training (126 sunglasses, 128 no glasses), 12 testing (185 sunglasses and 185 no glasses)
42
ACADS-SVMConclusions Introduction CMU-MMAC Localization result
43
ACADS-SVMConclusions Introduction CMU-MMAC Experiment 2 – car vs. no car 400 images, half contains cars and other half no cars. Each image 10,000 SIFT descriptors and a vocabulary of 1,000 visual words.
44
ACADS-SVMConclusions Introduction CMU-MMAC Localization result
45
ACADS-SVMConclusions Introduction CMU-MMAC Bad localization cases
46
ACADS-SVMConclusions Introduction CMU-MMAC Classification performance Human labels Our method outperforms SVM with human labels!!! whole image discriminative regions
47
ACADS-SVMConclusions Introduction CMU-MMAC Experiment 3 – synthetic data Positive class Negative class Result k: maximum number of disjoint intervals. Accuracy
48
ACADS-SVMConclusions Introduction CMU-MMAC Experiment 4 – mouse activity Mouse activities: – Drinking, eating, exploring, grooming, sleeping
49
ACADS-SVMConclusions Introduction CMU-MMAC Result – F1 scores
50
ACADS-SVMConclusions Introduction CMU-MMAC Conclusions CMU Multimodal Activity database Unsupervised discovery of events in time-series – Aligned Cluster Analysis for summarization, indexing and visualization of time-series – Code online (www.humansensing.cs.cmu.edu)www.humansensing.cs.cmu.edu – Open problems: automatic selection of number of clusters Weakly-supervised discovery of events in time-series – DS-SVM – Novel & efficient algorithm for time series – Outperform methods with human labeled data Kernel methods a fundamental framework for multimodal data fusion.
51
ACADS-SVMConclusions Introduction CMU-MMAC Thanks Questions?
Similar presentations
© 2025 SlidePlayer.com. Inc.
All rights reserved.