D YNA MM O : M INING AND S UMMARIZATION OF C OEVOLVING S EQUENCES WITH M ISSING V ALUES Lei Li joint work with Christos Faloutsos, James McCann, Nancy Pollard June 9, 2009 Microsoft Research

C HALLENGE Multidimensional coevolving time series: Motion Capture sequence Temperature/humidity monitoring Daily Chlorine level measurement in drinking water system Big challenge: Missing observations Mining with missing values Find hidden patterns Use of hidden patterns 2 Li et al. DynaMMo: Mining and Summarization of Coevolving Sequences with Missing Values

M ISSING VALUES IN M OCAP 3 Li et al. DynaMMo: Mining and Summarization of Coevolving Sequences with Missing Values Time Marker/Sensor Left Hand position Right Hand position

B LANK - OUTS IN M OCAP 4 Li et al. DynaMMo: Mining and Summarization of Coevolving Sequences with Missing Values Time Marker/Sensor Left Hand position Right Hand position

G OAL We want recovering, mining and summarization algorithms to be: 1. Effective: low reconstruction error, agreeing with human intuition (e.g. natural reconstructed motion for mocap) 2. Scalable: to time-duration T of the sequences. 3. Black-outs: It should be able to handle “black-outs”, when all markers dissappear (eg., a person running behind a wall, for a moment). 4. Automatic: The method should require no parameters to be set by the user. 5 Li et al. DynaMMo: Mining and Summarization of Coevolving Sequences with Missing Values

O UTLINE Scenario and Motivation Proposed Methods – DynaMMo Experimental Results Conclusion Recovering missing values Compression and summarization Forecasting Segmentation 6 Li et al. DynaMMo: Mining and Summarization of Coevolving Sequences with Missing Values

S CENARIO : M OTION C APTURE Motion Capture: Markers on human actors Cameras used to track the 3D positions Duration: dimensional body-local coordinates after preprocessing (31- bones) Challenge: Occlusions Other general scenario: Missing value in Sensor data: Out of battery, transmission error, etc Unable to observe, e.g. historical/future observation 7 From mocap.cs.cmu.edu Li et al. DynaMMo: Mining and Summarization of Coevolving Sequences with Missing Values

O BSERVATION AND M OTIVATION Left Hand Right Hand Left Foot Right Foot Dynamics: temporal moving pattern Correlation among multiple markers Use both dynamics and correlation to solve occlusion. 8 Li et al. DynaMMo: Mining and Summarization of Coevolving Sequences with Missing Values

T HE U NDERLYING T IME S ERIES M ODEL L INEAR D YNAMICAL S YSTEMS z 1 = z 0 +ω 0 z n+1 = F∙z n +ω n x n = G∙z n +ε n Z1Z1 Z1Z1 Z2Z2 Z2Z2 Z3Z3 Z3Z3 Z4Z4 Z4Z4 X1X1 X1X1 X2X2 X2X2 X3X3 X3X3 X4X4 X4X4 N (F∙z 1, Λ) N (z 0, Γ) N (G∙z 3, Σ) N (F∙z 2, Λ) N (G∙z 1, Σ) N (G∙z 2, Σ) N (G∙z 4, Σ) N (F∙z 3, Λ) N (F∙z 4, Γ) … Model parameters: θ={ z 0, Γ, F, Λ, G, Σ} 9 Li et al. DynaMMo: Mining and Summarization of Coevolving Sequences with Missing Values

D YNA MM O R ECOVERING A LGORITHM Expectation Maximization Intuition: Finding the best model parameters (θ) and missing values for X to minimize the expected loglikelihood: See details in paper 10 Li et al. DynaMMo: Mining and Summarization of Coevolving Sequences with Missing Values

D YNA MM O I NTUITION : Left Hand Right Hand missing correlation dynamics 11 Li et al. DynaMMo: Mining and Summarization of Coevolving Sequences with Missing Values

D YNA MM O C OMPRESSION : I NTUITION observations w/ missing values get hidden variables and model parameters keep only a (best) portion of them Same idea could be used in segmentation and forecasting 12 Li et al. DynaMMo: Mining and Summarization of Coevolving Sequences with Missing Values

E XPERIMENT Dataset: – Chlorine: Chlorine level in drinking water system Duration 4310 time ticks 166 sequences – Mocap: full body human motion capture dataset – 58 motions – each with duration , 93 dimensions Occlusion: random mask out Baseline: – linear interpolation and spline – MSVD: Missing value SVD algorithm EM flavored version of SVD. 13 Li et al. DynaMMo: Mining and Summarization of Coevolving Sequences with Missing Values

R ESULTS Reconstruction Error for random mask out Scalability: computation time to duration Forecasting case study Compression: error versus space Segmentation for synthetic and real data 14 Li et al. DynaMMo: Mining and Summarization of Coevolving Sequences with Missing Values

D YNA MM O R ECONSTRUCTION R ESULT ( AVERAGE OVER 10 REPEATS ) 15 Li et al. DynaMMo: Mining and Summarization of Coevolving Sequences with Missing Values Chlorine dataset Reconstruction error better Mocap dataset average consecutive occlusion length worse easydifficult

S CATTER COMPARISON : D YNA MM O VS MSVD 16 Li et al. DynaMMo: Mining and Summarization of Coevolving Sequences with Missing Values error of MSVD error of DynaMMo

D YNA MM O S CALABILITY 17 Li et al. DynaMMo: Mining and Summarization of Coevolving Sequences with Missing Values Computation time sequence length (chlorine data)

D YNA MM O F ORECASTING 18 Li et al. DynaMMo: Mining and Summarization of Coevolving Sequences with Missing Values Actual Data Predicted signal using learned model

D YNA MM O C OMPRESSION 19 Li et al. DynaMMo: Mining and Summarization of Coevolving Sequences with Missing Values Compression ratio error better worse more space less space DynaMMo optimal compression

D YNA MM O S EGMENTATION 20 Li et al. DynaMMo: Mining and Summarization of Coevolving Sequences with Missing Values original data segmentation using reconstruction error

M OCAP S EGMENTATION R UNNING T RANSITION MOTION ( MOCAP #16.8) 21 Li et al. DynaMMo: Mining and Summarization of Coevolving Sequences with Missing Values left hip left femur segmentation using reconstruction error

C ONTRIBUTION We propose algorithms DynaMMo for DynaMMo meets all goals: 1. Effective: low reconstruction error, agreeing with human intuition (e.g. natural reconstructed motion for mocap) 2. Scalable: computation time linear to length/duration T of the sequences. 3. Black-outs: able to handle “black-outs”, when all markers dissappear. 4. Automatic: The methods should require few parameter to be set by the user. 22 Li et al. DynaMMo: Mining and Summarization of Coevolving Sequences with Missing Values Recovering missing values Compression and summarization Forecasting Segmentation

Q UESTION Thanks! Contact: 23 Christos Faloutsos Lei Li Nancy Pollard Jim McCann Li et al. DynaMMo: Mining and Summarization of Coevolving Sequences with Missing Values

B ACKUP SLIDES 24 Li et al. DynaMMo: Mining and Summarization of Coevolving Sequences with Missing Values

25 Li et al. DynaMMo: Mining and Summarization of Coevolving Sequences with Missing Values