Lei Li Computer Science Department Carnegie Mellon University Pre Proposal Time Series Learning completed work 11/27/2015

Outline Completed Work –Mining w/ Missing Value –Parallel Learning –Natural Motion Stitching Ongoing & Proposed Work Other Related Work 2

Outline Completed Work –Mining w/ Missing Value Motivation Problem Definition Proposed Method Results –Parallel Learning –Natural Motion Stitching Ongoing & Proposed Work Other Related Work 3

Occlusion in Motion Capture Motion Capture: –Markers on human actors –Cameras used to track the 3D positions –Duration: –93 dimensional body-local coordinates after preprocessing (31-bones) Challenge: –Occlusions Other general scenario: –Missing value in Sensor data: Out of battery, transmission error, etc –Unable to observe, e.g. historical/future observation 4 From mocap.cs.cmu.edu

Time Marker/Sensor blackout Given To find algorithm for: –mining hidden variables and evolving patterns –recovering missing values –compression/summarization –segmentation Problem Definition 5

Time Marker/Sensor blackout Problem Definition (cont’) Want the algorithms to be: –Effective –Scalable: to duration of sequences –Blackouts –Automatic: no/few parameters to be set 6

Proposed Method: Intuition 7 Left Hand Right Hand missing Recover using Correlation among multiple markers

Proposed Method: Intuition 8 Left Hand Right Hand missing Recover using Dynamics temporal moving pattern

Underlying Model 9 z 1 = z 0 +ω 0 z n+1 = F∙z n +ω n x n = G∙z n +ε n Z1Z1 Z1Z1 Z2Z2 Z2Z2 Z3Z3 Z3Z3 Z4Z4 Z4Z4 X1X1 X1X1 X2X2 X2X2 X3X3 X3X3 X4X4 X4X4 N (F∙z 1, Λ) N (z 0, Γ) N (G∙z 3, Σ) N (F∙z 2, Λ) N (G∙z 1, Σ) N (G∙z 2, Σ) N (G∙z 4, Σ) N (F∙z 3, Λ) N (F∙z 4, Λ) … Model parameters: θ={ z 0, Γ, F, Λ, G, Σ} Use Linear Dynamical Systems to model whole sequence.

DynaMMo Intuition How to recover the missing values? 10

DynaMMo: How to Recover? 11 ×

DynaMMo: How to Recover? 12 × × ×

DynaMMo: How to Recover? 13 × × × ×

DynaMMo: How to Recover? 14 × × × × ×

DynaMMo: How to Recover? 15 × × × × × ×

How to Compress Naive idea #1: use SVD Naive idea #2: store parameters of LDS Naive idea #3: store parameters of LDS and all hidden variables (expectation) Proposed Methods: use check points –Fixed hop –Optimal (dynamic programming) –Near optimal (adaptive) 16

DynaMMo Compression: Intuition 17 observations w/ missing values get hidden variables and model parameters keep only a (best) portion of them Same idea could be used in segmentation and forecasting and

DynaMMo w/ Optimal Compression: Intuition 18 observations w/ missing values get hidden variables and model parameters keep only a (best) portion of them Same idea could be used in segmentation and forecasting and

How to Segment Segment by threshold on prediction error 19 original data reconstruction error

Outline Completed Work –Mining Missing Value Motivation Problem Definition Proposed Method Results –Parallel Learning –Natural Motion Stitching Ongoing & Proposed Work Other Related Work 20

Results – Better Missing Recovery 21 Reconstructionerror occlusion length Ideal Proposed MSVD

Results – Better Compression 22 Compression ratio error DynaMMo w/ optimal compression Ideal

Results – Segmentation Find the transition during “running” to “stop”. 23 left hip left femur reconstruction error

Outline Completed Work –Mining Missing Value Contribution: the most accurate mining algorithms for TS with missing value so far. –Parallel Learning –Natural Motion Stitching Ongoing & Proposed Work Other Related Work 24

Outline Completed Work –Mining Missing Value –Parallel Learning Motivation Problem Definition Proposed Method Results –Natural Motion Stitching Ongoing & Proposed Work Other Related Work 25

Challenge for Learning LDS on SMP 26 Time * Measured Estimated Position of left elbow step 1

Challenge for Learning LDS on SMP 27 Time * Measured Estimated Position of left elbow step 2 * Intuition: #2 may be close to #1

Challenge for Learning LDS on SMP 28 Time * Measured Estimated Position of left elbow * * * * * Forward

Challenge for Learning LDS on SMP 29 Time * Measured Estimated Position of left elbow Backward * * * * * *

Challenge for Learning LDS on SMP 30 Time * Measured Estimated Position of left elbow Backward * * * * * * * * * *

Challenge for Learning LDS on SMP 31 Time * Measured Reconstructed Signal Position of left elbow Backward * * * * * * * * * *

Outline Completed Work –Mining Missing Value –Parallel Learning Motivation Problem Definition Proposed Method Results –Natural Motion Stitching Ongoing & Proposed Work Other Related Work 32

Problem Definition 33 Problem: –Given a sequence of numbers, find the best model parameters for Linear Dynamical System Goal: –Achieve ~ linear speed up on multi-core Assumption: –shared memory architecture

Cut-And-Stitch Intuition 34 z1z1 z1z1 y1y1 y1y1 y2y2 y2y2 z3z3 z3z3 y3y3 y3y3 z4z4 z4z4 y4y4 y4y4 z5z5 z5z5 y5y5 y5y5 y6y6 y6y6 z2z2 z2z2 z6z6 z6z6 υ2,Φ2,η2,Ψ2υ2,Φ2,η2,Ψ2 υ1,Φ1,η1,Ψ1υ1,Φ1,η1,Ψ1 z1z1 z1z1 y1y1 y1y1 y2y2 y2y2 z' 2 z2z2 z2z2 z3z3 z3z3 y3y3 y3y3 z4z4 z4z4 y4y4 y4y4 z' 4 z5z5 z5z5 y5y5 y5y5 y6y6 y6y6 z6z6 z6z6 υ3,Φ3,η3,Ψ3υ3,Φ3,η3,Ψ3 reconcile later Cut start computation without feedback from previous node Stitch

Cut-And-Stitch: illustration 35 Time * Measured Estimated Position of left elbow * * Cut-Forward 1

Cut-And-Stitch: illustration 36 Time * Measured Estimated Position of left elbow * * * * * Cut-Forward 2

Cut-And-Stitch: illustration 37 Time * Measured Position of left elbow * * * * * * * * Cut-Backward

Cut-And-Stitch: illustration 38 Time * Measured Position of left elbow * * * * * * * * Stitch * * reconciliation

Outline Completed Work –Mining Missing Value –Parallel Learning Motivation Problem Definition Proposed Method Results –Natural Motion Stitching Ongoing & Proposed Work Other Related Work 39

Near Linear Speedup 40 speedup # of processors ideal Dataset: CMU Mocap #16 mocap.cs.cmu.edu

No loss of accuracy 41 ~ IDENTICAL

Outline Completed Work –Mining Missing Value –Parallel Learning Contribution: the 1 st parallel algorithm for learning LDS –Natural Motion Stitching Ongoing & Proposed Work Other Related Work 42

Outline Completed Work –Mining Missing Value –Parallel Learning –Natural Motion Stitching Motivation Problem Definition Proposed Method Results Ongoing & Proposed Work Other Related Work 43

How to generate new natural motion? Computer Game industry –E.g. generate a smooth “goal kick” in soccer game Movie Industry –E.g. Shrek 44

A Database Approach Select best stitchable segments from a set of basic motion pieces and generate new natural motions 45

Problem Definition Given two motion-capture sequences that are to be stitched together, how can we assess the goodness of the stitching? Euclidean will fail Best stitchable motion?

Outline Completed Work –Mining Missing Value –Parallel Learning –Natural Motion Stitching Motivation Problem Definition Proposed Method Results Ongoing & Proposed Work Other Related Work 47

Minimizing Stitching Effort Minimize the energy/effort spent by human during the transition Compute the effort using dynamics from Kalman Filters 48

Result 49

Outline Completed Work –Mining Missing Value –Parallel Learning –Natural Motion Stitching Contribution: A principled distance function for motion stitching Ongoing & Proposed Work Other Related Work 50

Ongoing Work Time series clustering by fingerprinting Bone length constrained occlusion filling DynaMMo+, enhanced algorithm for missing value recovery Mining multiple categorical streams 51

Proposed Work Multi-resolution learning for long time series Data center thermal management by mining sensor data 52

Roadmap: Time Series Learning and Mining 53 Motion stitching Parallel LDS Learning (cut and stitching) TS Clustering Multi-resolution learning Non Constrained Constrained Complete Missing DynaMMo DynaMMo+ Bone Length Constrained Occlusion Filling DC modeling (proposed)

Reference [Li 2009] DynaMMo: Mining and Summarization of Coevolving Sequences with Missing Values. KDD '09. [Li 2008c] Cut-and-stitch: efficient parallel learning of linear dynamical systems on SMPs. KDD '08. [Li 2008l] Laziness is a virtue: Motion stitching using effort minimization. Eurographics