1. Lei Li Computer Science Department Carnegie Mellon University Pre Proposal Time Series Learning completed work 11/27/2015

2. Outline Completed Work –Mining w/ Missing Value –Parallel Learning –Natural Motion Stitching Ongoing & Proposed Work Other Related Work 2

3. Outline Completed Work –Mining w/ Missing Value Motivation Problem Definition Proposed Method Results –Parallel Learning –Natural Motion Stitching Ongoing & Proposed Work Other Related Work 3

4. Occlusion in Motion Capture Motion Capture: –Markers on human actors –Cameras used to track the 3D positions –Duration: 100-500 –93 dimensional body-local coordinates after preprocessing (31-bones) Challenge: –Occlusions Other general scenario: –Missing value in Sensor data: Out of battery, transmission error, etc –Unable to observe, e.g. historical/future observation 4 From mocap.cs.cmu.edu

5. Time Marker/Sensor blackout Given To find algorithm for: –mining hidden variables and evolving patterns –recovering missing values –compression/summarization –segmentation Problem Definition 5

6. Time Marker/Sensor blackout Problem Definition (cont’) Want the algorithms to be: –Effective –Scalable: to duration of sequences –Blackouts –Automatic: no/few parameters to be set 6

7. Proposed Method: Intuition 7 Left Hand Right Hand missing Recover using Correlation among multiple markers

8. Proposed Method: Intuition 8 Left Hand Right Hand missing Recover using Dynamics temporal moving pattern

9. Underlying Model 9 z 1 = z 0 +ω 0 z n+1 = F∙z n +ω n x n = G∙z n +ε n Z1Z1 Z1Z1 Z2Z2 Z2Z2 Z3Z3 Z3Z3 Z4Z4 Z4Z4 X1X1 X1X1 X2X2 X2X2 X3X3 X3X3 X4X4 X4X4 N (F∙z 1, Λ) N (z 0, Γ) N (G∙z 3, Σ) N (F∙z 2, Λ) N (G∙z 1, Σ) N (G∙z 2, Σ) N (G∙z 4, Σ) N (F∙z 3, Λ) N (F∙z 4, Λ) … Model parameters: θ={ z 0, Γ, F, Λ, G, Σ} Use Linear Dynamical Systems to model whole sequence.

10. DynaMMo Intuition How to recover the missing values? 10

11. DynaMMo: How to Recover? 11 ×

12. DynaMMo: How to Recover? 12 × × ×

13. DynaMMo: How to Recover? 13 × × × ×

14. DynaMMo: How to Recover? 14 × × × × ×

15. DynaMMo: How to Recover? 15 × × × × × ×

16. How to Compress Naive idea #1: use SVD Naive idea #2: store parameters of LDS Naive idea #3: store parameters of LDS and all hidden variables (expectation) Proposed Methods: use check points –Fixed hop –Optimal (dynamic programming) –Near optimal (adaptive) 16

17. DynaMMo Compression: Intuition 17 observations w/ missing values get hidden variables and model parameters keep only a (best) portion of them Same idea could be used in segmentation and forecasting and

18. DynaMMo w/ Optimal Compression: Intuition 18 observations w/ missing values get hidden variables and model parameters keep only a (best) portion of them Same idea could be used in segmentation and forecasting and

19. How to Segment Segment by threshold on prediction error 19 original data reconstruction error

20. Outline Completed Work –Mining Missing Value Motivation Problem Definition Proposed Method Results –Parallel Learning –Natural Motion Stitching Ongoing & Proposed Work Other Related Work 20

21. Results – Better Missing Recovery 21 Reconstructionerror occlusion length Ideal Proposed MSVD

22. Results – Better Compression 22 Compression ratio error DynaMMo w/ optimal compression Ideal

23. Results – Segmentation Find the transition during “running” to “stop”. 23 left hip left femur reconstruction error

24. Outline Completed Work –Mining Missing Value Contribution: the most accurate mining algorithms for TS with missing value so far. –Parallel Learning –Natural Motion Stitching Ongoing & Proposed Work Other Related Work 24

25. Outline Completed Work –Mining Missing Value –Parallel Learning Motivation Problem Definition Proposed Method Results –Natural Motion Stitching Ongoing & Proposed Work Other Related Work 25

26. Challenge for Learning LDS on SMP 26 Time * Measured Estimated Position of left elbow step 1

27. Challenge for Learning LDS on SMP 27 Time * Measured Estimated Position of left elbow step 2 * Intuition: #2 may be close to #1

28. Challenge for Learning LDS on SMP 28 Time * Measured Estimated Position of left elbow * * * * * Forward

29. Challenge for Learning LDS on SMP 29 Time * Measured Estimated Position of left elbow Backward * * * * * *

30. Challenge for Learning LDS on SMP 30 Time * Measured Estimated Position of left elbow Backward * * * * * * * * * *

31. Challenge for Learning LDS on SMP 31 Time * Measured Reconstructed Signal Position of left elbow Backward * * * * * * * * * *

32. Outline Completed Work –Mining Missing Value –Parallel Learning Motivation Problem Definition Proposed Method Results –Natural Motion Stitching Ongoing & Proposed Work Other Related Work 32

33. Problem Definition 33 Problem: –Given a sequence of numbers, find the best model parameters for Linear Dynamical System Goal: –Achieve ~ linear speed up on multi-core Assumption: –shared memory architecture

34. Cut-And-Stitch Intuition 34 z1z1 z1z1 y1y1 y1y1 y2y2 y2y2 z3z3 z3z3 y3y3 y3y3 z4z4 z4z4 y4y4 y4y4 z5z5 z5z5 y5y5 y5y5 y6y6 y6y6 z2z2 z2z2 z6z6 z6z6 υ2,Φ2,η2,Ψ2υ2,Φ2,η2,Ψ2 υ1,Φ1,η1,Ψ1υ1,Φ1,η1,Ψ1 z1z1 z1z1 y1y1 y1y1 y2y2 y2y2 z' 2 z2z2 z2z2 z3z3 z3z3 y3y3 y3y3 z4z4 z4z4 y4y4 y4y4 z' 4 z5z5 z5z5 y5y5 y5y5 y6y6 y6y6 z6z6 z6z6 υ3,Φ3,η3,Ψ3υ3,Φ3,η3,Ψ3 reconcile later Cut start computation without feedback from previous node Stitch

35. Cut-And-Stitch: illustration 35 Time * Measured Estimated Position of left elbow * * Cut-Forward 1

36. Cut-And-Stitch: illustration 36 Time * Measured Estimated Position of left elbow * * * * * Cut-Forward 2

37. Cut-And-Stitch: illustration 37 Time * Measured Position of left elbow * * * * * * * * Cut-Backward

38. Cut-And-Stitch: illustration 38 Time * Measured Position of left elbow * * * * * * * * Stitch * * reconciliation

39. Outline Completed Work –Mining Missing Value –Parallel Learning Motivation Problem Definition Proposed Method Results –Natural Motion Stitching Ongoing & Proposed Work Other Related Work 39

40. Near Linear Speedup 40 speedup # of processors ideal Dataset: CMU Mocap #16 mocap.cs.cmu.edu

41. No loss of accuracy 41 ~ IDENTICAL

42. Outline Completed Work –Mining Missing Value –Parallel Learning Contribution: the 1 st parallel algorithm for learning LDS –Natural Motion Stitching Ongoing & Proposed Work Other Related Work 42

43. Outline Completed Work –Mining Missing Value –Parallel Learning –Natural Motion Stitching Motivation Problem Definition Proposed Method Results Ongoing & Proposed Work Other Related Work 43

44. How to generate new natural motion? Computer Game industry –E.g. generate a smooth “goal kick” in soccer game Movie Industry –E.g. Shrek 44

45. A Database Approach Select best stitchable segments from a set of basic motion pieces and generate new natural motions 45

46. Problem Definition Given two motion-capture sequences that are to be stitched together, how can we assess the goodness of the stitching? Euclidean will fail 46 1 2 3 Best stitchable motion?

47. Outline Completed Work –Mining Missing Value –Parallel Learning –Natural Motion Stitching Motivation Problem Definition Proposed Method Results Ongoing & Proposed Work Other Related Work 47

48. Minimizing Stitching Effort Minimize the energy/effort spent by human during the transition Compute the effort using dynamics from Kalman Filters 48

49. Result 49

50. Outline Completed Work –Mining Missing Value –Parallel Learning –Natural Motion Stitching Contribution: A principled distance function for motion stitching Ongoing & Proposed Work Other Related Work 50

51. Ongoing Work Time series clustering by fingerprinting Bone length constrained occlusion filling DynaMMo+, enhanced algorithm for missing value recovery Mining multiple categorical streams 51

52. Proposed Work Multi-resolution learning for long time series Data center thermal management by mining sensor data 52

53. Roadmap: Time Series Learning and Mining 53 Motion stitching Parallel LDS Learning (cut and stitching) TS Clustering Multi-resolution learning Non Constrained Constrained Complete Missing DynaMMo DynaMMo+ Bone Length Constrained Occlusion Filling DC modeling (proposed)

54. Reference [Li 2009] DynaMMo: Mining and Summarization of Coevolving Sequences with Missing Values. KDD '09. [Li 2008c] Cut-and-stitch: efficient parallel learning of linear dynamical systems on SMPs. KDD '08. [Li 2008l] Laziness is a virtue: Motion stitching using effort minimization. Eurographics 2008. 54