1. Pre Proposal Time Series Learning completed work
Lei Li
Computer Science Department
Carnegie Mellon University
11/9/2018

2. Outline Completed Work Ongoing & Proposed Work Other Related Work
Mining w/ Missing Value
Parallel Learning
Natural Motion Stitching
Ongoing & Proposed Work
Other Related Work
Project #1: Mining Time Series with Missing Values
The most accurate mining algorithms for TS with missing value
Project #2: Parallel learning algorithm for LDS
The 1st parallel algorithm for learning LDS
Project #3: Natural Stitching of Human Motions with effort minimization
A principled distance function for motion stitching

3. Outline Completed Work Ongoing & Proposed Work Other Related Work
Mining w/ Missing Value
Motivation
Problem Definition
Proposed Method
Results
Parallel Learning
Natural Motion Stitching
Ongoing & Proposed Work
Other Related Work
Project #1: Mining Time Series with Missing Values
The most accurate mining algorithms for TS with missing value
Project #2: Parallel learning algorithm for LDS
The 1st parallel algorithm for learning LDS
Project #3: Natural Stitching of Human Motions with effort minimization
A principled distance function for motion stitching

4. Occlusion in Motion Capture
Markers on human actors
Cameras used to track the 3D positions
Duration:
93 dimensional body-local coordinates after preprocessing (31-bones)
Challenge:
Occlusions
Other general scenario:
Missing value in Sensor data: Out of battery, transmission error, etc
Unable to observe, e.g. historical/future observation
– 300 in duration
2. Putting the missing figure to front
3. Putting the high level ideas: 1. correlation 2. dynamics
4. Rework of the equation
5. Number the page 6.
From mocap.cs.cmu.edu

5. Problem Definition Given To find algorithm for:
mining hidden variables and evolving patterns
recovering missing values
compression/summarization
segmentation
Time
Marker/Sensor
blackout

6. Problem Definition (cont’)
Time
Marker/Sensor
blackout
Want the algorithms to be:
Effective
Scalable: to duration of sequences
Blackouts
Automatic: no/few parameters to be set
Effective: low reconstruction error, agreeing with human intuition (e.g. natural reconstructed motion for mocap)
Scalable: to time-duration T of the sequences.
Black-outs: It should be able to handle “black-outs”, when all markers dissappear (eg., a person running behind a wall, for a moment).
Automatic: The method should require no parameters to be set by the user.

7. Proposed Method: Intuition
Recover using
Correlation
among multiple sequences
Left Hand
Right Hand
missing

8. Proposed Method: Intuition
Recover using
Dynamics
temporal moving pattern
Left Hand
Right Hand
missing

9. Underlying Model (details)
Use Linear Dynamical Systems to model whole sequence. Z1 Z2 Z3 Z4 X1 X2 X3 X4
N(F∙z1, Λ)
N(z0, Γ)
N(G∙z3, Σ)
N(F∙z2, Λ)
N(G∙z1, Σ)
N(G∙z2, Σ)
N(G∙z4, Σ)
N(F∙z3, Λ)
N(F∙z4, Λ) …
partially
observed
observed
z1 = z0+ω0
zn+1 = F∙zn+ωn
xn = G∙zn+εn
Model parameters:
θ={z0, Γ, F, Λ, G, Σ}

10. DynaMMo Intuition
How to recover the missing values? x1 x2 x3

11. DynaMMo: How to Recover?
hidden variables
e.g. velocity,
acceleration z1 G ×
projection
matrix G x1

12. DynaMMo: How to Recover?
Transition
matrix F z1 × z2 G × G ×
projection
matrix x1 x2

13. DynaMMo: How to Recover?F F z1 × z2 × z3 G × G × x1 x2

14. DynaMMo: How to Recover?F F × × G × G × G × x1 x2 x3

15. DynaMMo: How to Recover?F F F × × × G × G × G × x1 x2 x3

16. DynaMMo Learning details
Guess Initial
Estimate Hidden
Recover Missing
Update Model 1
Fix X,
Estimate P(Z|X;):
E(zn|X;),
E(znz’n|X;)
E(znz’n+1|X;) 2
fix Z,
estimate
E(X_missing|Z;)
Using E(zn|X;),
Random
Guess model
parameters 
Fix both X and Z,
estimate new model
parameters 
argmax E[log(X,Z;)] 3

17. How to Compress Naive idea #1: use SVD
Naive idea #2: store parameters of LDS
Naive idea #3: store parameters of LDS and all hidden variables (expectation)
Proposed Methods: use check points
Fixed hop
Optimal (dynamic programming)
Near optimal (adaptive)

18. Why Not SVD? (naive idea #1)
No dynamics
Need more to compress w/ same accuracy

19. Why Not LDS? Naive idea #2: store parameters of LDS
bad reconstruction
Naive idea #3: store parameters of LDS + hidden variables
waste of space

20. DynaMMo Compression: Intuition
Original data w/ missing values
get hidden variables and model parameters
keep only a (best) portion of them
and
Same idea could be used in segmentation and forecasting

21. DynaMMo w/ Optimal Compression: Intuition
observations w/ missing values
get hidden variables and model parameters
keep only a (best) portion of them
and
Same idea could be used in segmentation and forecasting

22. How to Segment Segment by threshold on prediction error original data
reconstruction error

23. Outline Completed Work Ongoing & Proposed Work Other Related Work
Mining Missing Value
Motivation
Problem Definition
Proposed Method
Results
Parallel Learning
Natural Motion Stitching
Ongoing & Proposed Work
Other Related Work
Project #1: Mining Time Series with Missing Values
The most accurate mining algorithms for TS with missing value
Project #2: Parallel learning algorithm for LDS
The 1st parallel algorithm for learning LDS
Project #3: Natural Stitching of Human Motions with effort minimization
A principled distance function for motion stitching

24. Results – Better Missing Value Recovery
Reconstruction error
occlusion length
MSVD
Linear
Interpolation
Proposed
On human motion
Ideal

25. Results – Better Compression
Compression ratio
error
DynaMMo
w/ optimal compression
Ideal

26. Results – Segmentation
Find the transition during “running” to “stop”.
left hip
left femur
Mocap #16.08
reconstruction error

27. Results – Segmentation
Find the transition during “running” to “stop”.
left hip
run
slow
down
stop
left femur
Mocap #16.08
reconstruction error

28. Outline Completed Work Ongoing & Proposed Work Other Related Work
Mining Missing Value
Contribution: the most accurate mining algorithms for TS with missing value so far.
Parallel Learning
Natural Motion Stitching
Ongoing & Proposed Work
Other Related Work
Project #1: Mining Time Series with Missing Values
The most accurate mining algorithms for TS with missing value
Project #2: Parallel learning algorithm for LDS
The 1st parallel algorithm for learning LDS
Project #3: Natural Stitching of Human Motions with effort minimization
A principled distance function for motion stitching

29. Outline Completed Work Ongoing & Proposed Work Other Related Work
Mining Missing Value
Parallel Learning
Motivation
Problem Definition
Proposed Method
Results
Natural Motion Stitching
Ongoing & Proposed Work
Other Related Work
Project #1: Mining Time Series with Missing Values
The most accurate mining algorithms for TS with missing value
Project #2: Parallel learning algorithm for LDS
The 1st parallel algorithm for learning LDS
Project #3: Natural Stitching of Human Motions with effort minimization
A principled distance function for motion stitching

30. Challenge for Learning LDS on SMP
step 1
Position of left elbow
Measured
Estimated *
Time

31. Challenge for Learning LDS on SMP
step 2
Position of left elbow
Measured
Estimated * *
Time
Intuition: #2 may
be close to #1

32. Challenge for Learning LDS on SMP
Forward
Position of left elbow * * *
Measured *
Estimated * *
Time

33. Challenge for Learning LDS on SMP
Backward
Position of left elbow
Estimated * * *
Measured * * * *
Time

34. Challenge for Learning LDS on SMP
Backward
Position of left elbow * * * *
Estimated * *
Measured * * * *
Time *

35. Challenge for Learning LDS on SMP
Backward
Position of left elbow * *
Reconstructed
Signal * * * *
Measured * * * *
Time *

36. Outline Completed Work Ongoing & Proposed Work Other Related Work
Mining Missing Value
Parallel Learning
Motivation
Problem Definition
Proposed Method
Results
Natural Motion Stitching
Ongoing & Proposed Work
Other Related Work
Project #1: Mining Time Series with Missing Values
The most accurate mining algorithms for TS with missing value
Project #2: Parallel learning algorithm for LDS
The 1st parallel algorithm for learning LDS
Project #3: Natural Stitching of Human Motions with effort minimization
A principled distance function for motion stitching

37. Problem Definition Problem: Goal: Assumption:
Given a sequence of numbers, find the best model parameters for Linear Dynamical Systems
Goal:
Achieve ~ linear speed up on multi-core
Assumption:
shared memory architecture

38. Challenge Timeline for E-step in learning LDS 1 2 3 4 5 Step 1 Step 2

39. Proposed Method Timeline for Cut in Cut-And-Stitching Algorithm 1 2 34 5
Step 1
Step 2
Step 3
Step 4

40. Cut-And-Stitch Intuitionz1 z2 z3 z4 z5 z6
Cut
Stitch y1 y2 y3 y4 y5 y6
start computation without feedback from previous node
reconcile later
υ2,Φ2,η2,Ψ2
υ1,Φ1,η1,Ψ1 z1 y1 y2
z'2 z2 z3 y3 z4 y4
z'4 z5 y5 y6 z6
υ3,Φ3,η3,Ψ3

41. Cut-And-Stitch: illustration
Cut-Forward 1
Position of left elbow * *
Measured
Estimated *
Time

42. Cut-And-Stitch: illustration
Cut-Forward 2
Position of left elbow * * *
Measured *
Estimated * *
Time

43. Cut-And-Stitch: illustration
Cut-Backward
Position of left elbow * * * * *
Measured * * * *
Time

44. Cut-And-Stitch: illustration
Position of left elbow * * * * * *
Measured * * * *
Time *
reconciliation
reconciliation

45. Outline Completed Work Ongoing & Proposed Work Other Related Work
Mining Missing Value
Parallel Learning
Motivation
Problem Definition
Proposed Method
Results
Natural Motion Stitching
Ongoing & Proposed Work
Other Related Work
Project #1: Mining Time Series with Missing Values
The most accurate mining algorithms for TS with missing value
Project #2: Parallel learning algorithm for LDS
The 1st parallel algorithm for learning LDS
Project #3: Natural Stitching of Human Motions with effort minimization
A principled distance function for motion stitching

46. Near Linear Speedup speedup ideal # of processors Dataset:
CMU Mocap #16
mocap.cs.cmu.edu

47. No loss of accuracy
~ IDENTICAL

48. Outline Completed Work Ongoing & Proposed Work Other Related Work
Mining Missing Value
Parallel Learning
Contribution: the 1st parallel algorithm for learning LDS
Natural Motion Stitching
Ongoing & Proposed Work
Other Related Work
Project #1: Mining Time Series with Missing Values
The most accurate mining algorithms for TS with missing value
Project #2: Parallel learning algorithm for LDS
The 1st parallel algorithm for learning LDS
Project #3: Natural Stitching of Human Motions with effort minimization
A principled distance function for motion stitching

49. Outline Completed Work Ongoing & Proposed Work Other Related Work
Mining Missing Value
Parallel Learning
Natural Motion Stitching
Motivation
Problem Definition
Proposed Method
Results
Ongoing & Proposed Work
Other Related Work
Project #1: Mining Time Series with Missing Values
The most accurate mining algorithms for TS with missing value
Project #2: Parallel learning algorithm for LDS
The 1st parallel algorithm for learning LDS
Project #3: Natural Stitching of Human Motions with effort minimization
A principled distance function for motion stitching

50. How to generate new natural motion?
Computer Game industry
E.g. generate a smooth “goal kick” in soccer game
Movie Industry
E.g. Shrek

51. A Database Approach
Select best stitchable segments from a set of basic motion pieces and generate new natural motions

52. Problem Definition
Given two motion-capture sequences that are to be stitched together, how can we assess the goodness of the stitching?
Euclidean will fail 1 2 3
Best stitchable motion?

53. Outline Completed Work Ongoing & Proposed Work Other Related Work
Mining Missing Value
Parallel Learning
Natural Motion Stitching
Motivation
Problem Definition
Proposed Method
Results
Ongoing & Proposed Work
Other Related Work
Project #1: Mining Time Series with Missing Values
The most accurate mining algorithms for TS with missing value
Project #2: Parallel learning algorithm for LDS
The 1st parallel algorithm for learning LDS
Project #3: Natural Stitching of Human Motions with effort minimization
A principled distance function for motion stitching

54. Minimizing Stitching Effort
Minimize the energy/effort spent by human during the transition
Compute the effort using dynamics from Kalman Filters
Remind problem definition

55. Result – Synthetic Transition
Low effort transition
High effort transition

56. Result – Mocap Walk: make a left turn

57. Outline Completed Work Ongoing & Proposed Work Other Related Work
Mining Missing Value
Parallel Learning
Natural Motion Stitching
Contribution: A principled distance function for motion stitching
Ongoing & Proposed Work
Other Related Work
Project #1: Mining Time Series with Missing Values
The most accurate mining algorithms for TS with missing value
Project #2: Parallel learning algorithm for LDS
The 1st parallel algorithm for learning LDS
Project #3: Natural Stitching of Human Motions with effort minimization
A principled distance function for motion stitching

58. Roadmap: Time Series Learning and Mining
Complete
Missing
Motion stitching
Parallel LDS Learning (cut and stitching)
TS Clustering
Multi-resolution learning
DynaMMo
DynaMMo+
Non Constrained
Bone Length Constrained Occlusion Filling
DC modeling (proposed)
Constrained

59. Ongoing Work Time series clustering by fingerprinting
Bone length constrained occlusion filling
DynaMMo+, enhanced algorithm for missing value recovery
Mining multiple categorical streams

60. C1: Time Series Clustering by Fingerprinting
Problem:
cluster time sequences with harmonics
Idea:
Extract features (fingerprints) and compare

61. C2: Bone Length Constrained Occlusion Recovery
Problem:
recover the occlusions in human mocap data
Idea:
use body skeleton constraints

62. C3: DynaMMo+ Problem & Goal:
enhanced missing value recovery for time series

63. C4: Mining Categorical Streams
Problem:
mine pattern in multiple categorical streams (e.g. web click stream)
find anomalies

64. Proposed Work Multi-resolution learning for long time series
Data center thermal management by mining sensor data

65. F1: Data Center Thermal Management by Mining Sensor Data
Problem:
How to save energy in data center?
Specifically, which machines to turn on/off based on sensor measuring in data center?

66. F2: Multi-resolution Learning for Long Time Series
Problem:
find patterns in long coevolving time series (e.g. tropical sea surface temperature over past hundred years)

67. Reference
[Li 2009] DynaMMo: Mining and Summarization of Coevolving Sequences with Missing Values. KDD '09.
[Li 2008c] Cut-and-stitch: efficient parallel learning of linear dynamical systems on SMPs. KDD '08.
[Li 2008l] Laziness is a virtue: Motion stitching using effort minimization. Eurographics 2008.