1. Introduction to Data-driven Animation Jinxiang Chai Computer Science and Engineering Texas A&M University

2. Data-driven animation - Motion graphs - Motion interpolations - Statistical motion synthesis Outline

3. Motion capture data are easy to capture But we cannot capture all kinds of motion variations - different subjects - different styles - different emotions Key idea: reuse prerecorded motion data to achieve new goals! Motivations for Data-driven Approaches

4. Data-driven Animation Goal: convert motion data into a usable form. Can we automate this? –Must preserve realism and provide control Motion model User specifications Motion Motion processing Motion data

5. Data-driven animation - Motion graphs - Motion interpolations - Statistical motion synthesis Outline

6. Motion Graphs: Key Ideas Given: lots of prerecorded motion clips Concatenate them to create new motions!

7. Motion Graphs: Key Ideas Given: lots of prerecorded motion clips Concatenate them to create new motions!

8. Maze

9. Motion Concatenation Motion capture regionVirtual environment Obstacles Sketched path

10. Motion Concatenation Motion capture regionVirtual environment

11. Unstructured Input Data A number of motion clips Each clip contains many frames Each frame represents a pose A number of motion clips Each clip contains many frames Each frame represents a pose

12. Unstructured Input Data Connecting transition Between similar frames Connecting transition Between similar frames

13. Graph Construction

14. Building Motion Graphs So how can we find transition points between motion clips?

15. Building Motion Graphs - Every pair of frames has a distance. - Transitions are local minima below a threshold. Motion 2 Frames Motion 1 Frames

16. Finding Similar Frames Need derivatives (velocity, acceleration, etc.) Compare motion in joint angle space or 3d point space? Must account for coordinate invariance –Different camera ≠ different motion!

17. Distance Metric For more detail, refer to [Kovar and Gleicher, Lee et al]

18. Finding Transition Points Transition thresholds control quality vs. flexibility tradeoff. Threshold = 0 cmThreshold = 8 cmThreshold = 16 cm

19. Structures of Motion Graphs Motion data structure: a graph of frames/poses Avoid dead-ends: finding strongly connected components avoid transition to dissimilar contact state Contact states: avoid transition to dissimilar contact state

20. Interacting with Motion Graphs So given a motion graph, how can we generate an animation sequence?

21. Interacting with Motion Graphs So given a motion graph, how can we generate an animation sequence? - Random graph walk: Any sequence of edges is a motion!

22. Using Motion Graphs How can we control synthesized motions (e.g., moving from point A to point B, speeds, walking directions)? - Graph search: Find graph walks that minimize a cost function.

23. Path Synthesis Goal: extract motion that follows a path. User’s path ( ) Motion’s path ( ) Minimize

24. Motion Control Goal: extract motion (M) that satisfied constraints (C) specified by the user Minimize

25. Results See videos [click here]!here

26. Discussion Pros: + Fully automatic: work on unstructured data + High-quality animation: motion concatenations + Easy to control: graph search Cons - Poor generalization: cannot produce new poses - Control accuracy: cannot generalize new poses - Not compact: needs to retain original mocap data - Scalability

27. Data-driven animation - Motion graphs - Motion interpolations - Statistical motion synthesis Outline

28. Motion Interpolations: Key Ideas Given: lots of prerecorded motion clips Interpolating motions to achieve new goals!

29. Motion Interpolations: Key Ideas In research: more than decades [e.g., Rose et al. 98] In games for a long time - Interpolating motions needs build correspondences between motion examples - Thus, motion interpolations require structurally similar motion examples!

30. Canonical timeline t Time warping functions w(t) t Motion Decomposition

31. Canonical timeline t Time warping functions w(t) t Reference motion Motion Decomposition

32. Canonical timeline t Time warping functions w(t) t Contact Transitions Motion Decomposition

33. Canonical timeline t Time warping functions w(t) Motion 1 t Motion Decomposition

34. Canonical timeline t Time warping functions w(t) t Motion Decomposition

35. Canonical timeline t Time warping functions w(t) t Motion Decomposition

36. Canonical timeline t Time warping functions w(t) t Motion Decomposition Using dynamic time warping!

37. Canonical timeline t Time warping functions w(t) t Motion Decomposition

38. Canonical timeline t Time warping functions w(t) Motion 2 t Motion Decomposition

39. Canonical timeline t Time warping functions w(t) t Motion Decomposition

40. Canonical timeline t Time warping functions w(t) t Motion Decomposition

41. Canonical timeline t Time warping functions w(t) t Motion Decomposition

42. Canonical timeline t Time warping functions w(t) t Motion Decomposition

43. Canonical timeline t Time warping functions w(t) t Motion Decomposition

44. Motion Representation Registered motions Time warping functions Contact Transitions

45. Motion Annotation Preprocessed motions m i

46. Motion Annotation Preprocessed motions m i For each motion m i, we annotate the motion with control parameters s i - such as walking speed, direction, step size, kicking directions and positions, etc.

47. Motion Annotation Preprocessed motions m i Motion space: m Control parameter space: s

48. Motion Interpolations and Control Preprocessed motions m i Motion space: m Control parameter space: s How can we generate an animation that achieves the goals specified c* by the user?

49. Motion Interpolations and Control Preprocessed motions m i Motion space: m Control parameter space: s How can we generate an animation that achieves the goals specified c* by the user? Scattered data interpolation

50. Scatter Data Interpolations Preprocessed motions m i Motion space: m Control parameter space: s How can we generate an animation that achieves the goals specified c* by the user? Scattered data interpolation

51. Scattered Data Interpolations Motion space: m Control parameter space: s Many techniques for scattered data interp.: - Local interpolation/regression [Kovar and Gleicher 04] - Radial basis functions [Rose et al 98] - Gaussian processes [Mukai and Kuriyama 05]

52. Results Geostatistical motion interpolation [Mukai and Kuriyama 05] - check youtube video [click here]here

53. Discussions Pros: + high-quality animation + good generalization: motion interpolation and extrapolation + particularly suitable for high-level motion control Cons - all examples must be structurally similar. - not compact: needs to retain original mocap data - not suitable for detailed kinematic motion control such as such key frames - lack of planning schemes for task level control such as moving from one point to another.

54. Data-driven animation - Motion graphs - Motion interpolations - Statistical motion synthesis Outline

55. Bayesian Motion Synthesis Goal: Find the most likely motion x from control inputs c specified by the user 

56. Bayesian Motion Synthesis Goal: Find the most likely motion x from control inputs c specified by the user 

57. Bayesian Motion Synthesis Goal: Find the most likely motion x from control inputs c specified by the user  Likelihood: How well motion matches control input? Motion priors: How natural motion is?

58. Human motion database Human motion analysis Motion optimization Human motion prior User-defined constraints Generate natural human motion from a small set of user-defined constraints Statistical Motion Synthesis

59. Statistical Dynamic Model Character pose Low-dimensional pose space y t = C x t + D Human motion database Human motion analysis Statistical dynamic model x t = A 1 x t-1 +…+ A m x t-m + B u t Control input Temporal prediction

60. Complexity of Statistical Dynamic Model Character pose Low-dimensional pose space y t = C x t + D Human motion database Human motion analysis Statistical dynamic model x t = A 1 x t-1 +…+ A m x t-m + B u t Control input Temporal prediction dim(u t ) dim(x t )

61. Statistical Dynamic Model Learning Human motion database Human motion analysis Character pose Low-dimensional pose space y t = C x t + D x t = A 1 x t-1 +…+ A m x t-m + B u t Control input Temporal prediction Dynamic model matrices: A 1, …,A m, B, C, D

62. Statistical Dynamic Model Learning Human motion database Human motion analysis Dynamic model matrices: A 1, …,A m, B, C, D Dynamic system order: m, dimensionality of x t and u t Character pose Low-dimensional pose space y t = C x t + D x t = A 1 x t-1 +…+ A m x t-m + B u t Control input Temporal prediction

63. Reconstruction error Full-body motion data X t = A 1 X t-1 +…+A m X t-m +B u t Statistical linear dynamic model: m = 3, dim(u t ) = 4, error = 0.7 deg Y t = C X t + D

64. Generate natural motion from a small set of user-defined constraints Overview: Offline Animation Control Human motion database Human motion analysis Motion optimization Human motion prior User-defined constraints

65. Constrained Motion Optimization Human motion database Human motion analysis Motion optimization Human motion prior User-defined constraints Generate natural motion from a small set of user-defined constraints

66. Constrained Motion Optimization Motion optimization Human motion prior User-defined constraints Smoothness term

67. User-defined constraints: Objective function: Motion priorMotion smoothness Sequential Quadratic Programming Constrained Motion Optimization

68. Results Click herehere

69. Discussions Pros: + good generalization and accurate motion control due to the use of statistical models + compact: only needs to keep model parameters + suitable for kinematic motion control Cons - often need to specify contact constraints across the entire animation. - does not support task-level control (e.g., move from one point to another) - better models are needed to match the synthesis quality of motion graphs

70. Summary The use of prerecorded motion data allows us to create high-quality controllable animation for human characters Ideal data-driven animation techniques - High-quality: realistic animation without noticeable visual artifacts - control accuracy: often require strong generalizability to achieve new tasks. - control flexibility: support motion control in both kinematics and task/behavior level. - scalability: scale up well to huge and heterogeneous datasets. - compact: demand small or moderate memory sizes.