2. Animation From Motion Capture Motion Capture Assisted Animation: Texturing and Synthesis Kathy Pullen Chris Bregler Motion Capture Assisted Animation: Texturing and Synthesis Kathy Pullen Chris Bregler

3. Related Work: Animation With Style M. Brand and A. Hertzmann. Proc. SIGGRAPH 2000, pp 183-192 D. Chi, M. Costa, L. Zhao and N. Badler. Proc. SIGGRAPH 2000, pp 173-182 M. Gleicher. 1997 Symposium on Interactive 3D Graphics, pp 139-148 J. Hodgins, W. L. Wooten, D. C. Broogan and J. F. O’Brien. Animating Human athletics. Proc. SIGGRAPH 1995, PP 229-238 K. Perlin and A. Goldberg. Proc. SIGGRAPH 1996, PP 205-216 Z. Popovic and A. Witkin. Proc. SIGGRAPH 1999, pp 159-168 A. Witkin and M. Kass. Computer Graphics, 22:159-168, 1988

4. Related Work: Signal Processing A. Bruderlin and L. Williams. Proc. SIGGRAPH 1995, pp 97-104 J. S. De Bodnet. Proc. SIGGRAPH 1999, pp 21-28 D. J. Heeger and J. R. Bergen. Proc. SIGGRAPH 1995, pp 229-238 Z. Popovic and A. Witkin. Proc. SIGGRAPH 1999, pp 159-168 U. Unuma, K. Anjyo and R. Tekeuchi. Proc. SIGGRAPH 1995, pp 91-96 A. Witkin and Z. Popovic. Proc. SIGGRAPH 1995, PP 105-108

5. Related Work: Animation from Mocap O. Arikan and D. A. Forsyth. Interactive motion generation from examples. Proc. SIGGRAPH 2002 L. Kovar, M. Gleicher, and F. Pighin. Motion Graphs. Proc. SIGGRAPH 2002 J. Lee, J. Chai, P. S. A. Reitsma, J. K. Hodgins, and N. S. Pollard. Interactive control of avatars animated with human motion data. Proc. SIGGRAPH 2002 Y. Li, T. Wang, and H. Shum. Motion Texture: A two-level statistical model for character motion synthesis

6. Goal: Motion Capture Assisted Animation Create a method that allows an artist low-level control of the motion Create a method that allows an artist low-level control of the motion Combine the strengths of keyframe animation with those of mocap Combine the strengths of keyframe animation with those of mocap Create a method that allows an artist low-level control of the motion Create a method that allows an artist low-level control of the motion Combine the strengths of keyframe animation with those of mocap Combine the strengths of keyframe animation with those of mocap

7. Goal: Motion Capture Assisted Animation “Sketch” an animation by keyframing

8. Goal: Motion Capture Assisted Animation “Sketch” an animation by keyframing Animate only a few degrees of freedom Animate only a few degrees of freedom “Sketch” an animation by keyframing Animate only a few degrees of freedom Animate only a few degrees of freedom

9. Goal: Motion Capture Assisted Animation “Sketch” an animation by keyframing Animate only a few degrees of freedom Animate only a few degrees of freedom Set few keyframes Set few keyframes “Sketch” an animation by keyframing Animate only a few degrees of freedom Animate only a few degrees of freedom Set few keyframes Set few keyframes

10. Goal: Motion Capture Assisted Animation “Sketch” an animation by keyframing Animate only a few degrees of freedom Animate only a few degrees of freedom Set few keyframes Set few keyframes “Sketch” an animation by keyframing Animate only a few degrees of freedom Animate only a few degrees of freedom Set few keyframes Set few keyframes “Enhance” the result with mocap data

11. Goal: Motion Capture Assisted Animation “Sketch” an animation by keyframing Animate only a few degrees of freedom Animate only a few degrees of freedom Set few keyframes Set few keyframes “Sketch” an animation by keyframing Animate only a few degrees of freedom Animate only a few degrees of freedom Set few keyframes Set few keyframes “Enhance” the result with mocap data Synthesize missing degrees of freedom Synthesize missing degrees of freedom “Enhance” the result with mocap data Synthesize missing degrees of freedom Synthesize missing degrees of freedom

12. Goal: Motion Capture Assisted Animation “Sketch” an animation by keyframing Animate only a few degrees of freedom Animate only a few degrees of freedom Set few keyframes Set few keyframes “Sketch” an animation by keyframing Animate only a few degrees of freedom Animate only a few degrees of freedom Set few keyframes Set few keyframes “Enhance” the result with mocap data Synthesize missing degrees of freedom Synthesize missing degrees of freedom Texture keyframed degrees of freedom Texture keyframed degrees of freedom “Enhance” the result with mocap data Synthesize missing degrees of freedom Synthesize missing degrees of freedom Texture keyframed degrees of freedom Texture keyframed degrees of freedom

13. Goal: Motion Capture Assisted Animation Blue = Keyframed Purple = Textured/Synthesized

14. What is a Motion Texture? Every individual’s movement is unique “Motion texture” was coined by Ken Perlin Every individual’s movement is unique “Motion texture” was coined by Ken Perlin

15. What is a Motion Texture? Every individual’s movement is unique “Motion texture” was coined by Ken Perlin Dance! Acrobatics! Dance! Acrobatics! Every individual’s movement is unique “Motion texture” was coined by Ken Perlin Dance! Acrobatics! Dance! Acrobatics!

16. What is a Motion Texture? Every individual’s movement is unique “Motion texture” was coined by Ken Perlin Dance! Acrobatics! Dance! Acrobatics! Everyone walks, but not the same way Everyone walks, but not the same way Every individual’s movement is unique “Motion texture” was coined by Ken Perlin Dance! Acrobatics! Dance! Acrobatics! Everyone walks, but not the same way Everyone walks, but not the same way

17. Animating With Motion Texture Every individual’s movement is unique Synthetic motion should capture the texture Synthetic motion should capture the texture Every individual’s movement is unique Synthetic motion should capture the texture Synthetic motion should capture the texture

18. Animating With Motion Texture Every individual’s movement is unique Synthetic motion should capture the texture Synthetic motion should capture the texture To “texture” means to add style to a pre- existing motion To “texture” means to add style to a pre- existing motion Every individual’s movement is unique Synthetic motion should capture the texture Synthetic motion should capture the texture To “texture” means to add style to a pre- existing motion To “texture” means to add style to a pre- existing motion

19. Animating With Motion Texture Every individual’s movement is unique Synthetic motion should capture the texture Synthetic motion should capture the texture To “texture” means to add style to a pre- existing motion To “texture” means to add style to a pre- existing motion Technically, texturing is a special case of synthesis Technically, texturing is a special case of synthesis Every individual’s movement is unique Synthetic motion should capture the texture Synthetic motion should capture the texture To “texture” means to add style to a pre- existing motion To “texture” means to add style to a pre- existing motion Technically, texturing is a special case of synthesis Technically, texturing is a special case of synthesis

20. Keyframing vs. Mocap Keyframing Mocap DisadvantagesAdvantages Control

21. Keyframing vs. Mocap Keyframing Mocap DisadvantagesAdvantages Control Intuitive

22. Keyframing vs. Mocap Keyframing Mocap DisadvantagesAdvantages Control Intuitive Detail hard

23. Keyframing vs. Mocap Keyframing Mocap DisadvantagesAdvantages Control Intuitive Detail hard Many DOF

24. Keyframing vs. Mocap Keyframing Mocap DisadvantagesAdvantages Control Intuitive Detail hard Many DOF Detail easy

25. Keyframing vs. Mocap Keyframing Mocap DisadvantagesAdvantages Control Intuitive Detail hard Many DOF Detail easy All DOF

26. Keyframing vs. Mocap Keyframing Mocap DisadvantagesAdvantages Control Intuitive No control Detail hard Many DOF Detail easy All DOF

27. Keyframing vs. Mocap Keyframing Mocap DisadvantagesAdvantages Control Intuitive No control Not intuitive Detail hard Many DOF Detail easy All DOF

28. How an Animator Works A few degrees of freedom at first A few degrees of freedom at first Not in detail Not in detail Fill in detail with more keyframes later Fill in detail with more keyframes later A few degrees of freedom at first A few degrees of freedom at first Not in detail Not in detail Fill in detail with more keyframes later Fill in detail with more keyframes later

29. The Method in Words Choose degrees of freedom to drive the animation Compare these degrees of freedom from the keyframed data to mocap Find similar regions Look at what the rest of the body is doing in those regions Put that data onto the keyframed animation Choose degrees of freedom to drive the animation Compare these degrees of freedom from the keyframed data to mocap Find similar regions Look at what the rest of the body is doing in those regions Put that data onto the keyframed animation

30. Before Beginning: Choose Matching Angles Left Hip x Left Hip y Left Hip z Left Knee x Left Knee y Left Knee z Left Ankle x Left Ankle y Left Ankle z Left Ball x Left Ball y Left Ball z Right Hip x Right Hip y Right Hip z Right Knee x Right Knee y Right Knee z Right Ankle x Right Ankle y Right Ankle z Right Ball x Right Ball y Right Ball z Root x trans Root y trans Root z trans Root x rot Root y rot Root z rot Spine1 x Spine1 y Spine1 z Spine2 x Spine2 y Spine2 z Spine3 x Spine3 y Spine3 z Neck x Neck y Neck z Head x Head y Head z Head Aim x Head Aim y Head Aim z Left Clavicle x Left Clavicle y Left Clavicle z Left Shoulder x Left Shoulder y Left Shoulder z Left Elbow x Left Elbow y Left Elbow z Left Wrist x Left Wrist y Left Wrist z Right Clavicle x Right Clavicle y Right Clavicle z Right Shoulder x Right Shoulder y Right Shoulder z Right Elbow x Right Elbow y Right Elbow z Right Wrist x Right Wrist y Right Wrist z Time

31. Matching Angles Drive the Synthesis Left Hip x Left Hip y Left Hip z Left Knee x Left Knee y Left Knee z Left Ankle x Left Ankle y Left Ankle z Left Ball x Left Ball y Left Ball z Right Hip x Right Hip y Right Hip z Right Knee x Right Knee y Right Knee z Right Ankle x Right Ankle y Right Ankle z Right Ball x Right Ball y Right Ball z Root x trans Root y trans Root z trans Root x rot Root y rot Root z rot Spine1 x Spine1 y Spine1 z Spine2 x Spine2 y Spine2 z Spine3 x Spine3 y Spine3 z Neck x Neck y Neck z Head x Head y Head z Head Aim x Head Aim y Head Aim z Left Clavicle x Left Clavicle y Left Clavicle z Left Shoulder x Left Shoulder y Left Shoulder z Left Elbow x Left Elbow y Left Elbow z Left Wrist x Left Wrist y Left Wrist z Right Clavicle x Right Clavicle y Right Clavicle z Right Shoulder x Right Shoulder y Right Shoulder z Right Elbow x Right Elbow y Right Elbow z Right Wrist x Right Wrist y Right Wrist z Time

32. Motion Capture Data Left Hip x Left Hip y Left Hip z Left Knee x Left Knee y Left Knee z Left Ankle x Left Ankle y Left Ankle z Left Ball x Left Ball y Left Ball z Right Hip x Right Hip y Right Hip z Right Knee x Right Knee y Right Knee z Right Ankle x Right Ankle y Right Ankle z Right Ball x Right Ball y Right Ball z Root x trans Root y trans Root z trans Root x rot Root y rot Root z rot Spine1 x Spine1 y Spine1 z Spine2 x Spine2 y Spine2 z Spine3 x Spine3 y Spine3 z Neck x Neck y Neck z Head x Head y Head z Head Aim x Head Aim y Head Aim z Left Clavicle x Left Clavicle y Left Clavicle z Left Shoulder x Left Shoulder y Left Shoulder z Left Elbow x Left Elbow y Left Elbow z Left Wrist x Left Wrist y Left Wrist z Right Clavicle x Right Clavicle y Right Clavicle z Right Shoulder x Right Shoulder y Right Shoulder z Right Elbow x Right Elbow y Right Elbow z Right Wrist x Right Wrist y Right Wrist z Time

33. OverviewOverview Steps in texture/synthesis method Frequency analysis Frequency analysis Matching Matching Path finding Path finding Joining Joining Steps in texture/synthesis method Frequency analysis Frequency analysis Matching Matching Path finding Path finding Joining Joining

34. In the following series of slides: Hip angle = matching angle Spine angle = angle being synthesized In the following series of slides: Hip angle = matching angle Spine angle = angle being synthesized Example

35. Frequency Analysis: Break into Bands

36. Frequency Time Band-pass decomposition of matching angles Keyframed DataMotion Capture Data Frequency Analysis

37. Frequency Time Keyframed DataMotion Capture Data Chosen low frequency band Frequency Analysis

38. Keyframed DataMotion Capture Data Hip angle data (a matching angle) Frequency Time Chosen Low Frequency Band

39. Keyframed DataMotion Capture Data Frequency Time Making Fragments Break where first derivative changes sign

40. Keyframed DataMotion Capture Data Frequency Time Making Fragments Step through fragments one by one

41. Matching Keyframed Fragment

42. Matching Keyframed Fragment Motion Capture Data

43. Matching Keyframed Fragment Motion Capture Data

44. Matching Compare to all motion capture fragments Angle in degrees Time Keyframed Mocap

45. Matching Resample mocap fragments to be same length Angle in degrees Time Keyframed Mocap

46. Matching Keep the K closest matches Angle in degrees Time Keyframed Mocap

47. Matching Keyframed Fragment Motion Capture Data

48. Matching Keyframed Fragment Motion Capture Data Close Matches

49. Matching Hip Angle (Matching Angle) Spine Angle (For Synthesis)

50. Matching and Synthesis Low frequency hip angle data (a matching angle) Spine angle data to be synthesized

51. Matching and Synthesis Low frequency hip angle data (a matching angle) Spine angle data to be synthesized

52. Matching and Synthesis Low frequency hip angle data (a matching angle) Spine angle data to be synthesized

53. Matching and Synthesis Low frequency hip angle data (a matching angle) Spine angle data to be synthesized

54. Matching and Synthesis Low frequency hip angle data (a matching angle) Spine angle data to be synthesized

55. Matching and Synthesis Low frequency hip angle data (a matching angle) Spine angle data to be synthesized

56. Possible Synthetic Spine Angle Data Angle in degrees Time

57. Path Finding Angle in degrees Time

58. Path Finding Angle in degrees Time

59. Path Finding Angle in degrees Time

60. Path Finding Angle in degrees Time

61. Path Finding Angle in degrees Time

62. Joining Angle in degrees Time

63. Texturing Band-pass decomposition of keyframed data Frequency Time

64. Texturing Synthesize upper frequency bands Frequency Time

65. Keyframed Sketch

66. Motion Capture Data Two different styles of walk

67. Enhanced Animation Upper body is synthesized Lower body is textured

68. Keyframed Sketch With More Detail

69. Motion Capture Data

70. Textured Animation Blue = Keyframed Purple = Textured

71. Lazy Keyframed Sketch

72. Enhanced Animation Blue = Keyframed Purple = Textured/Synthesized

73. Different Paths

74. Summary of the Method Keyframed data Mocap Data Keyframed Data Mocap DataPossible Synthetic Data Matching Angles Sketch + Mocap Frequency Analysis Matching Path FindingJoining Enhanced Animation

75. Conclusions and Applications For more automatic generation with high level control, the previous methods are more appropriate For more automatic generation with high level control, the previous methods are more appropriate Appropriate for an artist interested in a very particular style of motion Appropriate for an artist interested in a very particular style of motion The artist may have a relatively small motion capture set of that style The artist may have a relatively small motion capture set of that style The artist may want precise control over parts of the motion The artist may want precise control over parts of the motion For more automatic generation with high level control, the previous methods are more appropriate For more automatic generation with high level control, the previous methods are more appropriate Appropriate for an artist interested in a very particular style of motion Appropriate for an artist interested in a very particular style of motion The artist may have a relatively small motion capture set of that style The artist may have a relatively small motion capture set of that style The artist may want precise control over parts of the motion The artist may want precise control over parts of the motion

76. For more info... http://graphics.stanford.edu/~pullen Special Thanks to: Reardon Steele, Electronic Arts

77. Choices the Animator Must Make 1. Which DOF to use as matching angles 2. Which DOF to texture, which to synthesize 3. Which frequency band to use in matching 4. How many frequency bands to use in texturing 5. How many matches to keep 1. Which DOF to use as matching angles 2. Which DOF to texture, which to synthesize 3. Which frequency band to use in matching 4. How many frequency bands to use in texturing 5. How many matches to keep

78. Conclusions and Further Work Direct incorporation of hard constraints Direct incorporation of hard constraints Fundamental units of motion Fundamental units of motion Direct incorporation of hard constraints Direct incorporation of hard constraints Fundamental units of motion Fundamental units of motion

79. Keyframe Data vs. Motion Capture Data

80. Texturing Synthesize upper frequency bands

81. Angle in degrees Time Matching Keyframed

82. Enhancing Animations: Texturing and Synthesis Keyframed

83. Enhancing Animations: Texturing and Synthesis Keyframed Textured

84. Enhancing Animations: Texturing and Synthesis Keyframed Textured Not keyframed

85. Enhancing Animations: Texturing and Synthesis Keyframed Textured Synthesized Not keyframed

86. Keyframing vs. Mocap Keyframing Mocap DisadvantagesAdvantages Control