1. Motion Capture Motion Capture

2. Motion Capture Motion capture, Motion Tracking or Mocap, is a technique of digitally recording movements for entertainment, sports, medical applications… Way of using a physical device to control animation

3. Motion Capture Technology−−magnetic Electromechanical transducers -Polhemus Fastrak Limited range/resolution Tethered Sensitive to metal No identification problem 6 dof information Realtime Low frequency: 30 to 120 Hz Few markers: 10−20

4. Motion Capture Technology−− Mechanical Capture Systems Any environment Measures joint angles Restricts the motion

5. Motion Capture Technology−− Optical Capture Systems Place markers on the object or on the actor Passive reflection Active light sources Cameras determine marker positions (Raw Point Data) 8 or more cameras Restricted volume High Frequency (240Hz) Occlusions

6. Motion Capture Technology Issues: Resolution/range of motion Calibration Occlusion/correspondence Accuracy Marker movement/placement Sensor noise Skew in measurement time Occlusion/correspondence

7. Motion Capture -- Optical motion capture process 1. 1. Find the skeleton dimensions and exact marker positions on the body 2. 2. Perform a motion trial 3. 3. Compute marker positions from camera images 4. 4. Identify and uniquely label markers 5. 5. Calculate joint angles from markers paths

8. Motion Capture -- Optical motion capture process 1. 1. Find the skeleton dimensions and exact marker positions on the body

9. Motion Capture -- Optical motion capture process   Automatic Calibration Design Goals: Fully automatic Any skeleton Accurate

10. Motion Capture -- Optical motion capture process   Input Generic Skeleton Actor’s kinematics structure, and rough handle positions  Input Calibration Data Initial path data that exercises all of the subject’s DOFs

11. Motion Capture -- Optical motion capture process Independent Variables DOFs Bone lengths Handle offsets Global scale

12. Motion Capture -- Optical motion capture process   Marker Identification   At each frame, motion capture gives us a set of points   We would like something more intuitive

13. Motion Capture -- Optical motion capture process   Marker Identification   Making sense of raw data…

14. Motion Capture -- Optical motion capture process   Calculate joint angles from marker paths   Inverse Kinematics problem (IK) 1. Create a handle on body - position or orientation 2. Pull on the handle 3. IK figures out how joint angles should change Inputs: An articulated skeleton with handles. Desired positions for handles. Outputs: Joint angles that move handles to desired positions.

15. Motion Capture Inverse Kinematics problem can be solved using a non-linear optimization technique, which seeks to minimize the deviation between the recorded data and the hierarchical model.

16. Motion Capture The minimized fitness function can be defined as F(  ) =  p (P j –  P j ) 2 +  O0 (O 0,j –  O 0,j ) 2 +  O1 (O 1,j –  O 1,j ) 2 +  c C j 2 where  is the set of joint angles for the set of joints J. P j is the position of the jth joint given and  P j is the recorded joint position from the capture phase. O 0,j and O 1,j are two vectors defining the orientation of the joint with  O 0,j and  O 1,j being the recorded orientations. Two vectors are used because, together with their cross product, they will form a right-handed coordinate system. The quantities  p,  O0, and  O1 are scalar weights that can be tuned to achieve better results. Additionally, a joint angle constraint term, C j in the form of a penalty function is employed. Joint angle constraints for humans have been measured and can be found in the biomechanical literature

17. BIOMECHANICAL MOVEMENTS There is a huge amount literature devoted simulating human movements. See, for instance, the seminal paper by A. Witkin and M. Kass, Spacetime Constraints, in proceedings of SIGGRAPH, 1988, 159- 168.[1] [2] and D. C. Brogan, K. P. Granata, and P. N. Sheth, Spacetime constraints for biomechanical movements, in proceedings Applied Modelling and Simulation, 2002 [2]

18. BIOMECHANICAL MOVEMENTS The authors of the paper [1] use a variant of sequential quadratic programming to perform an iterative, gradient-based optimization. The user provides the algorithm with an initial configuration, a final configuration, and initial guesses to populate the remaining n-2 character configurations and n joint torques. We can represent the algorithm’s elements symbolically by using S to represent the state vector, C to represent the vector of spatial and dynamical constraint violations, and R( S) to represent the objective function. We must solve for S such that C =0 and R( S) is minimized. Provided an initial S, three elements are computed: the constraint violations, C, the Jacobian of the constraint functions,, and the Hessian of the objective function,.

19. BIOMECHANICAL MOVEMENTS A two-step process first solves for a local change in the state vector, S α, that minimizes the objective function without consideration of constraint violations. To minimize the objective function, the deriviate R'(S) = 0 is used in [2]. A second-order Taylor series expansion is used to approximate R' at S :, Letting S α = X – S, we can solve. Although we know R(S) is minimized at S + S α, spatial and dynamical constraints may be violated.

20. BIOMECHANICAL MOVEMENTS We must find a second change in the state vector, S β, that preserves the minimization of R while eliminating any constraint violations at S + S α. We pursue a second step that projects S α to the null space of the constraint Jacobian: J (S α + S β ) + C = 0 Note both the constraint vector, C, and the Jacobian, J, are evaluated at S + S α. Solving for S β in: - C = J (S α + S β ) drives C to zero. The new value of S is incremented by (S α + S β ) The algorithm computes new state vectors until the reduction in the objective requires violating constraints.

21. Biomechanics To better understand human movements, biomechanical models must be developed that accurately describe human physiology and control strategies. The standardized definition of the spacetime constraints state vector, constraints vector, and objective function facilitates the reconfiguration and reuse of biomechanical models.

22. Biomechanics Experiment To implement a simulation that determines the movement trajectory simultaneously with the optimum activation torques. The simulated walker is placed on a downward-sloped plane such that if it were passively simulated (with no internal torque sources) it would settle into a steady gate. The movement trajectories of the passive- dynamic walker require zero-torque activation and are known to be the optimal trajectories for actively powered walkers. A bipedal walker. Lumped masses are positioned at the hip (m H ) and on each leg (m L ). The spacetime- constraints walker applies torques at the stance foot (A) and at the hip (B).

23. Biomechanics Experiment The (a passive-dynamic) model a planar knee-less walker included two legs of mass m L, joined by a revolute joint located at the hip with point mass, m H The walker moves along a plane of slope γ with respect to horizontal. A time- dependent vector θ = [θ S, θ N ] T represents the walker configuration where θ S and θ N are the angles of the stance-leg and nonstance-leg versus ground normal. During walking only one foot is in contact with the ground at any time, i.e. single- stance. Ground clearance of the swing-leg is ignored. A bipedal walker. Lumped masses are positioned at the hip (m H ) and on each leg (m L ). The spacetime- constraints walker applies torques at the stance foot (A) and at the hip (B).

24. Biomechanics Experiment The masses and limb lengths of the spacetime-constraints walker are modeled exactly as the passive-dynamic walker. However, the spacetime constraints algorithms permit non-zero torques about the hip and stance leg contact point and the classical homogeneous forward integration techniques are used to compute constraint violations, not to compute movement explicitly. The state vector, S, contains the entire movement trajectory and is composed of a scalar, dt, and angle vector, θ = [θ S, θ N ] T for every time increment t = 1…n. By including the time increment, dt, as a variable the swing period is permitted to approach an optimum.

25. Biomechanics Experiment The duration of a walk cycle can vary and resides under the control of the optimization algorithm. Such an extension was unnecessary for computer animators who prefer to specify when events start and end, but human movement certainly capitalizes on efficiencies obtained by changing a movement's pace.

26. Spacetime Constraints Andrew Witkin, Michael Kass, SIGGRAPH 1988 Early Computer Animations (Pixar’s Luxo, Jr. 1986) Can we hope to generate such motions automatically?

27. Spacetime Constraints Some aspects of animation are best left to the animator (personality, appeal, etc.) But perhaps we can relieve the animator from the burden of making the motion look physically plausible?   Forward Simulation Solves an initial value problem: Given initial state, forces along the way, and equations describing the system… Integrate the equations.

28. Spacetime Constraints But What About… What if the animator wants to specify both initial and final conditions? Two-point boundary problems And what about specifying some conditions describing what happens during the motion? And what if some of the forces are unknown?

29. Spacetime Constraints An example   We have a physical simulation that can be driven as an initial- value problem Given a sequence of torques that are applied by a bicyclist, we can simulate its final path   We don’t have the inverse Given a sequence of landmarks, compute the torques required to make a bicyclist ride through them… make the torques “optimal”

30. Spacetime Constraints An example   Given a sequence of landmarks, compute the torques required to make a bicyclist ride through them… make the torques “optimal” One could propose trajectories that go through landmarks but require unreasonable amounts of energy One could propose low-energy trajectories that don’t go through landmarks

31. Spacetime Constraints   Constrained optimization problem:   Physical systems tend to minimize energy   Spatial and physical constraints

32. Spacetime Constraints   Constrained optimization problem:   Physical systems tend to minimize energy   Spatial and physical constraints

33. Spacetime Constraints   Example of Boundary Value Problem- moving particle Particle position: x(t) Force on particle: f(t) Equation of motion: m− f− mg = 0 Boundary conditions: x(t 0 ) = a, x(t 1 ) = b Objective function :

34. Spacetime Constraints   Boundary Value Problem Find f(t), such that the resulting x(t) satisfies the boundary conditions … … subject to the requirement that R is minimal. Possible approach: solve for f(t) and x(t) simultaneously, using the equation of motion as a constraint between them.

35. Spacetime Constraints   Boundary Value Problem   Discretization Discretize the time interval [t 0,t 1 ] with step h The variables are: x 1, x 2, …x i, … x n and f 1, f 2, …f i, … f n Solve a constrained optimization problem in these unknowns. Enforcing the equation of motion results in n constraints: p i = m(x i-1 -2x i + x i+1 )/h 2 – f i - mg  i, 1  i  n Boundary constraints: c a =x 1 −a = 0, c b =x n −b= 0.

36. Spacetime Constraints   Constrained Optimization Minimize an objective function: R(s 1,…,s 1 ) in n variables s 1,…,s n subject to m constraints: C i (s 1,…,s n ) = 0 Sequential Quadratic Programming In order to minimize R we must zero its gradient. At the same time, we must also make sure that the constraints are satisfied. Idea: iterate, alternating between zeroing the gradient of R and zeroing C.

37. Spacetime Constraints Iterative Convergence