1. CSCE 441 Computer Graphics: Animation with Motion Capture
Jinxiang Chai

2. Outline of Mocap (Motion Capture)
Mocap history
Mocap technologies
Mocap pipeline
Mocap data format

3. Motion Capture
“ …recording of motion for immediate or delayed analysis or playback…”
David J. Sturman
“The creation of a 3d representation of a live performance”
- Alberto Menache
“…is a technique of digitally recording movements for entertainment, sports, and medical applications.”
- Wikipedia

4. History of Motion Capture
Eadweard Muybridge ( )
first person to photograph movement sequences
He was the first person to photograph movement sequences. At the time, it is not known if a horse ever had all feet off the ground while trotting. By recording multiple pictures over a short

5. History of Motion Capture
Eadweard Muybridge ( )
first person to photograph movement sequences
whether during a horse's trot, all four hooves were ever off the ground at the same time.
the flying horse
Sequence of a horse jumping (courtesy of E. Muybridge)

6. History of Motion Capture
Eadweard Muybridge ( )
first person to photograph movement sequences
the flying horse
animal locomotion (20k pictures about men, women, children, animals, and birds).
Woman walking downstairs (courtesy of E. Muybridge)

7. Rotoscope
Allow animators to trace cartoon character over photographed frames of live performances
invented by Max Fleischer in 1915

8. A horse animated by rotoscoping from Muybridge’s photos
Rotoscope
Allow animators to trace cartoon character over photographed frames of live performances
invented by Max Fleischer in 1915
2D manual motion capture
A horse animated by rotoscoping from Muybridge’s photos

9. Rotoscoping Mocap Overview
“rotoscoping can be thought of as a primitive form or precursor to motion capture, where the motion is ‘captured’ painstakingly by hand.” - Sturman

10. Another example

11. Rotoscope
Allow animators to trace cartoon character over photographed frames of live performances
invented by Max Fleischer in 1915
2D manual capture
the first cartoon character to be rotoscoped -- “Koko the clown”
the human character animation -- snow white and her prince (Walt Disney, 1937)
In 1915, using Muybridge’s idea, cartoonist Max Fleischer created the rotoscope, a
device that allowed animators to trace cartoon characters over photographed frames
of live performers. A time consuming process, mainly used for human motion, it
was the first time a real-life performance was used to help create an animated
character.
The first cartoon character to be rotoscoped was Koko the Clown.
The rotoscope was subsequently used by Walt Disney in 1937 to get realistic human
motion for Snow White and her prince. Rotoscoping is a two-dimensional approach
to capturing motion.
[Source: Menache, Alberto, “Understanding Motion Capture for Computer
Animation and Video Games,” Morgan Kaufmann, 2000]

12. Current Motion Capture Technologies
“3D Rotoscoping”: measuring 3D positions, orientations, velocities or accelerations automatically
Current motion capture systems
Electromagnetic
Electromechanical
Fiber optic
Optical

13. Electromagnetic Mocap
Each sensor record 3D position and orientation
Each sensor placed on joints of moving object
Full-body motion capture needs at least 15 sensors
Popular system:

14. Electromagnetic Mocap
See video demo [1, 2]!

15. Electromagnetic Mocap
Pros
measure 3D positions and orientations
no occlusion problems
can capture multiple subjects simultaneously
Cons
magnetic perturbations (metal)
small capture volume
cannot capture deformation (facial expression)
hard to capture small bone movement (finger movement)
not as accurate as optical mocap systems

16. Electromechanical Mocap
Each sensor measures 3D orientations
- including 3D accelerometers, 3D gyros, and 3D magnetometers

17. Electromechanical Mocap
Each sensor measures 3D orientations
Each sensor placed on joints of moving object
Full-body motion capture needs at least 15 sensors
Popular systems:

18. Electromechanical Mocap
See video demo [1,2]!

19. Electromechanical Mocap
Pros
measure 3D orientations
no occlusion problems
can capture multiple subjects simultaneously
large capture volume
portable and outdoors capture (e.g. skiing)
Cons
getting 3D position info is not easy
the root positions is often measured with ultrasonic position sensors
cannot capture deformation (facial expression)
hard to capture small bone movement (finger motion)
not as accurate as optical mocap system

20. Fiber Optic Mocap Measures 3D position and orientation of entire tape
Binding the tape to the body
Popular systems:
Optical fiber systems such as data gloves use a fiber-optic sensor along the fingers.
As the fingers bend, they bend the fiber and the transmitted light is attenuated. The
finger joint rotation measurements are controlled by the strength of the attenuated
light.

21. Fiber Optic Mocap
See video demo [1,2]!

22. Fiber Optic Mocap Pros measure 3D orientations and positions
no occlusion problems
can capture multiple subjects simultaneously
go anywhere mocap system
can capture hand/finger motion
Cons
intrusive capture
cannot capture deformation (facial expression)
not as accurate as optical mocap system

23. Optical Mocap
Multiple calibrated cameras (>=8) digitize different views of performance
Wears retro-reflective markers
Accurately measures 3D positions of markers

24. Optical Mocap Vicon mocap system: http://www.vicon.com
See video demo [1,2,3,4]!

25. Optical Mocap Pros measure 3D positions and orientations Cons
the most accurate capture method
very high frame rate
can capture very detailed motion (body, finger, facial deformation, etc.)
Cons
has occlusion problems
hard to capture interactions among multiple actors
limited capture volume
expensive

26. Video-based Mocap
Capturing 3D performance from single-camera video streams

27. Video-based Mocap
Capturing 3D performance from single-camera video streams
Click video here

28. Video-based Mocap Pros: Cons: - not a mature technology
- capturing human motion anytime, anywhere
- very cheap
- zillions of films, sports footage, and internet videos.
Cons:
- not a mature technology
- quality is not as good as other capturing technologies.

29. Mocap Pipeline Optical Mocap pipeline Planning Calibration
Data processing

30. Planning Motion capture requires serious planning
Anticipate how the data will be used
Garbage in garbage out
Shot list
Games
motions need to be able to blend into one an another
capture base motions and transitions
which motions transition into which other transitions
cycles/loops

31. Movement Flowchart for Games
Planning and Directing Motion Capture For Games By Melianthe Kines Gamasutra January 19, 2000 URL:

32. Planning Character/prop set up
- character skeleton topology (bones/joints number, Dofs for each bone)
- location and size of props
Marker Setup
- the number of markers
- where to place markers
Given a storyboard or game design, the first step in planning a motion capture shoot
is to determine the character and marker setup of any characters or props to be
captured. Character setup deals with determining the skeletal topology of the
character including number of joints and/or bones and degrees of freedom of each
joint.
Marker setup refers to the locations of the markers on the performer/prop that will
be used to capture the motion. Character setup depends on the marker setup since
bone rotations are determined from marker positions. The marker positions used
must be adequate to determine bone rotations in the character.

33. Calibration Camera Calibration:
determine the location and orientation of each camera
determine camera parameters (e.g. focal length)
Subject calibration
- determine the skeleton size of actors/actresses (.asf file)
- relative marker positions in terms of bones
- determine the size and location of props

34. Processing Markers Each camera records capture session
Extraction: markers need to be identified in the image
determines 2d position
problem: occlusion, marker is not seen
use more cameras
Markers need to be labeled
which marker is which?
problem: crossover, markers exchange labels
may require user intervention
Compute 3d position: if a marker is seen by at least 2 cameras then its position in 3d space can be determined

35. Data Process Complete 3D marker trajectories (.c3d file)
3D marker positions (.c3d file)
Fill in missing data
Filter mocap data

36. Data Process Complete 3D marker trajectories (.c3d file)
3D marker positions (.c3d file)
Fill in missing data
Filter mocap data
Inverse Kinematics
Joint angle data (.amc file)

37. Data Process How to represent motion data in joint angle space?
Complete 3D marker trajectories (.c3d file)
3D marker positions (.c3d file)
Fill in missing data
Filter mocap data
Inverse Kinematics
How to represent motion data in joint angle space?
Joint angle data (.amc file)

38. Motion Capture Data Files
Each sequence of human motion data contains two files:
Skeleton file (.asf): Specify the skeleton model of character
Motion data file (.amc): Specify the joint angle values over the frame/time
Both files are generated by Vicon software 38

39. Human Skeletal File
Described in a default pose 39

40. Human Skeletal Model
This is still a tree! 40

41. Human Skeletal Model How to describe the skeletal model?
What should you know about each bone?
This is still a tree! 41

42. Human Skeletal File (.asf)
individual bone information
- length of the bone
- direction of the bone
- local coordinate frame
- number of Dofs
- joint limits
bone hierarchy/connections 42

43. Individual Bone Information
begin id bone_id                  /* Unique id for each bone */ name bone_name        /* Unique name for each bone */ direction dX dY dZ    /* Vector describing direction of the bone in world */ coor. system length            /* Length of the bone*/ axis XYZ         /* Rotation of local coordinate system for                                    this bone relative to the world coordinate                                    system. In .AMC file the rotation angles                                     for this bone for each time frame will be                                    defined relative to this local coordinate                                     system**/ dof rx ry rz                /* Degrees of freedom for this bone. limits ( ) /* joint limits*/             ( )             ( ) end 43

44. Individual Bone Information
begin id 2                 name lfemur        direction     length            axis XYZ         dof rx ry rz                limits ( )             ( )             ( ) end xw yw zw 44

45. Individual Bone Information
begin id 2                 name lfemur        direction     length            axis XYZ         dof rx ry rz                limits ( )             ( )             ( ) end xw yw zw 45

46. Individual Bone Information
begin id 2                 name lfemur        direction     length            axis XYZ         dof rx ry rz                limits ( )             ( )             ( ) end xw yw zw 46

47. Individual Bone Information
begin id 2                 name lfemur        direction     length            axis XYZ         dof rx ry rz                limits ( )             ( )             ( ) end yk xk zk xw yw zw
Euler angle representation:
Rk=Rz(γ)Ry(β)Rx(α) 47

48. Individual Bone Information
begin id 2                 name lfemur        direction     length            axis XYZ         dof rx ry rz                limits ( )             ( )             ( ) end yk xk zk xw yw zw
- The number of dof for this joint
- The minimal and maximum joint angle for each dof 48

49. Individual Bone Information
begin id 2                 name lfemur        direction     length            axis XYZ         dof rx ry rz                limits ( )             ( )             ( ) end yk xk zk xw yw zw
1-dof joint
2-dof joint
3-dof joint 49

50. Individual Bone Information
begin id 2                 name lfemur        direction     length            axis XYZ         dof rx ry rz                limits ( )             ( )             ( ) end yk xk zk
yk+1
Xk+1
begin id 3                 name ltibia        direction     length            axis XYZ         dof rx            limits ( ) end
zk+1 50

51. Root Representation :root order TX TY TZ RX RY RZ axis XYZ
position 0 0 0
orientation 0 0 0 xw yw zw 51

52. Root Representation :root order TX TY TZ RX RY RZ axis XYZ
position 0 0 0
orientation 0 0 0 xw yw zw
How to compute the coordinate of a joint in the world coordinate frame? 52

53. Root Representation :root order TX TY TZ RX RY RZ axis XYZ
position 0 0 0
orientation 0 0 0 xw yw zw
How to compute the coordinate of a joint in the world coordinate frame? 53

54. Hierarchy/Bone Connections
begin
root lhipjoint rhipjoint lowerback
lhipjoint lfemur
lfemur ltibia
ltibia lfoot
lfoot ltoes
rhipjoint rfemur
rfemur rtibia
rtibia rfoot
rfoot rtoes
lowerback upperback
upperback thorax
thorax lowerneck lclavicle rclavicle …
end 54

55. Hierarchy/Bone Connections
begin
root lhipjoint rhipjoint lowerback
lhipjoint lfemur
lfemur ltibia
ltibia lfoot
lfoot ltoes
rhipjoint rfemur
rfemur rtibia
rtibia rfoot
rfoot rtoes
lowerback upperback
upperback thorax
thorax lowerneck lclavicle rclavicle …
end
lowerback
root
rhipjoint 55

56. Hierarchy/Bone Connections
begin
root lhipjoint rhipjoint lowerback
lhipjoint lfemur
lfemur ltibia
ltibia lfoot
lfoot ltoes
rhipjoint rfemur
rfemur rtibia
rtibia rfoot
rfoot rtoes
lowerback upperback
upperback thorax
thorax lowerneck lclavicle rclavicle …
end
lowerback
root
rhipjoint
lhipjoint
lfemur 56

57. Hierarchy/Bone Connections
begin
root lhipjoint rhipjoint lowerback
lhipjoint lfemur
lfemur ltibia
ltibia lfoot
lfoot ltoes
rhipjoint rfemur
rfemur rtibia
rtibia rfoot
rfoot rtoes
lowerback upperback
upperback thorax
thorax lowerneck lclavicle rclavicle …
end
lowerback
root
rhipjoint
lhipjoint
lfemur
ltibia 57

58. Hierarchy/Bone Connections
begin
root lhipjoint rhipjoint lowerback
lhipjoint lfemur
lfemur ltibia
ltibia lfoot
lfoot ltoes
rhipjoint rfemur
rfemur rtibia
rtibia rfoot
rfoot rtoes
lowerback upperback
upperback thorax
thorax lowerneck lclavicle rclavicle …
end
lowerback
root
rhipjoint
lhipjoint
lfemur
ltibia
lfoot 58

59. Hierarchy/Bone Connections
begin
root lhipjoint rhipjoint lowerback
lhipjoint lfemur
lfemur ltibia
ltibia lfoot
lfoot ltoes
rhipjoint rfemur
rfemur rtibia
rtibia rfoot
rfoot rtoes
lowerback upperback
upperback thorax
thorax lowerneck lclavicle rclavicle …
end
lowerback
root
rhipjoint
lhipjoint
lfemur
ltibia
lfoot
ltoe 59

60. What Can We Do With .asf File?
We can visualize the default pose
We can compute various transforms in the default pose
- between world coordinate frame and local coordinate
- between parent coordinate frame and child coordinate frame 60

61. From Local Coordinate to World Coordinatexw yw zw yk xk zk 61

62. From Local Coordinate to World Coordinatexw yw zw ? ? yk xk zk 62

63. From Local Coordinate to World Coordinatexw yw zw yk 63

64. From Local Coordinate to World Coordinatexw yw zw yk xk zk 64

65. From Child to Parent Node
How to Compute the transformation Tkk-1 from a child local coordinate frame to its parent local coordinate frame
Tkk-1 x 65

66. Bone Transform
parent
Tkk-1?
world
child 66

67. Bone Transform
parent
Tkk-1?
world
child 67

68. Bone Transform
parent
Tkk-1?
world
child 68

69. Motion Data File (.amc) i // frame number
root // root position and orientation
lowerback // joint angles for lowerback joint
upperback // joint angles for thorax joint
thorax
lowerneck
upperneck
head
rclavicle e e-014
rhumerus
rradius
rwrist
rhand
rfingers
rthumb
lclavicle e e-014
lhumerus 69

70. Motion Data File (.amc) - Rotation described in local coordinate frame
i // frame number
root // root position and orientation
lowerback // joint angles for lowerback joint
upperback // joint angles for thorax joint
thorax
lowerneck
upperneck
head
rclavicle e e-014
rhumerus
rradius
rwrist
rhand
rfingers
rthumb
lclavicle e e-014
lhumerus
- Rotation described in local coordinate frame
- Euler angle representation x-y-z 70

71. Composite 3D Transformation
From .asf file 71

72. Composite 3D Transformation
From .amc file 72

73. Composite 3D Transformation73

74. Composite 3D Transformation74

75. Composite 3D Transformation75

76. Composite 3D Transformation76

77. Online Motion Capture Database

78. Computational Photography
CSCE489: Special topics in Computer Graphics [click here for syllabus].
Spring 2011
Computational Photography is an emerging new field created by the convergence of computer graphics, computer vision and photography.
Its role is to overcome the limitations of the traditional camera by using computational techniques to produce a richer, more vivid, perhaps more perceptually meaningful representation of our visual world.