1. A system for automatic animation of piano performances
Alysha Bogaers Joel Morisset

2. Overview Generates 3D animations for piano performance from MIDI file.
Combines Graph Theory, Geometric Constraints, and Piano Theory.
Generates realistic finger placement, including:
finger crossover
Arpeggio

3. Problem Statement
Purpose: Automatically create 3D animations of piano playing
Why?
Hard for amateur players (actors?) to play "properly" considering finger placements
Hard to animate:
high dimension poses with many joints
exact musical timing constraints which have to hold
occlusions make motion capture hard
even if motion capture worked, piece must be played exactly like recording to have correct velocity/volume/timing etc

4. Related Work

5. String Instruments Use of statistical approaches
Learning method that evaluate a cost function, given a playing style.
Use of genetic algorithm to create fingering. (not elegant, but playable)
Interactive program that models various performance possibilities for the same music for different instruments
The Handrix system that generates guitar fingering using procedural algorithm.

6. Piano
Rule based expert systems that generates piano fingering. (Cases where the rules are not applicable)
2D piano key animation given any input music.
Use of a Greedy algorithm to generate the optimal fingering for Piano or Guitar

7. Human Modeling Anatomical approach to building a hand model
Kinematic approach
Effective physics-based approach

8. In Depth Explanation

9. In Depth Explanation Basic steps:
Find placement of the “used” fingers for chords
Find sequence of chord fingerings with a minimum loss of energy
Find positions of the “unused” fingers such that these are in natural positions or “ready to play” the next chords
Find natural hand position from constraints on the fingers, wrist and contact surface

10. Instructed Fingers
Compute cost of maintaining a hand pose for a specific chord
Compute cost for moving between different fingerings of subsequent chords
Find shortest path between chord fingerings

11. Non-Instructed Fingers
Consider next 1-4 chords for positioning of “unused” fingers
If a finger is used in the next chord, it should be positioned such that it has to move little from the current chord fingering
Fingers should not be too far apart for comfortable playing

12. Hand Model
16 Joints
27 DOFs
6 DOFs for the wrist
1 DOFs for DIP
1 DOFs for PIP
2 DOFs for MCP
3 DOFs for thumb base
Each finger has IK from the base to the tip
IK used for pressing/releasing the keys
FK is used for the in between movements

13. Fingers, Wrist, and Hand Position And Rotation
The generated fingering is used to decide the position of the fingertips along the Z axis
The finger base is decided by the wrist position
The fingertip, wrist position and orientation have to be calculated

14. Fingers, Wrist, and Hand Position And Rotation
Fingertips:
Z is determined by which keys are pressed
X is determined by the wrist position and the piano key range
Y is the height of a black or white key
Wrist Position:
Determined by the fact that X will move forward and the Y down the more spread the fingers are.
Fingers relative position along X in a standard pose with influence prioritized 1,5,2,4,3
The Z will move further from the thumb and closer to the little finger
Weighted sum of all fingertip position with larger weights for the thumb and little finger

15. Fingers, Wrist, and Hand Position And Rotation
Along Z: The wider the range the lower the wrist
Along X: keeps hand parallel with the piano
Uses the ring and index finger to define a line in 3D which projection is perpendicular to X and can be evaluated to get the rotation along X
Wrist Orientation:
Along Y:
Largest wrist orientation
Orientation of ray from wrist to Fingertip i for each finger i (1-5)
Largest Angle
Rotation weight influenced by the finger distribution
Precomputed based on Motion Capture Data
Standard pose pressing 5 neighboring keys

16. Optimized Hand Poses
Optimize by finding a wrist pose sequence which minimizes the overall cost of the wrist movements between chords

17. Hand Motion Animation
Generate a natural up- and downwards motion of the wrist from motion capture of performances
Introduce dependencies between instructed and non-instructed fingers
Generate realistic “feedback” on the wrist after pressing keys

18. DEMO
0:20 0:55 1:46

19. Critical Analysis Negative: Positive: Result seems Stiff
There where a few Erratas
Positive:
Good Paper
Well Structured
Explained in detail

20. Future Work Extension to other key pressing instruments
Method does not resolve interpenetration of fingers and the sides of black keys
Generate more emotional piano playing
Could be beneficial to apply machine learning to determine standard fingering sequences
Implement various hand size models
Playing music that require the two hand to play together
Consider rotation of joints within fingers

21. Questions