A system for automatic animation of piano performances Alysha Bogaers Joel Morisset

Overview Generates 3D animations for piano performance from MIDI file. Combines Graph Theory, Geometric Constraints, and Piano Theory. Generates realistic finger placement, including: finger crossover Arpeggio

Problem Statement Purpose: Automatically create 3D animations of piano playing Why? Hard for amateur players (actors?) to play "properly" considering finger placements Hard to animate: high dimension poses with many joints exact musical timing constraints which have to hold occlusions make motion capture hard even if motion capture worked, piece must be played exactly like recording to have correct velocity/volume/timing etc

Related Work

String Instruments Use of statistical approaches Learning method that evaluate a cost function, given a playing style. Use of genetic algorithm to create fingering. (not elegant, but playable) Interactive program that models various performance possibilities for the same music for different instruments The Handrix system that generates guitar fingering using procedural algorithm.

Piano Rule based expert systems that generates piano fingering. (Cases where the rules are not applicable) 2D piano key animation given any input music. Use of a Greedy algorithm to generate the optimal fingering for Piano or Guitar

Human Modeling Anatomical approach to building a hand model Kinematic approach Effective physics-based approach

In Depth Explanation

In Depth Explanation Basic steps: Find placement of the “used” fingers for chords Find sequence of chord fingerings with a minimum loss of energy Find positions of the “unused” fingers such that these are in natural positions or “ready to play” the next chords Find natural hand position from constraints on the fingers, wrist and contact surface

Instructed Fingers Compute cost of maintaining a hand pose for a specific chord Compute cost for moving between different fingerings of subsequent chords Find shortest path between chord fingerings

Non-Instructed Fingers Consider next 1-4 chords for positioning of “unused” fingers If a finger is used in the next chord, it should be positioned such that it has to move little from the current chord fingering Fingers should not be too far apart for comfortable playing

Hand Model 16 Joints 27 DOFs 6 DOFs for the wrist 1 DOFs for DIP 1 DOFs for PIP 2 DOFs for MCP 3 DOFs for thumb base Each finger has IK from the base to the tip IK used for pressing/releasing the keys FK is used for the in between movements

Fingers, Wrist, and Hand Position And Rotation The generated fingering is used to decide the position of the fingertips along the Z axis The finger base is decided by the wrist position The fingertip, wrist position and orientation have to be calculated

Fingers, Wrist, and Hand Position And Rotation Fingertips: Z is determined by which keys are pressed X is determined by the wrist position and the piano key range Y is the height of a black or white key Wrist Position: Determined by the fact that X will move forward and the Y down the more spread the fingers are. Fingers relative position along X in a standard pose with influence prioritized 1,5,2,4,3 The Z will move further from the thumb and closer to the little finger Weighted sum of all fingertip position with larger weights for the thumb and little finger

Fingers, Wrist, and Hand Position And Rotation Along Z: The wider the range the lower the wrist Along X: keeps hand parallel with the piano Uses the ring and index finger to define a line in 3D which projection is perpendicular to X and can be evaluated to get the rotation along X Wrist Orientation: Along Y: Largest wrist orientation Orientation of ray from wrist to Fingertip i for each finger i (1-5) Largest Angle Rotation weight influenced by the finger distribution Precomputed based on Motion Capture Data Standard pose pressing 5 neighboring keys

Optimized Hand Poses Optimize by finding a wrist pose sequence which minimizes the overall cost of the wrist movements between chords

Hand Motion Animation Generate a natural up- and downwards motion of the wrist from motion capture of performances Introduce dependencies between instructed and non-instructed fingers Generate realistic “feedback” on the wrist after pressing keys

DEMO 0:20 0:55 1:46

Critical Analysis Negative: Positive: Result seems Stiff There where a few Erratas Positive: Good Paper Well Structured Explained in detail

Future Work Extension to other key pressing instruments Method does not resolve interpenetration of fingers and the sides of black keys Generate more emotional piano playing Could be beneficial to apply machine learning to determine standard fingering sequences Implement various hand size models Playing music that require the two hand to play together Consider rotation of joints within fingers

Questions