Music-Driven Motion Editing Marc Cardle Rainbow Group Computer Laboratory University of Cambridge Marc Cardle Rainbow Group Computer Laboratory University of Cambridge

Music-Driven Motion Editing – Marc Cardle Overview of Talk Background: Example Animation Background: Example Animation Research goals and framework overview Research goals and framework overview Musical Features Musical Features Motion Editing Motion Editing Examples Examples Background: Example Animation Background: Example Animation Research goals and framework overview Research goals and framework overview Musical Features Musical Features Motion Editing Motion Editing Examples Examples

Music-Driven Motion Editing – Marc Cardle What is driving what? Film Soundtracks Music Videos

Music-Driven Motion Editing – Marc Cardle Animusic Approach (Lytle ’94)

Music-Driven Motion Editing – Marc Cardle A framework for motion editing using music analysis A framework for motion editing using music analysis Simplify and automate the generation of synchronised musical animations Simplify and automate the generation of synchronised musical animations A framework for motion editing using music analysis A framework for motion editing using music analysis Simplify and automate the generation of synchronised musical animations Simplify and automate the generation of synchronised musical animations Research Goal

Music-Driven Motion Editing – Marc Cardle MIDIMIDI Parametric representation Parametric representation Pitch, velocity Pitch, velocity Start-time and End-time Start-time and End-time Parametric representation Parametric representation Pitch, velocity Pitch, velocity Start-time and End-time Start-time and End-time Synthesizer Sampler Audio Signal (Lytle ’94)

Music-Driven Motion Editing – Marc Cardle Audio and MIDI Analysis is based on both MIDI and Audio Analysis is based on both MIDI and Audio Complementary perspectives on the analysed music Complementary perspectives on the analysed music Analysis is based on both MIDI and Audio Analysis is based on both MIDI and Audio Complementary perspectives on the analysed music Complementary perspectives on the analysed music + + Audio MIDI

Music-Driven Motion Editing – Marc Cardle System Overview

Music-Driven Motion Editing – Marc Cardle Musical Features

Music-Driven Motion Editing – Marc Cardle MIDI-Based Analysis

Music-Driven Motion Editing – Marc Cardle MIDI-Based Analysis Low-level MIDI note information Low-level MIDI note information MIDI Channel and Instrument information MIDI Channel and Instrument information Chord Recognizer Chord Recognizer Virtual Pitch Virtual Pitch Beat Tracking Beat Tracking Meter Meter Low-level MIDI note information Low-level MIDI note information MIDI Channel and Instrument information MIDI Channel and Instrument information Chord Recognizer Chord Recognizer Virtual Pitch Virtual Pitch Beat Tracking Beat Tracking Meter Meter

Music-Driven Motion Editing – Marc Cardle MIDI-Based Analysis Segmentation Segmentation Pattern Matching Pattern Matching Segmentation Segmentation Pattern Matching Pattern Matching

Music-Driven Motion Editing – Marc Cardle MIDI-Based Analysis Register Register Loudness Loudness Density Density Duration Duration Attack Speed Attack Speed Focus and Delay Focus and Delay Register Register Loudness Loudness Density Density Duration Duration Attack Speed Attack Speed Focus and Delay Focus and Delay Time Max Pitch Min Pitch Pitch

Music-Driven Motion Editing – Marc Cardle Audio-Based Analysis

Music-Driven Motion Editing – Marc Cardle Audio-Based Analysis Zero-Crossing Rate Zero-Crossing Rate Spectrum Amplitude Variations Spectrum Amplitude Variations Spectral Centroid Spectral Centroid Zero-Crossing Rate Zero-Crossing Rate Spectrum Amplitude Variations Spectrum Amplitude Variations Spectral Centroid Spectral Centroid

Music-Driven Motion Editing – Marc Cardle Features Animation MIDI Soundtrack Register Density - spread Density - notes Novelty Frequency Duration Dynamics Attack Speed Audio Soundtrack Mean Roll-off Mean Zero-Cr. Std Centroid Std Flux Std RMS Mean Centroid Mean Flux Mean RMS Std Roll-off Std Zero-Cr.

Music-Driven Motion Editing – Marc Cardle Motion Editing

Music-Driven Motion Editing – Marc Cardle Motion Editing Cyclification Cyclification (Silva ‘99) Time warping Time warping (Bruderlin ‘95) Various motion linear and non-linear filtering techniques Various motion linear and non-linear filtering techniques Cyclification Cyclification (Silva ‘99) Time warping Time warping (Bruderlin ‘95) Various motion linear and non-linear filtering techniques Various motion linear and non-linear filtering techniques (Gleicher ’00) (Litwinowicz ‘91) (Bruderlin ‘95)

Music-Driven Motion Editing – Marc Cardle Motion Editing Frequency spectrum re-scaling Frequency spectrum re-scaling (Bruderlin ‘95) Standard and multi-target blending Standard and multi-target blending Frequency spectrum re-scaling Frequency spectrum re-scaling (Bruderlin ‘95) Standard and multi-target blending Standard and multi-target blending

Music-Driven Motion Editing – Marc Cardle Motion Editing Emotional Transforms Emotional Transforms (Polich. ‘01) Noise addition Noise addition (Perlin ‘02) Motion warping Motion warping (Witkin ’95) Emotional Transforms Emotional Transforms (Polich. ‘01) Noise addition Noise addition (Perlin ‘02) Motion warping Motion warping (Witkin ’95) (Gleicher ’00)

Music-Driven Motion Editing – Marc Cardle Example - 1 Original Keyframe Motion Boring…

Music-Driven Motion Editing – Marc Cardle Example - 1 Modified Keyframe Motion

Music-Driven Motion Editing – Marc Cardle Example - 2 Velocity End-effectors Time Minima

Music-Driven Motion Editing – Marc Cardle Example - 2 Time Motion Amplitude Density Event New Motion

Music-Driven Motion Editing – Marc Cardle Animation - 2

Music-Driven Motion Editing – Marc Cardle Example - 3 Zero-Crossing Rate RMS Volume

Music-Driven Motion Editing – Marc Cardle Animation - 3

Music-Driven Motion Editing – Marc Cardle Future Work Add more music analysis and motion editing methods Add more music analysis and motion editing methods Maya Plugin with high-level descriptions Maya Plugin with high-level descriptions User testing (interested ?) User testing (interested ?) Add more music analysis and motion editing methods Add more music analysis and motion editing methods Maya Plugin with high-level descriptions Maya Plugin with high-level descriptions User testing (interested ?) User testing (interested ?)

Music-Driven Motion Editing – Marc Cardle Thank you Any Questions? Marc Cardle Rainbow Group Computer Laboratory University of Cambridge Thank you Any Questions? Marc Cardle Rainbow Group Computer Laboratory University of Cambridge

Music-Driven Motion Editing – Marc Cardle S2002 Bookman Old Style, Bold, 37 points This subtitle is 31 points Bullets are orange; text is 26 points Bullets are orange; text is 26 points They have 110% line spacing, 6 points before/after They have 110% line spacing, 6 points before/after Longer bullets in the form of a paragraph are harder to read if there is insufficient line spacing. This is the maximum recommended number of lines per slide (seven). Longer bullets in the form of a paragraph are harder to read if there is insufficient line spacing. This is the maximum recommended number of lines per slide (seven). Sub-bullets look like this. This subtitle is 31 points Bullets are orange; text is 26 points Bullets are orange; text is 26 points They have 110% line spacing, 6 points before/after They have 110% line spacing, 6 points before/after Longer bullets in the form of a paragraph are harder to read if there is insufficient line spacing. This is the maximum recommended number of lines per slide (seven). Longer bullets in the form of a paragraph are harder to read if there is insufficient line spacing. This is the maximum recommended number of lines per slide (seven). Sub-bullets look like this.