Automatic Transcription of Polyphonic Music Jana Eggink Supervisor: Guy J. Brown University of Sheffield {j.eggink g.brown}@dcs.shef.ac.uk

me Master in systematic musicology with computer science as a second subject, University of Hamburg, Germany, 2001 Interested in perception and computational auditory scene analysis And, of course, music Jana Eggink, Sheffield, UK Automatic Transcription of Music

Automatic Transcription of Music General idea: get a score from an audio file • Useful for: automatic music indexing and analysis, detection of copyright infringement, ‘query-by-humming’ systems... Many subtasks: F0 estimation, onset detection, instrument recognition flute Jana Eggink, Sheffield, UK Automatic Transcription of Music

Missing Data Ideas from speech recognition and speaker identification in noisy environments Missing data: ignore time frequency regions dominated by interfering sound sources Determination of unreliable of missing features based on F0s of interfering tones Works well with two simultaneous sounds But problems already with F0 estimation with 3 or 4 concurrent tones Jana Eggink, Sheffield, UK Automatic Transcription of Music

Change of Focus… Instead on using missing data masks based on interfering sound sources, concentrate on the signal you want to identify Task: Identify the solo instrument in accompanied sonatas and concertos Jana Eggink, Sheffield, UK Automatic Transcription of Music

Identify Solo Instrument • Instrument sounds are harmonic, energy is concentrated in partials which are least likely to be masked by other sounds • Only most dominant F0 needs to be identified • Features based only on frequency position and power of lowest 15 partials • Statistical recogniser (GMMs) trained on monophonic music flute clarinet oboe violin cello audio signal recog-niser features F0 and partials Jana Eggink, Sheffield, UK Automatic Transcription of Music

Results Instrument Identification • Solo instrument with accompaniment (piano or orchestra), commercially available CDs, 90 examples, 2-3 min. each • Instrument 86% correct, as good as other systems that only work with monophonic music Jana Eggink, Sheffield, UK Automatic Transcription of Music

Find the Melody (assuming the solo instrument is known) • Extract multiple F0 candidates • Include additional knowledge about instrument range, tone duration, likely interval transitions to pick correct candidate F0 strength (~loudness) F0 likelihood (absolute frequency | instrument range) instrument likelihood (recogniser output) LOCAL KNOWLEDGE TEMPORAL KNOWLEDGE tone length interval transitions find most likely ‘path’ through time- frequency space of F0 candidates AUDIO F0 candidates silence estimation (only accompaniment?) MELODY Jana Eggink, Sheffield, UK Automatic Transcription of Music

Results • Improving the number of correct frames from 40% to 54%, greatly reducing the number of spurious tones • Beginning of Mozart’s Clarinet Concerto, manually annotated F0s (gray) and estimated melody (black) Melody based on strongest F0 time (frames) F0 (Hz) F0 (Hz) Melody based on knowledge integrating path finding time (frames) Jana Eggink, Sheffield, UK Automatic Transcription of Music

Experiences in the Network Many techniques used for speech recognition are (with modifications) transferable to musical applications, but are often not well known among people who work with music, very good training in Sheffield Insight into other related areas in the HOARSE meetings, Which are also great for personal contacts Jana Eggink, Sheffield, UK Automatic Transcription of Music