1. What is automatic music transcription? Transforming an audio signal of a music performance in a symbolic representation (MIDI or score). Aim: This prototype is conceived as a research platform for developing and applying interactive and multimodal techniques to the monotimbral transcription task. Problem decomposition (summary): AUDIO F0 frame by frame estimation Note pitch detection Transcription More accurate problem decomposition (multimodal & interactive) : SIGNAL SCORES Music models Envelope (amplitude) F0 frame by frame Note pitch detection Transcription Tonality Meter Tempo Note onsets Description and Retrieval of Music and Sound Information Descripción y Recuperación de Información Musical y Sonora PROJEC T Operation diagram: New PROJECT Onsets Pulses Notes Onsets Text / Harmony Text / Harmony Rhythm: tempo + meter Rhythm: tempo + meter ANALYSIS (information source) ANALYSIS (information source) INTERACTION with INTERACTION with TRANSCRIPTION based on TRANSCRIPTION based on Spectrogram Onsets Pulses Frames Physical level Musical level off-line melodic and harmonic models Multimodality: it uses three different sources of information to detect notes in a musical audio excerpt: signal, note onsets, and rhythm information. Interactive: Designed to make use of user feedback on onsets, beats, and notes in a left-to- right validation approach: a user interaction validates what remains at the left-hand side, interactions are used to re-compute the rest of the output. Structure overview Signal F0 (in Hz) Piano roll Music score State-of-the-art techniques are far from being accurate, specially in the case of polyphony and multitimbral sounds. So nothing even close to 100% can be expected  User corrections are needed. (off-line) XML file Rhythm Interactions allowed Interface structure: Interaction assistance Menus Play Markers & timing area Tempo and meter area Transcription area: piano roll / score Audio signal area Textual transcription area Chord segmentation area Audio properties Keyboard / staves reference Tonality Rhythm properties Text properties Raw (frame by frame) transcription: Screencast: Based just on harmonic energies in the spectrogram. Smoothed by a frame context. Filtered by a length threshold (in frames). Many short false psoitives and negatives. Spectrogram Pitches in frames Onset-based transcription: Onsets Pitches Spectrogram Onsets impose a segmentation. Only at onsets notes can change. Times are still physical. Transcription is much more accurate. Interaction with onsets affect the transcription Pulse-based transcription: Pulses Notes Spectrogram Beat, tempo and meter are derived from pulses. Transcription is driven by them using a division of beat. Times are now musical. Transcription is score-oriented. A false negative is corrected by the user: This correction solves other FN: Trasncription is recomputed with the new onset Changes are propagated Harmonic analysis (chord segmentation) is provided This work is supported by the Consolider Ingenio 2010 research programme (project MIPRCV, CSD2007-00018), the project DRIMS (TIN2009-14247-C02), and the PASCAL2 Network of Excellence (IST-2007-216886). The authors want to thanks to the people that is involved in this project, specially those who do not appear as Authors of this paper, like Carlos Pérez-Sancho, David Rizo, Javier Sober, José Bernabeu, or Gabriel Meseguer. Acknowledgements: Spectrogram  Frames  Set of pitch candidates  Selection by “salience”  Smoothing in short context  Set of pitches by frame Very short notes can be filtered out by merging or deleting them by parameters controlled by the user. Signal  Rate of change of pitched energy  Threshold  Onsets  Segmentation  Segment transcription  Set of pitches by segment Signal  Energy fluctuations  Pulses  Beats and Tempo  Quantization  Quantized transcription  Notes (pitch and duration) Note durations acquire musical meaning. Required if a music score is aimed as the final output, otherwise only a piano roll can be obtained. Frame-based transcription: Onset-based transcription: Pulse-based transcription: Interactions: Implemented or planned: onsets (add, remove, edit), pulses (modify beat and meter), notes (add, remove, edit), and harmony (chord segmentation). Transcription modes: A Multimodal Music Transcription Prototype First steps in an interactive prototype development Tomás Pérez-García, José M. Iñesta, Pedro J. Ponce de León, Antonio Pertusa Universidad de Alicante, Spain Warning: This is a project in its very early stage, so there are many functionalities still not implemented and it is far from being bug-free. More information: At http://miprcv.iti.upv.es/ a video screencast and an on-line demo are available.