Automatic synchronisation between audio and score musical description layers Antonello D’Aguanno, Giancarlo Vercellesi Laboratorio di Informatica Musicale Università degli Studi di Milano Speaker: Antonello D’Aguanno

Introduction Contemporary digital music archives consist of huge collections of heterogeneous documents The heterogeneity of music information makes retrieval hard to accomplish Synchronization is one of the open problems in Music Information Retrieval (MIR) Automatic synchronisation between audio and score musical description layers 2 /15

Synchronisation 3 /15 Stream Audio Score Synchronized score Synchronization Synchronisation means that for a given event in a music score we can determine the timing of the corresponding audio events Automatic synchronisation between audio and score musical description layers

Related Works Many algorithms have been proposed in literature that deal with synchronisation All the algorithms proposed use MIDI score representation and PCM audio 4 /15 Automatic synchronisation between audio and score musical description layers

Related Works The unsolved problems of synchronisation are not limited to finding a suitable algorithm How can this result generalised to use it in other applications? No answers in literature… MX can solve the problem 5 /15 Automatic synchronisation between audio and score musical description layers

The linking structure in MX 6 /15 Notational Layer Spine Sub-Layer Performance Layer Audio Layer Automatic synchronisation between audio and score musical description layers

The COMSI algorithm Three phases: Score Analysis Audio Analysis Recursive Decisional Matching 7 /15 Automatic synchronisation between audio and score musical description layers

The three phases Score Analysis: The MX score is read in order to extract all relevant musical events Audio Analysis: The PCM audio signal is analysed to identify all possible musical notes Recursive Decisional Matching: Relates the event at the score level with the same event at the audio level 8 /15 Automatic synchronisation between audio and score musical description layers

MX score analysis 1.For each measure, only the notes with strong accent are selected 2.For every strong accent, the verticalisation of the score is computed 9 /15 Observation: COMSI Algorithm can manage conveniently even MIDI score Automatic synchronisation between audio and score musical description layers

Audio Analysis 1.The audio signal is filtered with a notch filter centred on the frequency of the note examined 2.The audio signal in time-windows is split at 100ms 3.For each note, a possible attack-time is the audio segment which has an energy value above a threshold, obtained from the average energy of the filtered signal 10 /15 Automatic synchronisation between audio and score musical description layers

Execution Events The execution events are all the audio events having an energy above the threshold The set of these execution events is named pseudo-score 11 /15 Automatic synchronisation between audio and score musical description layers

The Decisional Matching 12 /15 Automatic synchronisation between audio and score musical description layers Measure Synchronisation Recursive research of attack-time related to the first event of each measure in the score Musical Event Synchronization Sequential research of each musical event contained in a measure

Results 13 /15 TrackTime Signature Correct Measure One Error Measure Event Twice Error Measure Event Chopin 3/466%24%10% Beethoven 4/471%18%11% Beethoven Faster 4/459%36%15% Fiore di Maggio 4/480%17%13% Cavatina 6/843%48%9% Aria 6/861%31%8% Automatic synchronisation between audio and score musical description layers

Conclusions In this work we have described algorithm dedicated to score and audio alignment using the MX / IEEE P1599 format This algorithm allows alignment of an MX score and its execution, coded in PCM format It produces an output for the MX Spine that contains synchronisation between notes and audio signal 14 /15 Automatic synchronisation between audio and score musical description layers

Future works A new test infrastructure is being developed, which will use the MX capabilities More tests in a shorter time! Will take into account every musical event Automatic synchronisation between audio and score musical description layers 15 /15

Thanks! Titolo della presentazione16 di 15