Onset Detection, Tempo Estimation, and Beat Tracking J.-S Roger Jang (張智星) http://mirlab.org/jang MIR Lab, CSIE Dept. National Taiwan University

What Are Onsets? Music instrument event is usually divided into 4 stages of ADSR (attack, decay, sustain, release) based on its energy profile. Onset is the time when the slope is the highest, during the attack time. Quiz!

Characteristics of Onset Detection Music types Monophonic (single sound source)  Easier Polyphonic (multiple sound sources)  Harder Instrument types Percussive instruments  hard onsets which are easier to detect String instruments  Soft onsets which are harder to detect

Why Onset Detection is Important? It is a fundamental step in audio music analysis Music transcription (from wave to midi) Music editing (Song segmentation) Tempo estimation and beat tracking Musical fingerprinting (the onset trace can serve as a robust feature for fingerprinting)

Onset Detection Function ODF (onset detection function) creates a curve of onset strength, aka Onset strength curve (OSC) Novelty curve Most ODFs are based on time-frequency representation Magnitude of STFT  spectrogram Phase of STFT Mel-band of STFT Constant-Q transform Short-time Fourier transform

Spectrogram wave2specGram.m X(n, k)

ODF: Spectral Flux Concept Sum the positive change in each frequency bin Quiz!

Flowchart of OSC Steps of OSC Check out wave2osc.m to see these steps. Spectrogram Mel-band spectrogram (optional) Spectral flux Smoothed OSC via Gaussian smoothing Trend of OSC via Gaussian smoothing Trend-subtracted OSC Check out wave2osc.m to see these steps.

Mel-freq Spectrogram 40 filters in Mel-freq filter bank Spectrograms Linear freq spec1 Mel freq spec2 Matrix M spec2=M*spec1 melBinPlot.m

Mel-freq Representation About mel-freq spectrogram Advantage: More correlated to human perception (just like MFCC in speech recognition) The degree of effectiveness is yet to be verified

rawOsc=mean(max(0, diff(magSpec, 1, 2))); Spectral Flux spec2oscPlot.m magSpec rawOsc rawOsc=mean(max(0, diff(magSpec, 1, 2))); Order 1 Order 1 Dim 2

Gaussian Smoothing s is small  Smoothed OSC s is large  Trend of OSC

Example of OSC wave2osc.m

What Can You Do With OSC... OSC  onsets Pick peaks to have onsets OSC  tempo (BPM, beats per minute) Apply ACF (or other PDF) to find the BPM OSC  beat tracking Pick equal-spaced peaks to have beat positions Quiz!

Example of Beat Tracking beatTrack.m Identified beat pos. Max BPM=250 Min BPM=50 Tempo estimation via ACF 8 candidate sets for beat positions

Demo of Beat Tracking Demo link Examples http://mirlab.org/demo/beatTracking Examples 陪我看日出 分享

Performance Indices in Information Retrieval Figure source: http://bryannotes.blogspot.tw/2015/06/precisionrecall.html Desired Retrieved Quiz! References F1-score Precision & recall Information retrieval

Performance Indices of Beat Tracking Many performance indices of BT Check out audio beat tracking task of MIREX Mostly adopted ones Precision, recall, f-measure, accuracy Try simSequence.m in SAP toolbox Precision = tp/(tp+fp)=3/(3+4) = 0.428571 Recall = tp/(tp+fn)=3/(3+2) = 0.600000 F-measure = tp/(tp+(fn+fp)/2)=3/(3+(2+4)/2) = 0.500000 Accuracy = tp/(tp+fn+fp)=3/(3+2+4) = 0.333333 Quiz!