1. Applications Statistical Graphical Models in Music Informatics Yushen Han Feb 10 2011 I548 Presentation

2. Statistical Graphical Models graph-based representation for a probabilistic distribution in high-dimensional space while specifying conditional independence structure – directed acyclic graph(DAG) - Bayesian Network – undirected graph(UG) - Markov Random Field – Mixed graph

3. Saturated (Undirected) Graph

4. Markov Condition on a Bayesian Network Markov Condition: X A and X B are conditionally independent given X S whenever S separates A and B From www.eecs.berkeley.edu/ewainwrig/ probabilistic distribution High- dimensional space Conditional independence

5. Musical Application: Bayesian Identification of Chord from Audio tuning octave root Chroma (pitch- class) mode key Peak frequencies and corresponding amplitude Attributes Observation: factorization of the joint by Bayesian Inference

6. Application I: Score Following/Alignment Music score (given) Performance audio of this piece (given) Establishing a correspondence between the two above Score Following: Online (real time) Score Alignment: Offline (no need to be in real time)

7. Application I: Score Following Best “guess” of the current location given what the computer had heard SINCE THE BEGINNING UP TO THE MOMENT

8. Observation - “Frames” of Audio Audio (waveform) Spectrogram (via Short-time Fourier transform)

9. Score Following Difficulties – Tempo rubato (expressive and rhythmic freedom) – Pitch / amplitude vibrato ( ) – Polyphony music – Noise – (occasional) wrong notes etc. – Realtime computational requirement Solutions – Assuming tempo change is smooth (mostly desirable) – Robust probabilistic data model on normalized semigram – Training to learn a prior (e.g. note length distribution) – Optimized particle filtering for 2-D State-space model

10. Proposed Solution with 2-D State- space Model (Bayesian Network) Assuming smooth tempo change for tempo rubato Two-dimensional state-space model Proposing a unit of tempi: S(t) - Musical time elapse per audio frame at frame t Interpretation: during one audio frame of fixed length (roughly 64ms, 512 samples at 8000Hz sampling rate), how much musical time (in terms of 1/384 notes) is elapsing In another word, how much of the score the performer covers every 64ms (not precisely the conventional tempo)

11. 2-D State-space Model State variables in one audio frame (notice conditional independence) Current “speed” at this frame 0.64ms Accumulative “speed” up to this frame – location

12. Physical Analogy – Integration of the Speed 1 1 1 1 1 2 3 4 speed location

13. Physical Analogy – Integration of the Speed with stochastic components 1.0 1.1 1.05 0.95 1.0 2.123.19 4.11 speed location Observation error also involved Speed fluctuation

14. Physical Analogy - to make a tractable problem 1.0 1.1 1.05 0.95 1.0 2.123.19 4.11 speed location Observation error also involved Speed fluctuation Assuming discrete state transition with {-0.1, -0.05, 0, 0.05, 0.1} with prob distribution {0.1, 0.2, 0.4,0.2, 0.1} for smoothness. Assuming Gaussian noise in observation

15. Relationship to Kalman Filter Particle filtering (Using White board)

16. Score Following – Data Model Data model using semigram Regarding discretized observation as a histogram Chord template (pre-learned)

17. Score Following – Demo in R Data model in R Visualization of results in XCode

18. Application II: Graph Model to Estimate Expert Pianists’ Perceptual Present - with the help of audio-score alignment technique

19. Background Curtain eras of classical music – no improvisation, no wrong notes etc. For a certain piece of music, performance varies in tempo, dynamic, articulation, vibrato etc., depending on the interpretation of the performer (This research) focuses on the tempo change of piano music

20. Chopin Mazurka Op. 30, No.2 http://www.youtube.com/watch?v=PjYV7lJezvc http://www.youtube.com/watch?v=vGAQONeLnXk http://www.youtube.com/watch?v=qJmaz1OEGTU Rubinstein Horowitz Michelangeli

21. Motivation Musical Perceptual Present – Recent studies in diverse fields of inquiry, including music philosophy and psychology, lend converging evidence that musical attention of both performers and listeners is primarily focused successively small “chunks” of material (hypothetically 2–10 seconds in the past) rather than larger formal relationships.

22. Motivation Instead of individual style, we are in search of a “common interpretation” shared among a collection of expert pianists Focus purely on tempo change per beat (since the attack of piano note is easy to capture).

23. Data Human corrected accumulative time per beat which is equivalent to IBI Inter Beat Interval N = 32 performances ( include different performances of the same pianist ) For the existence of the MLE, we proceed a small chunk of data at a time I = { 7, 8, 9, 10 }

24. Data cont. - preprocessing # !!!performance-id: pid9062-19 # !!!title: Mazurka in B minor, Op. 30, No. 2 # !!!trials: 1 # !!!date: 2007/02/15/ # !!!reverse-conductor: Craig Stuart Sapp # !!!performer: Idil Biret # !!!performance-date: 1990 # !!!label: Naxos 8.550359 # !!!label-title: Chopin: Mazurkas (Complete) # !!!offset:0 0.5780:3 1.3981:1 1.7081:2 2.2281:3 2.6682:1 3.0882:2 3.7482:3 4.3363:1 4.5883:2 4.9983:3 5.4984:1 5.8284:2 6.4284:3 7.0285:1 7.3555:2 7.9685:3 8.4286:1 8.8386:2 9.4186:3 9.8787:1 10.1587:2 10.6787:3 0.720 0.855 0.745 0.800 0.490 0.610 0.530 0.540 0.550 0.540 0.530 0.570 … Original Data: Accumulative time per beat InterBeatInterval (IBI) IBI difference between beats 0.135 -0.110 0.055 -0.310 0.120 -0.080 0.010 -0.010 …

25. Data cont. - preprocessing “Normalization” – Since the overall duration of each performance varies significantly E.g. Mazurka Op. 30. No. 2 We “stretch” the overall duration of each performance to line up with the median of all performances - can be problematic Sec. Performance index

26. Model For each “trunk” of timing data X of I dimensions (beats) across N performances: N performances are considered i.i.d. repetitions we assume: X ~ Ν( 0, Σ ) where the difference in IBI equals to 0 suggests that the tempo is nearly constant “on average” (of course, but could be problematic) We study the structure of I by I covariance matrix Σ Can obtain an estimate of

27. Model cont. – check the normality See the movie in R

28. Graph Models A toy example for I = 4 case -> 3 hypotheses – H: Fully saturated model (I-1=3 order Markov chain) – H 0 : A smaller model (I-2=2 order Markov chain) – H 00 : The smallest model (I-3=1 order Markov chain) diff. IBI

29. Graph Models cont. What does this graph mean? Conditional Independence!

30. Graph Models cont. Conditional independence in the graph suggests different structures in the covariance matrix ? ?? To apply reconstruction algorithm

31. Graph Models cont. - Testing Testing each pair of hypotheses -2Log(Q) ~ Accepting the result only when every single pair of hypotheses of the smallest difference between the alternative and the null hypotheses are not rejected (as small step as possible) Apply an appropriate degree of freedom ( = difference in number of edges between 2 graphs )

32. Results - Testing See R plot

33. Results – Interpretation “smoothed” results by using a sliding window of different lengths A “voting” mechanism Room to interpret …

34. Results – Interpretation “smoothed” results by using a sliding window of different lengths

35. Results – Interpretation “smoothed” results by using a sliding window of different lengths

36. Results – Interpretation Using “anchor points” to summarize the results