1. Performing expressive music using Case-Based Reasoning Ramon López de Mántaras IIIA - CSIC mantaras@iiia.csic.es www.iiia.csic.es/~mantaras

2. Outline Reminding CBR & Introducing Saxex main components Case representation –The musical knowledge Retrieval using perspectives Reuse –Fuzzy combination SaxEx Results TempoExpress Conclusions and future work

3. Case-based reasoning (CBR) Solving problems by means of examples of already solved similar problems (reasoning from precedents) The task of our system is to infer, via CBR and musical knowledge, a set of expressive transformations to be applied to the notes of inexpressive musical phrases given as input The precedents are examples of expressive human interpretations

4. Saxex Components

5. SMS Snapshot

6. Saxex-CBR

7. Outline Reminding CBR & Introducing Saxex main components Case representation –The musical knowledge Retrieval using perspectives Reuse –Fuzzy combination SaxEx Results TempoExpress Conclusions and future work

8. Case representation Score Musical knowledge –implication-realization, metrical structure, time-span reduction & prolongational reduction Performance representation (solution description) sound transformation operations: –eg: high dynamics, medium rubato, very legato, etc. SOLUTIONSOLUTION

9. Transformations Transformations (for each note) –Dynamics (5 possible values) –Rubato (5 possible values) –Vibrato (5 possible values) ----->1250 possibilities –Articulation (5 possible values) –Attack (2 possible values) Vibr. Din. Rub Art.

10. Score

11. Musical knowledge Implication/Realization model (Narmour) –Basic structures: –Melodic direction, durational cumulation GTTM theory (Lerdahl & Jackendoff) –Metrical structure (metrical strength of notes) –Time-span reduction (relative importance of notes within phrases or sub-phrases) –Prolongational reduction (tensions, relaxations) Jazz Theory –Harmonic Progressions (duration, harmonic stability)

12. Implication/Realization Model

13. GTTM Theory

14. Performance

15. Outline Reminding CBR & Introducing Saxex main components Case representation –The musical knowledge Retrieval using perspectives Reuse –Fuzzy combination SaxEx Results TempoExpress Conclusions and future work

16. A Retrieval Perspective

18. Case Memory Problem IdentifySearchSelect Retrieval Example

19. Outline Reminding CBR & Introducing Saxex main components Case representation –The musical knowledge Retrieval using perspectives Reuse –Fuzzy combination SaxEx Results TempoExpress Conclusions and future work

20. Saxex-Reuse Transformations –Dynamics –Rubato –Vibrato –Articulaction –Attack Criteria –Most similar –Majority –Minority –Continuity –Random –Fuzzy combination (DEFAULT) Vibr. Din. Rub Art.

21. Problem Din. Rub Art. Single case retrieved Din. Rub Art. Saxex-Reuse Example

22. Saxex-Reuse (Fuzzy Combination) 20320Tempo 0 1 Very Low LowMedium High High The notes in the human-performed musical phrases are qualified by means of five ordered linguistic values. Those for rubato are: Assume that SaxEx has retrieved and selected two notes whose rubato values are 72 and 190 respectively. The fuzzy combination followed by a defuzzification gives the rubato value to be applied to the input note: 72 123 190 0.9 0.7 COA LowMedium

23. Outline Reminding CBR & Introducing Saxex main components Case representation –The musical knowledge Retrieval using perspectives Reuse –Fuzzy combination SaxEx Results TempoExpress Conclusions and future work

24. Saxex Results Autumn Leaves Inexpressive Input phrase Expressive Output phrase SaxEx

25. Affective Labels Three orthogonal dimensions –Tender-Aggressive –Sad-Joyful –Calm-Restless Relating to notions such as –activity –tension vs. relaxation –Brightness...

26. Inexpressive Input phrase SaxEx Results SaxEx Aff. values Joyful Sad All of me

27. Reminding CBR & Introducing Saxex main components Case representation –The musical knowledge Retrieval using perspectives Reuse –Fuzzy combination SaxEx Results TempoExpress Conclusions and future work

28. Goal: –Changing the original performing tempo of a melody, preserving expressiveness, in the context of jazz standards. Application: Audio editing software Video / Audio post-production (video constrains audio) Why not applying uniform time stretching to the audio? Timing of notes w.r.t. beat may have to change Other expressive phenomena (e.g. ornamentations, consolidations, fragmentations) may have to change as a function of the tempo TempoExpress

29. Musical explanation: Expressivity is a result of the conception of the music by the performer, and this conception changes with tempo [Desain & Honing, 1994] Original tempo (180 ) Transformed tempo (90) Uniform time stretching Melody: “Up Jumped Spring” Recording TempoExpress

30. Some basic music performance concepts and their relations Expressive Transformations

31. Onset deviations at different tempos (Body and Soul A1)

32. “Hand crafted” –Let a music expert formulate rules for music performance (Friberg, CMJ 1991, Friberg et al. CMJ 2000) Machine learned –Derive expressivity rules automatically from examples (Widmer, ICMC 2000, JNMR 2002) Eager approach: Builds a model based on many training examples and uses the learned model to solve new problems –Imitate expressivity using examples of concrete human performances by means of CBR (Arcos & Lopez de Mantaras, JNMR 1998, Lopez de Mantaras & Arcos, AI Mag 2002) Lazy approach: Take the solution of the training example that resembles most to the new problem, and adapt it to solve it “That an expressive effect is applied only once does not mean it is insignificant” (Sundberg, MP 2001) Approches to expressive music generation

33. TempoExpress Architecture Desired Tempo

34. Performance Annotation Expressivity in jazz is more than timing / dynamics deviations. It is also spontaneous note ornamentations, fragmentations, etc. To model this, we define a set of Performance Events: And we use them as edit operations to obtain an edit-distance-based alignment between the score and the performance

35. Goal of the annotation process –Automatic case base acquisition Comparing Score vs recordings

36. Body and Soul Once I Loved Examples F C C I I

37. Goal: Assessing the distance between two sequences –Calculated as the minimal cost of transforming S 1 into S 2 –Requires: Edit operations Cost functions Edit (Levenshtein) distance

38. R R RI

39. TTF Case Annotation examples (I)

40. TT T CTC T Case Annotation examples (II)

41. T TT T I Case Annotation examples (III)

42. Rationale: the expressivity of a performed note is not just determined by the note itself. Ergo: Some representation of the melodic context of the note is needed We use the Implication / Realization model of melodic structure (Narmour, 1990) –It captures the pattern of fulfillment / violation of expectations created by the melodic surface –Groups notes based on gestalt principles Representing melodic context

43. Repeated for each tempo Case Representation

44. 1. Filter cases by tempo: keep cases containing performances at relevant tempos (one of the tempos is similar to the original tempo of the target melody and there is another performed tempo similar to the desired tempo to which the target melody has to be transformed) 2. Rank the cases that passed the previous filter by I/R similarity to the score of the target melody (using edit-distance) 3. Partition the phrases of the most similar cases into segments using the I/R parser or any other melodic segmentation algorithm (for instance Temperley, 2001) 4. Form a “new” case base containing the obtained segments (space of partial solutions) as cases Retrieval

45. Solutions for the target melody are generated segment-wise via a best first search through the space of partial solutions (segments) Procedure: 1. Retrieve best matching segment (using edit-distance) 2. Align target melody and retrieved segment 3. Transfer performance events For aligned notes T and R, let T i (R) -----> T o (R) represent the tempo transformation of note R; use the annotations differences between T i (R) and T o (R) to generate the solution T o (T) from T i (T) 4.For non-aligned target notes use UTS to transform T i (T) into T o (T) Reuse

46. TempoExpress overall view

47. Uniform time stretching CBR Human 55 100 bpm Example of TempoExpress Result

48. Four jazz standards recordings by a professional musician (12 tempos for each: 48 recordings) 14 different phrases containing a total of 64 different melodic segments More than 8000 tempo-transformation problems in the case base Experimental comparison to UTS

49. TempoExpress vs. UTS as a function of the ratio of original tempo to transformed tempo. The lower plot shows the probability of incorrectly rejecting the hypothesis (that there is no difference between TempoExpress and UTS) for the Wilcoxon signed-rank test.

50. Conclusions & Future CBR is a powerful technique to imitate human solutions (performances): Human-like output SaxEx successfully retrieves relevant cases Fuzzy combination increases output variation SaxEx as a pedagogical tool: –Users can experiment with the system –Helps understanding how to use the different expressive resources TempoExpres: an application to audio post-production that clearly outperforms UTS Further TempoExpress experimentation with fast tempos (more example cases at fast tempos are needed) Add within-note descriptions: –Energy envelpe features: attack, sustain, decay, tremolo –Pitch envelope features: vibrato, glissando Add between-notes descriptions: –Articulation (legato,, staccato)