1. 1 Roberto Bresin - An Introduction to Affective Music – 2004.08.30 An Introduction to Affective Music, Theory and Some Applications Roberto Bresin http://www.speech.kth.se/music/performance

2. 2 Roberto Bresin - An Introduction to Affective Music – 2004.08.30 Outlook Aim To explain how it is possible to communicate different emotions with the same music score Part I: The science of music performance Analysis & synthesis of music performance The most important techniques for measuring and modelling a performance The acoustical cues of importance for communicating expressivity How the use of acoustical cues can influence the performance style A rule-based system for the synthesis of music performance Part II: Emotion in music performance Emotionally expressive music performance Real-time Visualization of Musical Expression Examples Applications –Emotional colouring of music performance –expressive ringtones in mobile phones –visual display of emotion in music performance

3. 3 Roberto Bresin - An Introduction to Affective Music – 2004.08.30 Part I The Science of Music Performance

4. 4 Roberto Bresin - An Introduction to Affective Music – 2004.08.30 Musical Communication composer musician instrument listener scoregestures sound Computational models

5. 5 Roberto Bresin - An Introduction to Affective Music – 2004.08.30 The Musician  The musician Musical communication Modelling music performance Emotional colouring Applications The Listener Recognition of emotion Visualisation of musical expression

6. 6 Roberto Bresin - An Introduction to Affective Music – 2004.08.30 What is communicated? The music Emotions Imagined and real motion … The musician  Musical communication Modelling music performance Emotional colouring Applications The Listener Recognition of emotion Visualisation of musical expression

7. 7 Roberto Bresin - An Introduction to Affective Music – 2004.08.30 Score - Notes, harmony, melody, rhythm, pitch, texture, instruments Performance - Tempo, phrasing, articulation, intonation Auditive Environment - Concert, club, home, live/recording Audience Social/cultural Body movements and gestures People, clothes, stage lightning, etc. Visual Memory Musical knowledge Past experience Different factors in the communication

8. 8 Roberto Bresin - An Introduction to Affective Music – 2004.08.30 What can be studied? What is a musical performance? Emotional communication: Accuracy, musical factors Emotional affect Couplings to motion: Musicians gestures and the resulting sound Visual perception of musicians body movements The musician  Musical communication Modelling music performance Emotional colouring Applications The Listener Recognition of emotion Visualisation of musical expression

9. 9 Roberto Bresin - An Introduction to Affective Music – 2004.08.30 The Score and the Performance How important is the performance? Dead-pan by computer and sampler Schumann’s Träumerei by Alfred Brendel IOI (%) Brendel Time deviation from score The musician  Musical communication Modelling music performance Emotional colouring Applications The Listener Recognition of emotion Visualisation of musical expression

10. 10 Roberto Bresin - An Introduction to Affective Music – 2004.08.30 The Score and the Performance IOI (%) Brendel Time deviation from score IOI (%) Schnabel IOI (%) Horowitz 65 The musician  Musical communication Modelling music performance Emotional colouring Applications The Listener Recognition of emotion Visualisation of musical expression

11. 11 Roberto Bresin - An Introduction to Affective Music – 2004.08.30 Collecting Data of Expressive Performances Expert musicians (Lars Frydén for KTH) –Expertise is translated into rules Measurements of recorded performances –Commercial recordings (CDs) –Computer controlled acoustical instruments (Disklavier, Böserndorfer) The musician  Musical communication Modelling music performance Emotional colouring Applications The Listener Recognition of emotion Visualisation of musical expression

12. 12 Roberto Bresin - An Introduction to Affective Music – 2004.08.30 Design of Performance Rules Performance rules obtained mainly with 2 methods: analysis-by-synthesis analysis-by-measurement  Generative grammar for automatic music performance The musician Musical communication  Modelling music performance Emotional colouring Applications The Listener Recognition of emotion Visualisation of musical expression

13. 13 Roberto Bresin - An Introduction to Affective Music – 2004.08.30 MUSIC SCORE (MIDI) PERFORMED MUSIC (MIDI) DIRECTOR MUSICES (performance rules) PROGRAMMER PROFESSIONAL MUSICIAN NEW / MODIFIED RULE K values (Rule quantity) Analysis-by-Synthesis of Music Performance

14. 14 Roberto Bresin - An Introduction to Affective Music – 2004.08.30 Dead-pan, K=0 Exaggerated, K = 4.4 Moderate, K = 2.2 Inverted, K = -2.2 Duration contrast rule Interonset Interval deviations (%)

15. 15 Roberto Bresin - An Introduction to Affective Music – 2004.08.30 Analysis-by-Measurement Designing Articulation Rules 3 main classes of articulation: Legato (overlapping) Staccato (detaching) Repetition The musician Musical communication  Modelling music performance Emotional colouring Applications The Listener Recognition of emotion Visualisation of musical expression

16. 16 Roberto Bresin - An Introduction to Affective Music – 2004.08.30 Legato and Staccato Tones IOI = Inter-onset-Interval DR = Tone Duration KOT = Key Overlap Time KDT = Key Detached Time

17. 17 Roberto Bresin - An Introduction to Affective Music – 2004.08.30 Data Natural Glittering Dark Heavy Light Hard Soft Passionate Flat 5 pianists played the same score 9 times on a Disklavier The first sixteen bars of the Andante movement of W A Mozart’s Piano Sonata in G major, K 545 1 pianist played 13 Mozart piano sonatas on a computer-monitored Bösendorfer

18. 18 Roberto Bresin - An Introduction to Affective Music – 2004.08.30 Mean Legato (KOR)  Legato articulation rule

19. 19 Roberto Bresin - An Introduction to Affective Music – 2004.08.30 Mean Staccato (Key Detached Ratio, KDR)  Staccato articulation rule 19

20. 20 Roberto Bresin - An Introduction to Affective Music – 2004.08.30 Context Influence in Staccato Production Amount of staccato (KDR) in different contexts for the 2nd note in a three notes pattern. (S = staccato note, N = Non–staccato note)

21. 21 Roberto Bresin - An Introduction to Affective Music – 2004.08.30 Legato and walking Staccato and running Control model for step sounds Legato and Staccato Allude to Walking and Running?

22. 22 Roberto Bresin - An Introduction to Affective Music – 2004.08.30 WalkingRunning Footsteps

23. 23 Roberto Bresin - An Introduction to Affective Music – 2004.08.30 Controlling Footsteps Pd model for crumpling sounds controlled with performance rules The musician Musical communication  Modelling music performance Emotional colouring Applications The Listener Recognition of emotion Visualisation of musical expression

24. 24 Roberto Bresin - An Introduction to Affective Music – 2004.08.30 KTH Performance Rules Descriptions of different performance principles used by musicians General applicability K values change the overall quantity of each rule Context dependency ~ 30 rules 30 years of research at KTH Score Rules Performance K values The musician Musical communication  Modelling music performance Emotional colouring Applications The Listener Recognition of emotion Visualisation of musical expression

25. 25 Roberto Bresin - An Introduction to Affective Music – 2004.08.30 Director Musices A program for modelling music performance http://www.speech.kth.se/music/performance The musician Musical communication  Modelling music performance Emotional colouring Applications The Listener Recognition of emotion Visualisation of musical expression

26. 26 Roberto Bresin - An Introduction to Affective Music – 2004.08.30 Performance Rules PhrasingPhrase arch Final ritardando Punctuation High loud Harmonic/melodic tensionHarmonic/melodic charge Repetitive patterns and groovesSwing ArticulationPunctuation Staccato/legato AccentsAccent rule Ensemble timingEnsemble swing Melodic sync The musician Musical communication  Modelling music performance Emotional colouring Applications The Listener Recognition of emotion Visualisation of musical expression

27. 27 Roberto Bresin - An Introduction to Affective Music – 2004.08.30 Phrase Arch Rule Dead-pan Exaggerated Δ IOI ( %)

28. 28 Roberto Bresin - An Introduction to Affective Music – 2004.08.30 Moderate Inverted Phrase Arch Rule Δ IOI ( %)

29. 29 Roberto Bresin - An Introduction to Affective Music – 2004.08.30 The DM system (~30 rules) Differentiation Rules Example: Duration Contrast Rule Grouping Rules Example: Phrase Articulation Rule Synchronization/Ensemble Rules Example: Ensemble timing Other Rules Example: Repetition Articulation Rule The musician Musical communication  Modelling music performance Emotional colouring Applications The Listener Recognition of emotion Visualisation of musical expression

30. 30 Roberto Bresin - An Introduction to Affective Music – 2004.08.30 Part II Emotion in Music Performance

31. 31 Roberto Bresin - An Introduction to Affective Music – 2004.08.30 Happy or sad music?

32. 32 Roberto Bresin - An Introduction to Affective Music – 2004.08.30 Mapping from Emotional Expression to Rule Parameters For each emotion: Select a palette of rule parameters according to previous findings Mapping Emotional expression Rule parameters The musician Musical communication Modelling music performance  Emotional colouring Applications The Listener Recognition of emotion Visualisation of musical expression

33. 33 Roberto Bresin - An Introduction to Affective Music – 2004.08.30 Cues for the simulation of emotions in music performance (by A.Gabrielsson and P.Juslin, Psychology of Music, 1996, vol. 24) No expression Tenderness Solemnity Happiness Sadness Anger Fear Synthesis of Emotional Expression The musician Musical communication Modelling music performance  Emotional colouring Applications The Listener Recognition of emotion Visualisation of musical expression

34. 34 Roberto Bresin - An Introduction to Affective Music – 2004.08.30 TENDERNESS slow mean tempo (Ga96) slow tone attacks (Ga96) low sound level (Ga96) small sound level variability (Ga96) legato articulation (Ga96) soft timbre (Ga96) large timing variations (Ga96) accents on stable notes (Li99) soft duration contrasts (Ga96) final ritardando (Ga96) HAPPINESS fast mean tempo (Ga95) small tempo variability (Ju99) staccato articulation (Ju99) large articulation variability (Ju99) high sound level (Ju00) little sound level variability (Ju99) bright timbre (Ga96) fast tone attacks (Ko76) small timing variations (Ju/La00) sharp duration contrasts (Ga96) rising micro-intonation (Ra96) ANGER high sound level (Ju00) sharp timbre (Ju00) spectral noise (Ga96) fast mean tempo (Ju97a) small tempo variability (Ju99) staccato articulation (Ju99) abrupt tone attacks (Ko76) sharp duration contrasts (Ga96) accents on unstable notes (Li99) large vibrato extent (Oh96b) no ritardando (Ga96) SADNESS slow mean tempo (Ga95) legato articulation (Ju97a) small articulation variability (Ju99) low sound level (Ju00) dull timbre (Ju00) large timing variations (Ga96) soft duration contrasts (Ga96) slow tone attacks (Ko76) flat micro-intonation (Ba97) slow vibrato (Ko00) final ritardando (Ga96) FEAR staccato articulation (Ju97a) very low sound level (Ju00) large sound level variability (Ju99) fast mean tempo (Ju99) large tempo variability (Ju99) large timing variations (Ga96) soft spectrum (Ju00) sharp micro-intonation (Oh96b) fast, shallow, irregular vibrato (Ko00) Positive Valence Negative Valence High ActivityLow Activity From Juslin (2001)

35. 35 Roberto Bresin - An Introduction to Affective Music – 2004.08.30 Lens model: quantifies the expressive communication between performer and listener

36. 36 Roberto Bresin - An Introduction to Affective Music – 2004.08.30 Example: SADNESS The musician Musical communication Modelling music performance  Emotional colouring Applications The Listener Recognition of emotion Visualisation of musical expression

37. 37 Roberto Bresin - An Introduction to Affective Music – 2004.08.30 IOI deviations dB deviations articulation Example: SADNESS Model Score

38. 38 Roberto Bresin - An Introduction to Affective Music – 2004.08.30 Synthesis of Emotion: Listening Test Results The musician Musical communication Modelling music performance  Emotional colouring Applications The Listener Recognition of emotion Visualisation of musical expression

39. 39 Roberto Bresin - An Introduction to Affective Music – 2004.08.30 Conclusions Emotional expression can be derived directly from the music score, simply by enhancing music structure The musician Musical communication Modelling music performance  Emotional colouring Applications The Listener Recognition of emotion Visualisation of musical expression

40. 40 Roberto Bresin - An Introduction to Affective Music – 2004.08.30 Better Monophonic Ringtones! Today (nom.) Natural :-) Happy >:-( Angry :-( Sad =|:-| Solemn Mozart G minor Dead-pan Happy Angry Natural Sad Solemn www.notesenses.com The musician Musical communication Modelling music performance Emotional colouring  Applications The Listener Recognition of emotion Visualisation of musical expression

41. 41 Roberto Bresin - An Introduction to Affective Music – 2004.08.30 Usher, Burn Mechanical Musical Happy Jennifer Ellison, Bye Bye Boy Mechanical Musical Romantic Better Polyphonic Ringtones! www.notesenses.com The musician Musical communication Modelling music performance Emotional colouring  Applications The Listener Recognition of emotion Visualisation of musical expression

42. 42 Roberto Bresin - An Introduction to Affective Music – 2004.08.30 pDM –performance rules in real-time Anders Friberg + MEGA project (IST EU) The musician Musical communication Modelling music performance Emotional colouring  Applications The Listener Recognition of emotion Visualisation of musical expression

43. 43 Roberto Bresin - An Introduction to Affective Music – 2004.08.30 Real-time Visualization of Musical Expression

44. 44 Roberto Bresin - An Introduction to Affective Music – 2004.08.30 Background Feel-Me project Design a computer program for teaching students to play expressively The system includes a tool for automatic extraction of acoustic cues (CUEX): pitch, duration, sound level, articulation, vibrato, attack velocity, spectrum The musician Musical communication Modelling music performance Emotional colouring Applications  The Listener Recognition of emotion Visualisation of musical expression

45. 45 Roberto Bresin - An Introduction to Affective Music – 2004.08.30 Aim Design a tool for real-time visual feedback to expressive performance  Mapping of acoustic cues: –Non-verbal –Intuitive –Informative (including emotional expression) Previous studies: cross-modality speeds stimuli discrimination The musician Musical communication Modelling music performance Emotional colouring Applications  The Listener Recognition of emotion Visualisation of musical expression

46. 46 Roberto Bresin - An Introduction to Affective Music – 2004.08.30 Cue analysis Expression mapper AudioEmotion Tempo Sound level Articulation... Implementations CUEX Simplified real-time version Mult. Regression Fuzzy inspired Recognition of Emotion The musician Musical communication Modelling music performance Emotional colouring Applications The Listener  Recognition of emotion Visualisation of musical expression

47. 47 Roberto Bresin - An Introduction to Affective Music – 2004.08.30 Experiment 2 melodies, Brahms (minor) & Haydn (Major) 3 instruments (piano, guitar, saxophone) 12 performances per instrument (12 emotional intentions) 24 colour nuances 8 levels of hue 2 levels of brightness 2 levels of saturation 2 groups of 11 subjects each (1 group per melody) The musician Musical communication Modelling music performance Emotional colouring Applications The Listener Recognition of emotion  Visualisation of musical expression

48. 48 Roberto Bresin - An Introduction to Affective Music – 2004.08.30 Experiment: main results HUE Happiness  Yellow Fear  Blue Sadness  Violet & Blue Anger  Red Love  Blue & Violet BRIGHTNESS Observed tendency: Minor tonality  Low brightness (Dark colours) Major tonality  High brightness (Light colours ) Interaction involving sadness: Even for major tonality low brightness is preferred for all instruments The musician Musical communication Modelling music performance Emotional colouring Applications The Listener Recognition of emotion  Visualisation of musical expression

49. 49 Roberto Bresin - An Introduction to Affective Music – 2004.08.30 Experiment: main results Similar colour palettes within instruments The musician Musical communication Modelling music performance Emotional colouring Applications The Listener Recognition of emotion  Visualisation of musical expression

50. 50 Roberto Bresin - An Introduction to Affective Music – 2004.08.30 Experiment: main results

51. 51 Roberto Bresin - An Introduction to Affective Music – 2004.08.30 The ExpressiBall Expressive performance space as a mapping of acoustical cues and emotions X  Tempo Color  Emotion Y  Sound levelShape  Articulation Z  Attack velocity & Spectrum energy Loud Soft SlowFast Staccato Angry Fast attack High energy Loud Soft SlowFast Legato Sad Slow attack Low energy

52. 52 Roberto Bresin - An Introduction to Affective Music – 2004.08.30 The ExpressiBall DEMO!DEMO! The musician Musical communication Modelling music performance Emotional colouring Applications The Listener Recognition of emotion  Visualisation of musical expression

53. 53 Roberto Bresin - An Introduction to Affective Music – 2004.08.30 ExpressiBall: Current & Future Work… Sonification of the ExpressiBall Set-up depending on instrument/spectrum Use complex colour palettes (pictures?) Usability test with students Other possible applications: ”Colour Monitor” in discotheques Computer screen saver … The musician Musical communication Modelling music performance Emotional colouring Applications The Listener Recognition of emotion  Visualisation of musical expression

54. 54 Roberto Bresin - An Introduction to Affective Music – 2004.08.30 Outlook Aim To explain how it is possible to communicate different emotions with the same music score Part I: The science of music performance Analysis & synthesis of music performance The most important techniques for measuring and modelling a performance The acoustical cues of importance for communicating expressivity How the use of acoustical cues can influence the performance style A rule-based system for the synthesis of music performance Part II: Emotion in music performance Emotionally expressive music performance Real-time Visualization of Musical Expression Examples Applications –Emotional colouring of music performance –expressive ringtones in mobile phones –visual display of emotion in music performance

55. 55 Roberto Bresin - An Introduction to Affective Music – 2004.08.30 The End