1 Roberto Bresin – Humaine Workshop S2S 2 – _21 Real-time Visualization of Musical Expression Roberto Bresin Speech, Music and Hearing Dep. - KTH

2 Roberto Bresin – Humaine Workshop S2S 2 – _21 Outlook Expressive music performance Acoustical cues of importance for communicating expressivity How acoustical cues can influence the performance style/intended emotion Real-time Visualization of Musical Expression Examples and demonstration

3 Roberto Bresin – Humaine Workshop S2S 2 – _21 The Musician  The musician Musical communication Modelling music performance Emotional colouring Applications The Listener Recognition of emotion Visualisation of musical expression

4 Roberto Bresin – Humaine Workshop S2S 2 – _21 The Score and the Performance How important is the performance? Dead-pan by computer and sampler Schumann’s Träumerei by Alfred Brendel IOI (%) Brendel Time deviation from score The musician  Musical communication Modelling music performance Emotional colouring Applications The Listener Recognition of emotion Visualisation of musical expression

5 Roberto Bresin – Humaine Workshop S2S 2 – _21 The Score and the Performance IOI (%) Brendel Time deviation from score IOI (%) Schnabel IOI (%) Horowitz 65 The musician  Musical communication Modelling music performance Emotional colouring Applications The Listener Recognition of emotion Visualisation of musical expression

6 Roberto Bresin – Humaine Workshop S2S 2 – _21 Design of Performance Rules Performance rules obtained mainly with 2 methods: analysis-by-synthesis analysis-by-measurement  Generative grammar for automatic music performance The musician Musical communication  Modelling music performance Emotional colouring Applications The Listener Recognition of emotion Visualisation of musical expression

7 Roberto Bresin – Humaine Workshop S2S 2 – _21 Dead-pan, K=0 Exaggerated, K = 4.4 Moderate, K = 2.2 Inverted, K = -2.2 Duration contrast rule Interonset Interval deviations (%)

8 Roberto Bresin – Humaine Workshop S2S 2 – _21 Phrase Arch Rule Dead-pan Exaggerated Δ IOI ( %)

9 Roberto Bresin – Humaine Workshop S2S 2 – _21 KTH Performance Rules Descriptions of different performance principles used by musicians General applicability K values change the overall quantity of each rule Context dependency ~ 30 rules 30 years of research at KTH Score Rules Performance K values The musician Musical communication  Modelling music performance Emotional colouring Applications The Listener Recognition of emotion Visualisation of musical expression

10 Roberto Bresin – Humaine Workshop S2S 2 – _21 KTH Performance Rules PhrasingPhrase arch Final ritardando Punctuation High loud Harmonic/melodic tensionHarmonic/melodic charge Repetitive patterns and groovesSwing ArticulationPunctuation Staccato/legato AccentsAccent rule Ensemble timingEnsemble swing Melodic sync The musician Musical communication  Modelling music performance Emotional colouring Applications The Listener Recognition of emotion Visualisation of musical expression

11 Roberto Bresin – Humaine Workshop S2S 2 – _21 Director Musices A program for modelling music performance The musician Musical communication  Modelling music performance Emotional colouring Applications The Listener Recognition of emotion Visualisation of musical expression

12 Roberto Bresin – Humaine Workshop S2S 2 – _21 Mapping from Emotional Expression to Rule Parameters For each emotion: Select a palette of rule parameters according to previous findings Mapping Emotional expression Rule parameters The musician Musical communication Modelling music performance  Emotional colouring Applications The Listener Recognition of emotion Visualisation of musical expression

13 Roberto Bresin – Humaine Workshop S2S 2 – _21 Cues for the simulation of emotions in music performance (by A.Gabrielsson and P.Juslin, Psychology of Music, 1996, vol. 24) No expression Tenderness Solemnity Happiness Sadness Anger Fear Synthesis of Emotional Expression The musician Musical communication Modelling music performance  Emotional colouring Applications The Listener Recognition of emotion Visualisation of musical expression

14 Roberto Bresin – Humaine Workshop S2S 2 – _21 TENDERNESS slow mean tempo (Ga96) slow tone attacks (Ga96) low sound level (Ga96) small sound level variability (Ga96) legato articulation (Ga96) soft timbre (Ga96) large timing variations (Ga96) accents on stable notes (Li99) soft duration contrasts (Ga96) final ritardando (Ga96) HAPPINESS fast mean tempo (Ga95) small tempo variability (Ju99) staccato articulation (Ju99) large articulation variability (Ju99) high sound level (Ju00) little sound level variability (Ju99) bright timbre (Ga96) fast tone attacks (Ko76) small timing variations (Ju/La00) sharp duration contrasts (Ga96) rising micro-intonation (Ra96) ANGER high sound level (Ju00) sharp timbre (Ju00) spectral noise (Ga96) fast mean tempo (Ju97a) small tempo variability (Ju99) staccato articulation (Ju99) abrupt tone attacks (Ko76) sharp duration contrasts (Ga96) accents on unstable notes (Li99) large vibrato extent (Oh96b) no ritardando (Ga96) SADNESS slow mean tempo (Ga95) legato articulation (Ju97a) small articulation variability (Ju99) low sound level (Ju00) dull timbre (Ju00) large timing variations (Ga96) soft duration contrasts (Ga96) slow tone attacks (Ko76) flat micro-intonation (Ba97) slow vibrato (Ko00) final ritardando (Ga96) FEAR staccato articulation (Ju97a) very low sound level (Ju00) large sound level variability (Ju99) fast mean tempo (Ju99) large tempo variability (Ju99) large timing variations (Ga96) soft spectrum (Ju00) sharp micro-intonation (Oh96b) fast, shallow, irregular vibrato (Ko00) Positive Valence Negative Valence High ActivityLow Activity From Juslin (2001)

15 Roberto Bresin – Humaine Workshop S2S 2 – _21 Example: SADNESS The musician Musical communication Modelling music performance  Emotional colouring Applications The Listener Recognition of emotion Visualisation of musical expression

16 Roberto Bresin – Humaine Workshop S2S 2 – _21 IOI deviations dB deviations articulation Example: SADNESS Model Score

17 Roberto Bresin – Humaine Workshop S2S 2 – _21 Conclusions Emotional expression can be derived directly from the music score, simply by enhancing music structure The musician Musical communication Modelling music performance  Emotional colouring Applications The Listener Recognition of emotion Visualisation of musical expression

18 Roberto Bresin – Humaine Workshop S2S 2 – _21 Better Monophonic Ringtones! Today (nom.) Natural :-) Happy >:-( Angry :-( Sad =|:-| Solemn Mozart G minor Dead-pan Happy Angry Natural Sad Solemn The musician Musical communication Modelling music performance Emotional colouring  Applications The Listener Recognition of emotion Visualisation of musical expression

19 Roberto Bresin – Humaine Workshop S2S 2 – _21 Usher, Burn Mechanical Musical Happy Jennifer Ellison, Bye Bye Boy Mechanical Musical Romantic Better Polyphonic Ringtones! The musician Musical communication Modelling music performance Emotional colouring  Applications The Listener Recognition of emotion Visualisation of musical expression

20 Roberto Bresin – Humaine Workshop S2S 2 – _21 pDM –performance rules in real-time Anders Friberg + MEGA project (IST EU) The musician Musical communication Modelling music performance Emotional colouring  Applications The Listener Recognition of emotion Visualisation of musical expression

21 Roberto Bresin – Humaine Workshop S2S 2 – _21 Real-time Visualization of Musical Expression

22 Roberto Bresin – Humaine Workshop S2S 2 – _21 Background Feel-Me project Design a computer program for teaching students to play expressively The system includes a tool for automatic extraction of acoustic cues (CUEX): pitch, duration, sound level, articulation, vibrato, attack velocity, spectrum The musician Musical communication Modelling music performance Emotional colouring Applications  The Listener Recognition of emotion Visualisation of musical expression

23 Roberto Bresin – Humaine Workshop S2S 2 – _21 Lens model: quantifies the expressive communication between performer and listener

24 Roberto Bresin – Humaine Workshop S2S 2 – _21 Aim Design a tool for real-time visual feedback to expressive performance  Mapping of acoustic cues: –Non-verbal –Intuitive –Informative (including emotional expression) Previous studies: cross-modality speeds stimuli discrimination The musician Musical communication Modelling music performance Emotional colouring Applications  The Listener Recognition of emotion Visualisation of musical expression

25 Roberto Bresin – Humaine Workshop S2S 2 – _21 Cue analysis Expression mapper AudioEmotion Tempo Sound level Articulation... Implementations CUEX Simplified real-time version Mult. Regression Fuzzy inspired Recognition of Emotion The musician Musical communication Modelling music performance Emotional colouring Applications The Listener  Recognition of emotion Visualisation of musical expression

26 Roberto Bresin – Humaine Workshop S2S 2 – _21 From audio to emotion recognition Standardisation Audio input Cue extraction Fuzzy mapping - +0 ∑ / 3 Happiness 0-1 tempo sound level articulation ∑ / 3 Sadness 0-1 ∑ / 3 Anger 0-1 Fuzzy mapping - +0 Fuzzy mapping - +0 The musician Introduction Modelling music performance Emotional colouring Applications The Listener  Recognition of emotion Body Expression Ghost in the Cave

27 Roberto Bresin – Humaine Workshop S2S 2 – _21 Experiment 2 melodies, Brahms (minor) & Haydn (Major) 3 instruments (piano, guitar, saxophone) 12 performances per instrument (12 emotional intentions) 24 colour nuances 8 levels of hue 2 levels of brightness 2 levels of saturation 2 groups of 11 subjects each (1 group per melody) The musician Musical communication Modelling music performance Emotional colouring Applications The Listener Recognition of emotion  Visualisation of musical expression

28 Roberto Bresin – Humaine Workshop S2S 2 – _21 Experiment: main results HUE Happiness  Yellow Fear  Blue Sadness  Violet & Blue Anger  Red Love  Blue & Violet BRIGHTNESS Observed tendency: Minor tonality  Low brightness (Dark colours) Major tonality  High brightness (Light colours )  Interaction involving sadness: Even for major tonality low brightness is preferred for all instruments The musician Musical communication Modelling music performance Emotional colouring Applications The Listener Recognition of emotion  Visualisation of musical expression

29 Roberto Bresin – Humaine Workshop S2S 2 – _21 Experiment: main results

30 Roberto Bresin – Humaine Workshop S2S 2 – _21 Experiment: main results

31 Roberto Bresin – Humaine Workshop S2S 2 – _21 Experiment: main results Similar colour palettes within instruments The musician Musical communication Modelling music performance Emotional colouring Applications The Listener Recognition of emotion  Visualisation of musical expression

32 Roberto Bresin – Humaine Workshop S2S 2 – _21 Experiment: main results

33 Roberto Bresin – Humaine Workshop S2S 2 – _21 The ExpressiBall Expressive performance space as a mapping of acoustical cues and emotions X  Tempo Color  Emotion Y  Sound levelShape  Articulation Z  Attack velocity & Spectrum energy Loud Soft SlowFast Staccato Angry Fast attack High energy Loud Soft SlowFast Legato Sad Slow attack Low energy

34 Roberto Bresin – Humaine Workshop S2S 2 – _21 The ExpressiBall DEMO!DEMO! The musician Musical communication Modelling music performance Emotional colouring Applications The Listener Recognition of emotion  Visualisation of musical expression

35 Roberto Bresin – Humaine Workshop S2S 2 – _21 ExpressiBall: Current & Future Work… Sonification of the ExpressiBall Usability test with students Set-up depending on instrument/spectrum Use complex colour palettes (Pictures? Abstract patterns/shapes?) Other possible applications: ”Colour Monitor” in discotheques Computer screen saver … The musician Musical communication Modelling music performance Emotional colouring Applications The Listener Recognition of emotion  Visualisation of musical expression

36 Roberto Bresin – Humaine Workshop S2S 2 – _21 Outlook Expressive music performance Acoustical cues of importance for communicating expressivity How the use of acoustical cues can influence the performance style/intended emotion Real-time Visualization of Musical Expression Examples and demonstration

37 Roberto Bresin – Humaine Workshop S2S 2 – _21 FET-Open Coordination Action (contract no. IST ) 11 partners (3 Humaine partners) Started: June 1, 2004 Duration: 3 years Overall objective: ”To bring together the state-of-the-art research in the sound domain and in the proper combination of human sciences, technological research and neuropsychological sciences that does relate to sound and sense”. Sound to Sense, Sense to Sound

38 Roberto Bresin – Humaine Workshop S2S 2 – _21 The End