1. Slide 1 (of 45) A System for Detecting and Influencing the Emotions of Music in Computer Mediated Performance Steven R. Livingstone (BSc. BInfTech)

2. Slide 2 (of 45) Problem Statement There exists no automated method to detect and influence the emotions of music An audience’s response to computer music has previously been inaccessible to computers and thus lost

3. Slide 3 (of 45) Hypothesis Perceived emotional content of music can be influenced by controlling both the structural and performative aspects Audience feedback can be captured by computers to tailor the musical experience

4. Slide 4 (of 45) Methodology Score – Manipulate the structure and mark up with emotional performance metadata Audience – Determine emotional state and attitudes using affective computing tech. Architecture – Bring together for an awareness of score and audience

5. Slide 5 (of 45) Talk Overview 1) Introduction- In 2) Research and Contribution- Re 3) System& Testing- Sy 4) Future Work- Fu 5) Summary- Su

6. Slide 6 (of 45) Why Emotions? Principal target is the computer game Emotional Narrative is the key to game enjoyment In 1

7. Slide 7 (of 45) Why Music? Music is a universal human trait and found everywhere [a] : cinema, television, radio, commercials, ballet, shopping centres, transport, waiting rooms, restaurants … Within any waking 2 hour period, a person has a 44% chance of experiencing a musical event [b] In 2

8. Slide 8 (of 45) Why Gaming Music? Gaming music is important for: emotion, interest and information Game music static within scenes, unlike cinema music In 3

9. Slide 9 (of 45) How are we going to do it? Cross-disciplinary approach to research Study of emotion Music psychology Empirical Analysis Bring this knowledge together into a computing framework In 4

10. Slide 10 (of 45) Emotion - History 1600s – Primary Emotions (Descartes) 1800s - Biological reaction mechanisms (Darwin) 1800/1900s – Perception  Physiological Response  Emotion (James-Lange) 1960s – Cognition and Appraisal Theory (Arnold and Lazarus) 1990s – Somatic Markers (Damásio) R 1

11. Slide 11 (of 45) Emotion – Perceived VS Induced Perceived Emotion – The emotion observer believes the source or stimuli is experiencing or expressing Induced Emotion – The emotion felt by the observer as a result of the stimuli (very hard to capture/use) Fearful Speech R 2

12. Slide 12 (of 45) Emotion - Representation and Capture Required for Empirical analysis Computational Implementation Requirements Continuous capture over time Continuous representation of emotion (numerical) Data Consistency R 3

13. Slide 13 (of 45) Emotion - Representation and Capture Existing Methods TypeContinuous Capture Continuous Representation Consistent Open-EndedYes and NoPartialNo ChecklistYes and NoNoYes Rank & MatchYes and NoNo Rating ScalePartialYesPartial R 4

14. Slide 14 (of 45) Emotion – Dimensional Approach Originally proposed by Wilhelm Wundt in the 1800s We choose 2D with Arousal & Valence Arousal: Active  Passive Valence: Positive  Negative 2 dimensions offer a balance between ease of reporting and data richness [c] R 5

15. Slide 15 (of 45) Emotion - Representation and Capture 2 Dimensional Emotion Space [d] R 6

16. Slide 16 (of 45) Music Emotion Rules Need to influence the emotions somehow … Over a century of empirical music psychology has investigated the link between music  emotion [e] Two types Structural – Modifying the score Performative – Those applied by the performer when converting the score to audio R 7

17. Slide 17 (of 45) Music Emotion Rules – Structural Structural Music Rules An understanding of the musical structure Simplified emotional grouping (octants) and testing Varying degrees of musical theory required A New Approach [1] R 8

18. Slide 18 (of 45) OctantStructural Music Emotion Rules [2] 1 (happy) Mode Major(19), Tempo Fast(16), Harmony Simple(8), Loudness Loud(7), Articulation Staccato(5), Pitch High(3), Rhythm Flowing(3), Pitch Range High(2), Pitch Variation Large(2), Pitch Contour Up(2), Note Onset Rapid(2), Rhythm Smooth(2), Rhythm Activity(2), Loudness Medium(1), Loudness Soft(1), Loudness Variation Small(1), Loudness Variation Rapid(1), Loudness Variation Few(1), Pitch Low(1), Pitch Range Low(1), Pitch Contour Down(1), Timbre Few(1), Rhythm Rough(1) 3 (angry) Mode Minor(14), Loudness Loud(9), Tempo Fast(9), Harmony Complex(8), Note Onset Rapid(5), Pitch Contour Up(5), Pitch High(4), Pitch Range High(3), Pitch Variation Large(3), Loudness Soft(2), Rhythm Complex(2), Loudness Variation Large(2), Timbre Sharp(2), Articulation Non-legato(2), Pitch Variation Small(2), Articulation Staccato(2), Note Onset Slow(2), Timbre Many(1), Vibrato Fast(1), Rhythm Rough(1), Metre Triple(1), Tonality Tonal(1), Tonality Atonal(1), Tonality Chromatic(1), Loudness Variation Rapid(1), Pitch Low(1) 4 Mode Minor(12), Harmony Complex(6), Articulation Legato(3), Pitch Variation Small(3), Tempo Fast(3), Loudness Loud(2), Loudness Soft(2), Loudness Variation Large(2), Note Onset Rapid(2), Note Onset Sharp(2), Note Onset Slow(2), Timbre Sharp(2), Loudness Variation Rapid(1), Pitch High(1), Pitch Low(1), Pitch Range High(1), Pitch Variation Large(1), Pitch Contour Up(1), Pitch Contour Down(1), Timbre Many(1), Harmony Melodic(1), Tempo Slow(1), Articulation Staccato(1), Rhythm Complex(1), Tonality Atonal(1), Tonality Chromatic(1) 6 (dreamy) Loudness Soft(5), Tempo Slow(5), Pitch Variation Small(3), Articulation Legato(3), Note Onset Slow(3), Pitch Low(3), Pitch Range Low(2), Loudness Variation Rapid(1), Pitch High(1), Pitch Contour Down(1), Mode Minor(1), Timbre Few(1), Harmony Complex(1), Vibrato Deep(1), Metre Duple(1), Tonality Tonal(1) 7 Tempo Slow(10), Loudness Soft(9), Articulation Legato(5), Note Onset Slow(3), Pitch Low(2), Pitch Range Low(2), Pitch Variation Small(2), Timbre Soft(2), Harmony Simple(2), Mode Minor(1), Loudness Variation Rapid(1), Loudness Variation Few(1), Pitch High(1), Note Onset Rapid(1), Vibrato Intense(1), Rhythm Smooth(1), Rhythm Flowing(1), Rhythm Firm(1), Metre Duple(1)

19. Slide 19 (of 45) Music Emotion Rules – Structural Primary Music Emotion Structural Rules [2] Can you hear them? Quad 1 (happy) Quad 2 (angry) Quad 3 (sad) Quad 4 (dreamy->bliss) R 10

20. Slide 20 (of 45) Music Emotion Rules – Performative Performative Music Rules Requires fine-grained, continuous capture of emotion for testing Waveform modification (very complex stuff) Already been done (partially) R 11

21. Slide 21 (of 45) Music Emotion Rules – Performative Performance rules to accentuate emotion … Chord asynchrony (melody lead or lag) Rubato (especially at phrase boundaries) Melody notes louder Increase dynamic range (gradient) Increase vibrato amplitude Structural Structural + Performative R 12

22. Slide 22 (of 45) Music Emotion Rules – Vocal What does this table mean? OctantPerformanceVocalStructural 2 (Happy) Fast mean tempoFast speech rate/tempo Tempo Fast Fast tone attacksFast voice tone attacksNote Onset Rapid High sound levelMedium–high voice sound level Loudness Loud Much pitch variabilityPitch Variation Large High pitch levelPitch High Rising pitch contourPitch Contour Up Staccato articulationArticulation Staccato R 13

23. Slide 23 (of 45) Music Tension and Induced Emotion Is musical emotion really just 2 dimensional? Perceived maybe, induced definitely not Meyer believed that musical emotion is the inhibition or completion of musical expectations [f] “Tense Sad”, breaks the rules … important! R 14

24. Slide 24 (of 45) Audience Consideration Awareness of Audience Plays an important role in performances Affective Computing Attitudes User state R 15

25. Slide 25 (of 45) Audience Consideration Attitudes are a cognitive powerful tool: Quickly categorise data Influence decision making Relatively static R 16

26. Slide 26 (of 45) Audience Consideration User State A listeners response to the stimulus Guides the performer Continuous feedback Affective Computing Research from MIT Various mechanisms to detect AND affect R 17

27. Slide 27 (of 45) Research  Something Real Phew! A lot of research Many topics not examined today … What were we trying to do again? R 18

28. Slide 28 (of 45) Hypothesis Perceived emotional content of music can be influenced by controlling both the structural and performative aspects Audience feedback can be captured by computers to tailor the musical experience

29. Slide 29 (of 45) The Rule System – Influence Influencing perceived emotions E.g. make “happier” How? Apply octant-grouped structural rules E.g. “Influence to be upbeat and positive” Apply octant 2 rules (tempo [fast], loudness [loud] …) to music structure Sy 1

30. Slide 30 (of 45) The Rule System – Detection Why? Good influencing needs emotional context Requires Model of Music Tension Advanced pattern matching Advanced knowledge of music theory and composition Very tricky … Not attempted before Sy 2

31. Slide 31 (of 45) The Architecture Sy 3

32. Slide 32 (of 45) The Architecture – Application Intent Emotive Information The [Arousal, Valence] vector [3] Sy 4

33. Slide 33 (of 45) The Architecture – Audience Sensing Audience response provides a wealth of feedback data to performer Attitudes and audience response [A, V] incorporated E.g. Cap the fearfulness of a room’s music Affective computing Keystroke/mouse movement: Arousal and Tension Gaze tracking/Skin conductivity: Arousal and Interest Same problems as measuring induced emotion though Sy 5

34. Slide 34 (of 45) The Architecture – Emotive Algorithm Equalising unit for [A, V] coming from game and audience Player Cap: Room Value: Game Event: Resulting [A, V]: Sy 6

35. Slide 35 (of 45) Testing Progress [2] Aims Influence the perceived emotions of music with primary music emotion structural rules Rules can apply to both Western classical and standard computer game music Testing Listener played original work, followed by altered work (e.g. apply octant 2 rules) How did emotion baseline change? 11 participants, played 6 altered versions Sy 7

36. Slide 36 (of 45) Testing Progress Overall Results Looks OK but why the A, V discrepancy? AccuracyQuadrantArousalValence User Response57%90%62% Guess25%50% Weighted Improvement 130%80%24% Sy 8

37. Slide 37 (of 45) Testing Progress Quadrant Breakdown QuadrantAccuracySelection SkewSelection Rate % 181%Over56% 226%Under56% 371%Over3% 450%Correct- Sy 9

38. Slide 38 (of 45) Testing Progress Not Angrier, why? Music selection (something deeper going on..?) Incomplete rule implementation for quadrant 2 Sy 10

39. Slide 39 (of 45) Methodology Recap Score – Manipulate the structure and mark up with emotional performance metadata How are we going? Structure: Implemented and progressing Performative: Identified, future implementation

40. Slide 40 (of 45) Methodology Recap Audience – Determine emotional state and attitudes using affective computing tech. How are we going? Identified and developed a theoretical implementation

41. Slide 41 (of 45) Methodology Recap Architecture – Bring together for an awareness of score and audience How are we going? Theoretical Implementation at present

42. Slide 42 (of 45) Future Work Implement more structural music rules Expanded testing regime Implement performative rules Begin testing of performative rules Detection component Incorporate Affective Computing elements Fu

43. Slide 43 (of 45) Summary There exists no automated method to detect and influence the emotions of music We’re getting there An audience’s response to computer music has previously been inaccessible to computers and thus lost Theoretical, still Future work Su 1

44. Slide 44 (of 45) Questions? Contact srl@itee.uq.edu.au http://itee.uq.edu.au/~srl Papers [1] "Playing with Affect: Music Performance with Awareness of Score and Audience", 2005, Australasian Computer Music Conference [2] "Dynamic Response: Real-Time Adaptation for Music Emotion", 2005, Australasian Conference on Interactive Entertainment [3] “Influencing the Perceived Emotions of Music with Intent”, 2005, Third International Conference on Generative Systems (in review) Su 2

45. Slide 45 (of 45) References [a] Brown, S., B. Merker, and N.L. Wallin, An Introduction to Evolutionary Musicology, in The Origins of Music, S. Brown, B. Merker, and N.L. Wallin, Editors. 2000, MIT Press. [b] Sloboda, J.A. and S.A. O'Neill, Emotions in Everyday Listening to Music, in Music and Emotion, theory and research. 2001, Oxford Press. p. 415-429. [c] Russell, J.A., Measures of emotion., in Emotion: Theory research and experience., R.P.H. Kellerman, Editor. 1989, New York: Academic Press. p. 81-111. [d] Schubert, E., Measurement and Time Series Analysis of Emotion in Music. 1999, University of New South Wales. [e] Meyer, L.B., Emotion and Meaning in Music. 1956: The University of Chicago Press. Su 3