1. Jaewoo Chung Scotty Vercoe MAS630/Affective Computing Spring '05 The Affective Listener

2. Jaewoo Chung Scotty Vercoe MAS630/Affective Computing Spring '05 The Affective Listener “How remarkable would it be if one could experience and express the spectrum of emotions embodied in music originating from oneself, without the crutch of a composer’s intercession…Can the touch that lies behind music be tapped?” –Manfred Clynes

3. Jaewoo Chung Scotty Vercoe MAS630/Affective Computing Spring '05 The Affective Listener Goal: Direct music using affective cues Issues: Devise mapping scheme of music parameters Correlate affective signals with music parameters Disambiguate data collected during music listening Develop algorithm to navigate music map by affect

4. Jaewoo Chung Scotty Vercoe MAS630/Affective Computing Spring '05 The Affective Listener Presentation Outline 1. Existing Work 2. Project Overview: Music Parameters Affective Signals 3. Listening Experiment 4. Data analysis 5. Algorithm 6. Conclusions 7. Demonstration

5. Jaewoo Chung Scotty Vercoe MAS630/Affective Computing Spring '05 The Affective Listener 1. Existing Work 2. Project Overview: Music Parameters Affective Signals 3. Listening Experiment 4. Data analysis 5. Algorithm 6. Conclusions 7. Demonstration

6. Jaewoo Chung Scotty Vercoe MAS630/Affective Computing Spring '05 The Affective Listener Existing Research Berlyne (1974): liking is greatest for stimuli of moderate arousal arousal and liking should be dependent; familiarity and complexity affect arousal level Russell et al., (1981): Circumplex model of emotion puts emotions in a circle around grid of pleasant-unpleasant and arousing-sleepy North and Hargreaves (1997): correlated circumplex model to valence/arousal dimensions replaced pleasant-unpleasant axis with like-dislike axis in circumplex coordinates of liking-arousal was found to be a reliable predictor of emotional reaction Schubert (1996): people often enjoy music that is unpleasant Ritossa and Rickard (2004): tested dimensions of pleasantness and liking in circumplex one of eight emotions reliably predicted using arousal, familiarity & pleasantness pleasantness was more useful predictor of emotions than liking

7. Jaewoo Chung Scotty Vercoe MAS630/Affective Computing Spring '05 The Affective Listener Existing Research Marrin and Picard (1998): Conductor’s Jacket records a variety of affective signals during conducting Healey, Picard and Dabek (1998): Affective DJ uses a mixture of physiological and subjective data compares SCR from first 30 seconds of current song to last 30 seconds of previous song algorithm selects songs to change from current affective state to desired state Kim and André (2004): Composing Affective Music uses ECG, EMG, SCR and RESP data composes algorithmically SCR is useful indicator for unsettling-relaxing EMG useful indicator for positive-negative

8. Jaewoo Chung Scotty Vercoe MAS630/Affective Computing Spring '05 The Affective Listener 1. Existing Work 2. Project Overview: Music Parameters Affective Signals 3. Listening Experiment 4. Data analysis 5. Algorithm 6. Conclusions 7. Demonstration

9. Jaewoo Chung Scotty Vercoe MAS630/Affective Computing Spring '05 The Affective Listener Music Affect Mapping Music Parameter Mapping user study music box Project Overview 1) Small-scale listening experiment: Change music parameters, and observe physiological and self-report data. 2) Music generator: Develop real-time algorithm that modifies music parameters based on affect.

10. Jaewoo Chung Scotty Vercoe MAS630/Affective Computing Spring '05 The Affective Listener Music Affect Mapping Music Parameter Mapping user study music box layering complexit y instrument layering: frequency-domain density complexity: time-domain density Music Parameter Mapping

11. Jaewoo Chung Scotty Vercoe MAS630/Affective Computing Spring '05 The Affective Listener Music Affect Mapping Music Parameter Mapping user study music box arousal valence arousal: reaction level to music valence: subject’s like/dislike of music Music Affect Mapping

12. Jaewoo Chung Scotty Vercoe MAS630/Affective Computing Spring '05 The Affective Listener high arousal like Music Affect Mapping low arous al dislik e engagingannoying soothingboring

13. Jaewoo Chung Scotty Vercoe MAS630/Affective Computing Spring '05 The Affective Listener high arousal like The Challenge low arous al dislik e engagingannoying soothingboring Current StateGoal State

14. Jaewoo Chung Scotty Vercoe MAS630/Affective Computing Spring '05 The Affective Listener The Challenge engagingannoying Current StateGoal State engagingboring engagingsoothing annoying soothingboring

15. Jaewoo Chung Scotty Vercoe MAS630/Affective Computing Spring '05 The Affective Listener Changing Music Affect engagingannoying engagingboring engagingsoothing annoying soothingboring Music Parameters Probability ? f-1, t-1 f-1, t+1 f+1, t-1 f+1, t+1 Start StateEnd State Given set changes in parameters, what are the trends for motion in affective space? How do those trends change depending on the initial affective state?

16. Jaewoo Chung Scotty Vercoe MAS630/Affective Computing Spring '05 The Affective Listener 1. Existing Work 2. Project Overview: Music Parameters Affective Signals 3. Listening Experiment 4. Data analysis 5. Algorithm 6. Conclusions 7. Demonstration

17. Jaewoo Chung Scotty Vercoe MAS630/Affective Computing Spring '05 The Affective Listener Listening Experiment Preparation of Music Clips: Five pieces with constant tempo were composed, ranging from jazz, rock to electronic music. For each piece, looping audio segments were produced to reflect the layer/complexity map. Segments were arranged into a 4x4 matrix for each piece. For each piece, four clips were assembled by traversing the matrix as follows: 1. Increasing complexity 2. Increasing instrument layering 3. Decrease complexity 4. Decrease instrument layering ((Listen))

18. Jaewoo Chung Scotty Vercoe MAS630/Affective Computing Spring '05 The Affective Listener Listening Experiment Preparation for Data Collection: Physiological Data Galvactivator set up to measure GSR microphone connected to measure presence of foot-tapping Self-report Data dual 7-point scales were set up for subject’s affective response self-report two sets were prepared for subjects to report their initial and final reactions

19. Jaewoo Chung Scotty Vercoe MAS630/Affective Computing Spring '05 The Affective Listener Listening Experiment Conducting the Experiment: 8 participants: 6 male, 2 female 20 audio clips (25-45 seconds each) were played for each subject experiment lasted approximately 25 minutes Physiological Data GSR was sampled every second throughout entire listening, BPM and velocity of foot-tapping was sampled every second, and presence of foot- tapping was sampled every 2 seconds Self-report Data subject self-reported affective response twice during each clip (initial & final)

20. Jaewoo Chung Scotty Vercoe MAS630/Affective Computing Spring '05 The Affective Listener 1. Existing Work 2. Project Overview: Music Parameters Affective Signals 3. Listening Experiment 4. Data analysis 5. Algorithm 6. Conclusions 7. Demonstration

21. Jaewoo Chung Scotty Vercoe MAS630/Affective Computing Spring '05 The Affective Listener Data Analysis Total Physiological Response - Subject 3 Total Physiological Response - Subject 4

22. Jaewoo Chung Scotty Vercoe MAS630/Affective Computing Spring '05 The Affective Listener Data Analysis Subject #2: physiological data MUSIC CLIP #9: Increase Complexity layering complexit y Subject #2: self-report data like annoying engaging dislik e boring soothing

23. Jaewoo Chung Scotty Vercoe MAS630/Affective Computing Spring '05 The Affective Listener Data Analysis Subject #6: physiological data MUSIC CLIP #9: Increase Complexity layering complexit y Subject #6: self-report data like annoying engaging dislik e boring soothing UNUSUAL EXAMPLE:

24. Jaewoo Chung Scotty Vercoe MAS630/Affective Computing Spring '05 The Affective Listener Data Analysis engaging soothing boring annoying presence of foot-tapping GSR

25. Jaewoo Chung Scotty Vercoe MAS630/Affective Computing Spring '05 The Affective Listener Data Analysis GSR fallingGSR rising S1: engaging S2: soothing S3: boring S4: annoying

26. Jaewoo Chung Scotty Vercoe MAS630/Affective Computing Spring '05 The Affective Listener Data Analysis GSR: updown GSR: updown S1: engaging S2: soothing S3: boring S4: annoying S1S1 S2S2 S1S1 S2S2 S3S3 S4S4 S3S3 S4S4

27. Jaewoo Chung Scotty Vercoe MAS630/Affective Computing Spring '05 The Affective Listener 1. Existing Work 2. Project Overview: Music Parameters Affective Signals 3. Listening Experiment 4. Data analysis 5. Algorithm 6. Conclusions 7. Demonstration

28. Jaewoo Chung Scotty Vercoe MAS630/Affective Computing Spring '05 The Affective Listener Algorithm Overview A pair of state transition models was constructed based on physiological and self-report data: The first model tries to detect the listener’s current affective state. The second model chooses the direction most likely to induce the goal state.

29. Jaewoo Chung Scotty Vercoe MAS630/Affective Computing Spring '05 The Affective Listener Probability Table for Detecting Affective State S1: engaging S2: soothing S3: boring S4: annoying

30. Jaewoo Chung Scotty Vercoe MAS630/Affective Computing Spring '05 The Affective Listener Affective State Transition Model ACTION: Increase Complexity layering complexit y S1: engaging S2: soothing S3: boring S4: annoying

31. Jaewoo Chung Scotty Vercoe MAS630/Affective Computing Spring '05 The Affective Listener ACTION: Increase Layering layering complexit y S1: engaging S2: soothing S3: boring S4: annoying Affective State Transition Model

32. Jaewoo Chung Scotty Vercoe MAS630/Affective Computing Spring '05 The Affective Listener ACTION: Decrease Complexity layering complexit y S1: engaging S2: soothing S3: boring S4: annoying Affective State Transition Model

33. Jaewoo Chung Scotty Vercoe MAS630/Affective Computing Spring '05 The Affective Listener ACTION: Decrease Layering layering complexit y S1: engaging S2: soothing S3: boring S4: annoying Affective State Transition Model

34. Jaewoo Chung Scotty Vercoe MAS630/Affective Computing Spring '05 The Affective Listener Summary of State Transitions S1-engagingS4-annoying S3-boring S1 S2 S3 S4 S2-soothing S1 S2 S3 S4

35. Jaewoo Chung Scotty Vercoe MAS630/Affective Computing Spring '05 The Affective Listener Summary of State Transitions S1: engaging S2: soothing S3: boring S4: annoying

36. Jaewoo Chung Scotty Vercoe MAS630/Affective Computing Spring '05 The Affective Listener System Diagram Physiological data input Control Affective state detection algorithm Direction choosing algorithm Change music parameter Goal stateInitial state Action Music Affect induced Perception

37. Jaewoo Chung Scotty Vercoe MAS630/Affective Computing Spring '05 The Affective Listener 1. Existing Work 2. Project Overview: Music Parameters Affective Signals 3. Listening Experiment 4. Data analysis 5. Algorithm 6. Conclusions 7. Demonstration

38. Jaewoo Chung Scotty Vercoe MAS630/Affective Computing Spring '05 The Affective Listener Conclusions changes in music parameters can be correlated to affective response to music Markov chains are a useful tool for constructing a predictive listening system specific observations about mapping layering/complexity to arousal/valence: engaged listeners tend to stay engaged annoyed listeners tend to stay annoyed soothed listeners tend to stay soothed, but also easily bored or engaged bored listeners tend to become interested by any change in parameters annoyed listeners are more likely to be engaged if first induced to boredom

39. Jaewoo Chung Scotty Vercoe MAS630/Affective Computing Spring '05 The Affective Listener 1. Existing Work 2. Project Overview: Music Parameters Affective Signals 3. Listening Experiment 4. Data analysis 5. Algorithm 6. Conclusions 7. Demonstration

40. Jaewoo Chung Scotty Vercoe MAS630/Affective Computing Spring '05 The Affective Listener Future Work improve accuracy of predictions by incorporating more user data improve affective state predictions using additional affective signals apply affect-parameter mapping to algorithmic composition use machine learning to customize predictions to individual subject

41. Jaewoo Chung Scotty Vercoe MAS630/Affective Computing Spring '05 The Affective Listener References Berlyne, D.E. (1974) Studies in New Experimental: Steps Towards an Objective Psychology of Aesthetic Appreciation. Halstead Press. Clynes, M. (1977) Sentics. Anchor Press. Healey, J., Picard, R. and Dabek, F. (1998) ‘A new affect-perceiving interface and its application to personalized music selection,’ Proc. of the Perceptual User Interfaces Workshop, 4-6. Kim, S. and André, E. (2004) ‘Composing affective music with a generate and sense approach,’ Proc. of Flairs 2004. American Association for Artificial Intelligence. Marrin, T. (2000) Inside the Conductor’s Jacket: Analysis, Interpretation and Musical Synthesis of Expressive Gesture. PhD thesis, MIT Media Lab, Cambridge, MA. Meyer, L. (1956) Emotion and Meaning in Music. University of Chicago Press. North, A.C. and Hargreaves, D.J. (1997) ‘Liking, arousal potential and the emotions expressed by music’, Pscyhomusicology 14:77-93. Ritossa, D. and Rickard, N. (2004) The relative utility of ‘pleasantness’ and ‘liking’ dimensions in predicting the emotions expressed by music. Psychology of Music. v.32(1):5-22 Russell, J.A. (1980) ‘A Circumplex Model of Affect’, Journal of Personality and Social Psychology 39: 1161-78. Rutherford, J. and Wiggins, G.A. (2002) ‘An experiment in the automatic creation of music which has specific emotional content,’ Proc. for the 7th International Conference on music Perception and Cognition, Sydney, Australia. Schubert, E. (1996) ‘Enjoyment of negative emotions in music: an associative network explanation’, Pscyhology of Music 24: 18-28. Sloboda, J.A. and Juslin P.N. (2001) Music and Emotion: Theory and Research. Oxford University Press.