1 EmuPlayer Music Recommendation System Based on User Emotion Using Vital-sensor KMSF- sunny 親： namachan さん

2 Motivation Music – Emotion: mutual relationship Users choose songs based on current feelings Playlist constantly expanding  difficulty in picking appropriate song

3 Requirement Track User Emotion Recommend by Sorting playlist based on user’s current emotion Sort songs by 2 factors  Relevancy to User Preference  Effect on User Emotion

4 Related research Matching Music Mood and User Emotion (-) Emotion: declared by user (-) Subjectiveness on Music Mood? (-) Music Mood and User Emotion are always the same? EmuPlayer  Emotion: automatically detected  User Preference under each emotion is studied  Recommendation implying emotional effect’s feedback

5 EmuPlayer: Example from User A Before  NoRPulse = 76.73, NoRTemp = 33.6  Pulse = 70.04, Temp = 34.0  relax/~sleepy Song Information  SongNo = 3  SongID = 4, Title = “So Close” After  “Like”  Pulse = 79.0, Temp =  pleasure  Score = 1

6 Approach Emotion Recognition Music Recommendation EmuPlayer Vital Information Recommender Emo Detector User Emotion Sorted Playlist

7 Emotion Recognition Merit of Vital-sensor Requirements  Portability if integrated in Music Player  Continuity of output data  Sensitiveness towards changes in emotion Vital-sensor meets all those requirement

8 Emotion Recognition Vital-sensor v.s Other Methods Requirement regard for the use in MRS Method Portability if integrated in Music Player Continuity of output data Sensitiveness towards changes in emotion while listening to music Facial ExpressionXX △ Speech △ X △ Eyes movement △△△ BrainwaveX○ ◎ GestureXX △ Vital-sensor○○○

9 Emotion Recognition Russell model and Two Biosignals Russell’s model  Horizontal axis: Pleasure  SkinTemp  Vertical axis: Arousal  Heart Rate 0° 315° 270° 225° 180° 135° 90° 45°

10 Emotion Recognition Mapping in EmuPlayer Define Emotion Region  Based on Theory of the Fuzziness of words  8 equal regions Mapping  Based on angle  (1,1)  45°  Excitement

11 Music Recommendation 2 factors to evaluate a song  Relevancy to User Preference  Mental Effect on User Emotion 2 subjects  Study User Preference  Study Emotional Effect of songs

12 Music Recommendation Study User Preference Rating Like/Dislike Record listening history

13 Music Recommendation Study songs’ emotional effect Define emotional effect: Good-Bad  to avoid potentially harmful recommendations to user emotion bad good

14 Music Recommendation Effect Definition Survey Matching point = 42/48*100 = 87.5%

15 Music Recommendation Rating songs  Better songs rank higher

16 System Flow Data Receiver Data Pre Processor Emo Detector Recommender Evaluator Interface Database User 10’ ’ 1 RF-ECG sensor 5 5

17 Demonstration

18 Evaluation on Emotion Recognition Experiment 1: Testing Accuracy of Emotion Recognition through arranged situation  Survey: (1) if they experienced the emotion expressed through the situation,  and (2) if not, what emotions rather than the one in (1) they experienced. Number of Participant GenderAverage Age 10~Male21

19 Result of Experiment 1 Output (Result from Engine) Input (Verified Experimenting Emo) ArousalExciteme nt PleasureRelaxatio n Sleepine ss Disples ure DistressDepressi on Relaxation 4.25%7.96%22.77 % % 9.44%000 Excitement 10%63.34 % % 5%0000 Pleasure 1.4%10.87 % % % 0000 Arousal % 5%06.66% 000 Depression 8.33% %20%10%55.01 % Accuracy = 64.5%

20 Evaluation on Emotion Recognition Experiment 2 Change User Emotion by music Purpose  Verify whether the system can realize user’s emotional changes  Verify songs’ influence on listeners’ emotions

21 CaseExperimentMeanResult 1Arousal  Pleasure/Rela xation Classical music Arousal:81.68%  1.41% Pleasure:0%  54.91% Relaxation:6.66%  38.94% 2Normal  Pleasure Music participants like Pleasure: 66.86%  93.33% 3Normal  Excitement Fast beat Music Excitement: 10.87%  62.12% 4Normal  Depression Loud Heavy Music played in long time Pleasure:66.86%  Depression: 80.02%

22 Emotion Recognition Conclusion from 2 experiments Accuracy of Extracting Emotion: 64.5% Strong at detecting bad emotions Detect precisely regarding to changes of user emotion Hypothesis of music influencing on user emotion is true

23 Evaluation: EmuPlayer Performance Observing high-rating songs  % being “dislike” after being listened  % paying “bad influence” on user’s emotion after being listened  % being reduced in score after being listened Observing “like” song  Emotional change

24 EmuPlayer Performance Observing high-rating songs

25 EmuPlayer Performance Observing “like” song  No song influencing badly on users’ emotion Emo BeforeEmo AfterPercentage ArousalRelaxation1% ArousalSleepiness1% ExcitementArousal1% ExcitementPleasure2%2% Excitement2% Pleasure 77% RelaxationArousal1%1% RelaxationPleasure2%2% Relaxation 7%7% Sleepiness 4%4% Displeasure 2%2%

26 Conclusion of EmuPlayer Performance EmuPlayer algorithm ensures recommendation of songs meeting proposed two requirements  Songs influencing badly on user emotion: 0%  Songs being “dislike” in later listening time: 6.66%

27 Overall survey Are you interested in such a MRS system?  Yes: 90% Scale your satisfaction of EmuPlayer’s work  Average point = 3.6/5 Do you feel uncomfortable wearing RF-ECG?  Yes: 40% Did you experience bad emotion after listening to high- rating song  Yes: 10%  Reflect the truth: proposing ER method responds to only clear and strong emotions. Slight changes in emotion felt by users may not be recognized by the system

28 Conclusion Concept of EmuPlayer is essential  Evaluate song through 2 factors  Employ User Emotion as crucial input for MRS Accuracy of extracting emotion is not very high: 64.5% Strong at detecting bad emotion  Applicable in giving alert when playing music influences badly on listener’s emotional state EmuPlayer’s efficiency in suggesting songs meeting the two requirements

29 Future works Enhance the accuracy of detecting emotion by  Employing other means than Heart Rate and Skin Temperature  Alternate RF-ECG Enhance the work of Recommending music by combining proposed method with songs’ content analyzing Enhance reasoning user’s state by combining User Emotion with context analyzing

30 Thank you for listening