Distributed Computing Group Visually and Acoustically Exploring the High-Dimensional Space of Music Lukas Bossard Michael Kuhn Roger Wattenhofer SocialCom 2009 Vancouver, Canada

2 Michael Kuhn, ETH SocialCom 2009 Storage media –Vinyl records –Compact cassettes –Compact discs An Album is stored on a single physical storage medium –Sequence of songs given by album –Album is typically listened to as a whole History organization by album

3 Michael Kuhn, ETH SocialCom 2009 Music today Huge offer, easily available –filesharing, iTunes, amazon, etc. Large collections –The entire collection is stored on a single electronic storage medium –Organization by albums (and other lists) is no longer appropriate organize by similarity

4 Michael Kuhn, ETH SocialCom 2009 Organization by Similarity Our Goals –Mobile application (portable player) –Play songs the user likes –Overview of a collection Problems on mobile devices –Limited input –Limited output –Limited processing power –Limited memory Solution –Use song coordinates provided by

5 Michael Kuhn, ETH SocialCom 2009 Which songs are similar? Goussevskaia et al., WI 2008: –Each song is positioned in a Euclidean „Map of Music“ –Similar songs are close to each other in this Euclidean space

6 Michael Kuhn, ETH SocialCom 2009 The Map of Music Based on usage data –„behaviour of the crowd“ –Gathered from social music platform (last.fm) –NO audio-analysis! Underlying similarity measure –Item-to-item collaborative filtering (Amazon) [Linden et al., IEEE Internet Computing] –„users who listen to song A also listen to song B“ Coordinates available through webservice –

7 Michael Kuhn, ETH SocialCom 2009 Hey Jude Imagine My Prerogative I want it that way Praise you Galvanize rock pop electronic Using the Map Similar songs are close to each other Quickly find nearest neighbors Span (and play) volumes Create smooth playlists by interpolation Visualize a collection Low memory footprint –Well suited for mobile domain convenient basis to build music software

8 Michael Kuhn, ETH SocialCom 2009 That‘s easy – is it? 10 dimensional!

9 Michael Kuhn, ETH SocialCom 2009 Contributions Visual and acoustic guide to the high-dimensional music galaxy Proof-of-concept application for Android devices („Google-phone“)

10 Michael Kuhn, ETH SocialCom 2009 Visual Exploration

11 Michael Kuhn, ETH SocialCom 2009 The Reference: SensMe (Sony Ericsson) slow fast happy sad Create playlist by selecting areas Based on audio-analysis

12 Michael Kuhn, ETH SocialCom 2009 Requirements Global Overview Local Overview Orientation Our problem: 10 dimensions!

13 Michael Kuhn, ETH SocialCom 2009 Lens Metaphor Few details in the border rings Detailed view in the center

14 Michael Kuhn, ETH SocialCom 2009 Lens: Recursive Clustering High resolution in the center Few details in the border regions

15 Michael Kuhn, ETH SocialCom 2009 Cake Metaphor Used to represent song clusters

16 Michael Kuhn, ETH SocialCom 2009 The Visual Exploration Interface Browsing –Touch cluster to bring it to the center Playlist Generation –Select a number of seed songs –Playlist will consist of songs around these seeds –Similar to SensME (but songs are selected in a different interface) Touch to make this area the new center

17 Michael Kuhn, ETH SocialCom 2009 Evaluation (1) User Experiment –9 participants –Collection (1400 songs) –5 minutes to create playlist of 20 songs (for both systems) Evaluation: Participants had to... –...rate each individual song in the playlists –...fill in a questionaire vs.

18 Michael Kuhn, ETH SocialCom 2009 Evaluation (2) Average song rating (scale: 0..10): –5.5 (SensMe) –6.3 (this paper) Questionaire (scale: 1..5): SensMeThis paper Playlist (overall) Diversity (3 is best) Usability Underlying space Use again?44%67% Trade-off: Accurracy of high-dimensional space versus simplicity of interface

19 Michael Kuhn, ETH SocialCom 2009 Acoustic Exploration

20 Michael Kuhn, ETH SocialCom 2009 Idea Shuffle (play songs in random order) Can we do better?Yes! Idea: Learn on the fly which songs the user likes! Skip = bad song Listen = good song

21 Michael Kuhn, ETH SocialCom 2009 Realization Basic algorithm: Voronoi Tesselation First song was skipped Supposed to be the user‘s region of interest

22 Michael Kuhn, ETH SocialCom 2009 Improvements Weighting –Account for strong/weak feedback Aging –Allows to adapt to changing mood Centering –Border regions are risky => go to center Escaping –Sometimes play random song to avoid getting stuck somewhere Rating bar (left = skip, right = good)

23 Michael Kuhn, ETH SocialCom 2009 References Random shuffling (e.g. iPod-Shuffle) Pampalk et al. (ISMIR, 2005) –Designed for (Euclidean) audio feature spaces dgdg dbdb If there are songs with d g < d b : select such song with smallest d g Else: select song with largest ratio d g /d b

24 Michael Kuhn, ETH SocialCom 2009 Evaluation 9 Participants Song ratings are used as input and for evaluation Diversity clearly better than Pampalk Ratings clearly better than random

25 Michael Kuhn, ETH SocialCom 2009 Conclusion Similarity based methods for music organization on mobile devices Methods outperform other approaches in the corresponding domain –User study with 9 participants –Schemes are not restricted to music –Trade-off: High dimensional space vs. usability Mobile devices can cope with advanced interfaces for similarity based music exploration –Lots of room for innovations in the next years! People profit from attention data gathered from the social web (e.g. coordinates at

26 Michael Kuhn, ETH SocialCom 2009 Conclusion

27 Michael Kuhn, ETH SocialCom 2009 Questions?