1. A Game-Based Approach for Collecting Semantic Music Annotations Douglas Turnbull, Rouran Liu, Luke Barrington, Gert Lanckriet Computer Audition Lab UC San Diego ISMIR September 27, 2007

2. 2 Introduction Automatic audio content analysis helps to organize, search, recommend, retrieve and describe huge - and growing - music collections Computer audition systems require significant amounts of high-quality semantic labels for audio content Collecting this data can be difficult, expensive, slow, boring and inaccurate If only we could get someone else to do it...

3. 3 Sources of Semantic Information 10 2 Quality Quantity 10 1 10 3 10 4 10 5 10 6 web mine id3 tags

4. 4

5. 5 Sources of Semantic Information 10 2 Quality Quantity 10 1 10 3 10 4 10 5 10 6 CAL500 human tags web mine id3 tags

6. 6

7. 7 Sources of Semantic Information 10 2 Quality Quantity 10 1 10 3 10 4 10 5 10 6 CAL500 Pandora Last.fm Web mine id3 tags Human Computation

8. 8 Human Computation Many problems that are hard for computers can be easily solved by humans Many humans spend lots of time solving problems that are of little use How can we put these “gray cycles” to use?

9. 9 Music Games Multi-player Music is social Music can be subjective Use group consensus... but allow personal variations Collaborative There are no “right answers” But agreed-on answers earn more points... Fun Need to excite players in order to collect lots of data Sacrifice data collection in favor of a compelling game

10. 10 www.ListenGame.org 590 players have played at least 1 game 30,000 song-word associations collected ISMIR deadline

11. 11 Evaluation Evaluate the quality of collected semantic annotations by using them to train an automatic music retrieval system [SIGIR07] 0.705CAL-250 159 Words 0.609AllMusic 317 Words Retrieval ROC AreaDataset 0.609AllMusic 317 Words Retrieval ROC AreaDataset 0.661Listen-250 82 Words 0.705CAL-250 159 Words 0.609AllMusic 317 Words Retrieval ROC AreaDataset

14. A Game-Based Approach for Collecting Semantic Music Annotations Douglas Turnbull, Rouran Liu, Luke Barrington, Gert Lanckriet Computer Audition Lab UC San Diego ISMIR September 27, 2007

15. 15 Annotating Music Web mining Cheap, collect lots of data Noisy data, not necessarily related to music content Surveys / Hand-labelling e.g. Music Genome Project, LastFM tags, CAL500 Reliable, can be tailored to applications Expensive, slow, boring, unfocused, free vocabulary Games Engage users, free, offer new, social music interaction Need lots of players!