1. 1 Content-based Music Information Retrieval Student: DENG Jie. Supervisor: Prof. LEUNG, Clement Department of Computer Science Hong Kong Baptist University March, 2010

2. 2 Outline What will be Covered:  Introduction  A Brief Review of Music  MIR in the Real World and Challenges  Current Content-based MIR Key Techniques  Evaluation of MIR  Conclusion and Future Work

3. 3 Introduction  Music Information Retrieval (MIR) is the interdisciplinary science of retrieving information from music. –Mainly based on three-filed subjects: traditional information retrieval, musicology and digital audio.  Content-based MIR is the science of extracting features from musical content, such as melody, rhythm and tempo and so on to facilitate tasks such as analysis and music retrieval.  Aim: –To better understand “music” in the music work –To really search music by “music”

4. 4 Music Channels Chart Shows Record store Mate’s recommendation Motivation - Music Discovery + Gigs Radio

5. 5 Music IR Application  Digital Music Libraries / Sound Archives –Seeking for content-based access music libraries –Combined with metadata search of existing catalogues  Music Education –Voice or instrumental teaching  Music Related Legal and Copyright –Is the creative content of this music work based on something for which others hold the rights?  Musicology –Is this piece of music work similar to any other works? –Dose any part of this piece closely resemble any part of any other works? –Is this piece of music work is based on others?

6. 6 A Brief Review of Music Music Concepts  Three basic features of a musical sound –Pitch –Intensity / Dynamics –Timbre / Tone color  There are many other terms describing music –Tempo –Tonality –Time Signature –Key Signature

7. 7 A Brief Review of Music Music Characteristics  Music can be defined as the art of disposing and producing sounds and silences in time –Has horizontal and vertical dimensions  The main dimensions of music can be used for music retrieval ( Reference Nicola Orio ) –Timbre : Quality of the produced sound –Orchestration : Sources of sound production –Acoustics : Quality of the recorded sound –Rhythm : Patterns of sound onsets –Melody : Sequence of notes –Harmony : Sequence of chords –Structure : Organization of the musical work

8. 8 A Brief Review of Music Music Representation –Visual (musical scores, manuscripts) –Aural (digital music) –Text –Hybrid (visual representation of an audio music file ) __________________________________________________________________________________ E---------------0---------3-3------3--1-1-------------3-1-1---------------- B---1-------------3-2-----2-2------2--3-3-----3-2---2---3-3------------1--- G---0--0--0h1-------0-----0-0------0--2-2---2---0-0-----2-2------------2--- D---2--2------------------------------0---0-------------0----0-1-2-3-3---3- A-3-------------------0-0-----0--0--------------0-------------------------- E-------------0------------------------------------------------------------ Common Music Notation Tablature Example: Visual Representation

9. 9 MIR in the Real World There are mainly three-category USERS in MIR “Professional”“Amateur” “Academic” Just about anyone! Librarians Publishers Producers Performers Composers Lawyers... Vast numbers Very many Musicologists Educators Significant numbers

10. 10 MIR in the Real World Common Music Data and Format  Audio recordings –Sampled sound –Wave, MP3, AAC, etc.  Symbolic recordings –Abstract musical instructions, MusicXML –Scores, MIDI, Humdrum, etc.

11. 11 MIR in the Real World Overview of some existing music search systems  Search by music related metadata: (artists, albums, tracks, music reviews, new release, etc.) Yahoo! Music and Allmusic are the examples of this search type  Search by music lyrics: Lyrics.com and SongLyrics.com  Music Media Management and Track Identification: Identify metadata for music tracks, for example Gracenote and MusicIP  Recommend similar music: by mining some music feature elements (melody, rhythm, tone color, etc) to recommend user some similar music  Recommend personalized music: by mining some users’ information to recommend them some their favorite music

12. 12 Challenges in MIR  Began in the 1950’s, still an emerging discipline  Subjectivity and Versioning  Many levels of music knowledge  Lack of bibliographic control and data quality ________________________________________________________________________ Signal ProcessingMachine Learning Human Computer Interaction Hearing Representation Understanding Analysis Reacting Interaction MIR Pipeline

13. 13 A Simplified MIR Map integration of audio visual, symbolic and textual data This very schematic diagram highlights trends Extracted or produced information Actions External data

14. 14 Basic Steps of Content-based MIR  Representation of music contents –Features: melody, rhythms, etc.  Feature extraction from music data  Feature indexing  Query interface  Matching query features against the feature index

15. 15  Representation of Music Content –Most music features used to represent music are always melody. –Rhythm feature only consider the rhythm omitting the melody. –Melody contour method uses three characters to express the contour of melody. Content-based MIR

16. 16 Content-based MIR  Feature extraction from music data –There are two category algorithms: time domain (Autocorrelation function, Average magnitude difference function and Simple inverse filter tracking) and frequency domain models (Spectrum and Cepstrum) –Common extracted tools in the following: Short-term Fourier Transform features (FFT) Mel-Frequency Cepstral Coefficients (MFCC) Daubechies Wavelet Coefficient Histogram (DWCH) –Pitch is the main feature extracted in practice

17. 17 Content-based MIR  Feature Indexing –Index terms: play a similar role of words in textual documents. –Sequence matching techniques: consider both the query and the documents as sequences of symbols and model the possible difference between them. –Geometric methods: cope with polyphonic scores and also exploit the properties of continuous distance measures. –Based on the above methods, there are mainly three category music search on the melody feature. Melodic retrieval based on index terms (N-grams) Melodic retrieval based on sequence matching Melodic retrieval based on geometric methods

18. 18 Content-based MIR  Query Interface –Query by text (keywords: album, artist, track, etc.) –Query by aural (singing or humming) Wave input (sing the whole or part of the songs) Music notes segmentation Thematic melodies are extracted, translated into text representations of intervals, pith, and harmony Comparison procedure –Query by tapping Wave input by tapping Compute the duration of each note Similarity comparison

19. 19 Content-based MIR  Matching Query Features against the Feature Index –Approximate/Partial matching –Similarity measure (MFCC, GMM, KNN) –Precision: how many of the answers are in fact correct –Recall: how many of the correct answers are in fact retrieved –Relevance feedback  Vector space model –Documents and queries are presented by vectors –Each element in a vector is determined by an indexing scheme (N- grams or others) –The value of each element is determined by a weight scheme –The similarity between document d i and query q j :

20. 20 Evaluation of MIR  The community has established an array of software tools to support this work –see http://music-ir.org/evaluationhttp://music-ir.org/evaluation  In 2004, Audio Description Contest first attempted to build comparative benchmark of MIR algorithms.  Downie has already given us the foundations and future of the scientific evaluation of MIR systems.  Traditional information retrieval evaluation can also be adopted in MIR, for example precision and recall measures.

21. 21 Conclusion  Music is a complicated art form of information and requires special retrieval systems  MIR technology is improving, but the real application is still lacking  Basic music concepts and characteristics  Basic steps and models of MIR  Current Content-based MIR Key Techniques  Scientific evaluation of MIR ___________________________________________________________ Mining the semantic information in multimedia, especially in digital audio music, and then propose a comprehensive and adaptive method to automatically analysis and retrieve the high level semantic information of music, for example, emotion, mood, and style, etc. Future Work

22. 22 Related Researches and Projects  ISMIR since 2000 –International Symposium on Music Information Retrieval  WOCMAT since 2005 –Workshop On Computer Music and Audio Technology  Digital archive application –Data mining in digital music archive  Free music audio, sound processing tools and music- related visualization and mining tools –http://www.music-ir.org/evaluation/tools.htmlhttp://www.music-ir.org/evaluation/tools.html  Music IR evaluation since 2005 –http://www.music-ir.org/mirexwiki/index.php/Main_Pagehttp://www.music-ir.org/mirexwiki/index.php/Main_Page –Test collection: music documents, query sets, and judgment –Major handle: copyright issue

23. 23 Reference Michael S. LEW, Nicu Sebe. Content-Based Music Information Retrieval: Current Directions and Feature Challenges. Proceedings of the IEEE, April 2008 Nicola Orio. Music Retrieval: A Tutorial and Review. Foundations and Trends in Information Retrieval, Volume1, Issue 1, Pages 1-96, 2006. J. T. Foote, "An Overview of Audio Information Retrieval." In ACM- Springer Multimedia Systems, vol. 7 no. 1, pp. 2-11, ACM Press/Springer-Verlag, January 1999 Remco C. Veltkamp, Frans Wiering, Rainer Typke. Content Based Music Retrieval. In B. Furht (Ed.), Encyclopedia of Multimedia. Springer, 2006. Giovanna Neve, Nicola Orio: A Comparison of Melodic Segmentation Techniques for Music Information Retrieval. ECDL 2005: 49-56. Hwei-Jen Lin, Hung-Hsuan Wu. Efficient geometric measure of music similarity. Information Processing Letters, Volume 109, Issue2, Page 116-120, 2008. Iman S. H. Suyoto, Alexandra L. Uitdenbogerd, and Falk Scholer. Searching Musical Audio Using Symbolic Queries.IEEE. Transactions onAudio, Speech and Language Processing, 16(2):372–381, 2008. The Scientific Evaluation of Music Information. Retrieval Systems: Foundations and Future. Computer Music Journal, Computer Music Journal, 28:2, pp. 12–23, Summer 2004. Michael Fingerhut. Real music libraries in the virtual future: for an integrated view of music and music information. Digitale bibliotheken voor muziek, 2005.

24. 24 Thank you! Questions?