1. August 12, 2004IAML - IASA 2004 Congress, Olso1 Music Information Retrieval, or how to search for (and maybe find) music and do away with incipits Michael Fingerhut Multimedia Library and Engineering Bureau IRCAM – Centre Pompidou

2. August 12, 2004 IAML - IASA 2004 Congress, Olso 2 Why Music Information Retrieval Increased availability of musical contents in digital form (remotely) Networks... actions, methods and procedures for recovering stored data to provide information on music. Need for… Faster methods for processing contents and producing "meaning" Computing power Increased availability of musical contents in digital form (locally) Storage ImplicationTechnical Advance

3. pattern extraction, form recognition… synchronization, summarization indexation pitch, voice extraction work publications paper symbolic OCR score about sound audioprograms text fingerprint structure taste, mood user musicology data multimediaaudio data rights owner DRM annotation performer composer performance (live, recorded) information music stored data metadata genre automatic playlists librarian lyrics abstract digital symbolic physical semantic author interaction cognitive, social theory feature extraction, identification motives concept → sign → signal → sign → concept textual features characteristics musical acoustical interpretation similarity formorchestration melodykeyharmonypolyphonytempoinstrumentation chordssegmentsdynamicsrhythm … durationpitchtimbre intensityspatialroom analysis A (much simplified) MIR map similarity metadata

4. August 12, 2004 IAML - IASA 2004 Congress, Olso 4 A typology of MIR Preprocessing OCR, digitization, compression Encoding, notation Feature extraction Segmentation Instrument recognition Voice recognition Indexation Identification Clustering Classification Extraction Melody, key, harmony, rhythm… Structural analysis Polyphony Repetition Similarity Summarization Organization Databases, systems, networks Compression Synchronization Metadata Search Objective criteria Metadata indices (name, title, period, genre, instrumentation…) Full-text (with or without semantic tags) Query by example (audio excerpt, melody, contour, rhythm, tonality, harmony…) Similarity Acoustical characteristics Subjective criteria Mood Taste Retrieve, deliver, use Browsing Playlists Using and reusing (annotate, combine, transform) Rights management (recognition, watermarking…) Usability Evaluation User studies

5. August 12, 2004 IAML - IASA 2004 Congress, Olso 5 Database of 100,000 documents Common methods in MIR Modeling Start from a theory Look for patterns Statistical Look for patterns Build a theory Evaluation Relevance (recall, precision) Performance (speed, friendliness) Standardized test collections 1000 relevant documents 400 retrieved documents 700 100 300 Recall: 30% (300 out of 1000) Precision: 75% (300 out of 400) queryquery

6. August 12, 2004 IAML - IASA 2004 Congress, Olso 6 MIR as a multidisciplinary domain 000 – General 000 Generalities & computer science 001 Knowledge 001.4 Research 003 Systems 004 Data processing 004.5 Storage 004.6 Interfacing and communications 005 Computer programming 005.4 Systems programming and programs 005.7 Data in computer systems 005.8 Data security 006 Special computer methods 006.3 Artificial intelligence 006.4 Computer pattern recognition 006.5 Computer sound synthesis 020 Library and information sciences 025 Library operations 025.3 Bibliographic analysis and control 025.4 Subject analysis and control 025.5 Services to users 025.6 Circulation 100 – Philosophy and Psychology 150 Psychology 152 Perception 152.1 Sensory perception 152.3 Movements and motor functions, facial expressions, laterality 152.4 Emotions and feelings 152.5 Physiological drives 152.8 Quantitative threshold, discrimination, reaction-time studies 153 Mental processes and intelligence 153.7 Perceptual processes 300 Social sciences 300 Sociology and anthropology 302.2 Communication 306 Culture and Institutions 310 General Statistics 340 Law 341.7 Law of international cooperation 380 Commerce 384 Communications; Telecommunications 500 – Natural Science and Mathematics 510 Mathematics 516 Geometry 600 – General Technology 620 Engineering & allied operations 620.2 Sound and related vibrarions 621.3 Electric, electronic, magnetic, communications, computer engineering 681 Precision instruments & other devices 700 – The Arts 780 Music 780.7 Education, performances 780.92 musicians, composers 781 General principles & musical forms 781.1 Aesthetics, appreciation, taste 781.2 Elements of music 781.3 Composition 781.4 Techniques of music 781.5 Kinds of music 781.8 Musical forms

7. August 12, 2004 IAML - IASA 2004 Congress, Olso 7 Take One, or Summarizing Dave Brubeck’s Take Five (5m25s) all1:47:35A small bit of each segment each0:14:82One segment of each type longest0:03:25Longest segment most freq.0:03:25Most frequent segment similarity matrix The warmer the color (red = warmest), the more similar 1. Find which parts sound like other parts (timbre similarity) 2. Extract segments 3. Produce summaries 654321654321 Courtesy of Geoffroy Peeters, Ircam 0m0s 5m25s

8. August 12, 2004 IAML - IASA 2004 Congress, Olso 8 A tool to catalog and extract audio CD contents for online distribution Automatic identification of CDs Compute CDDB of the CD Metadata retrieval and correction Query Internet CDDB for metadata Allow correction Extraction and compression Transfer to a Web server

9. August 12, 2004 IAML - IASA 2004 Congress, Olso 9

10. August 12, 2004 IAML - IASA 2004 Congress, Olso 10

11. August 12, 2004 IAML - IASA 2004 Congress, Olso 11

12. August 12, 2004 IAML - IASA 2004 Congress, Olso 12

13. August 12, 2004 IAML - IASA 2004 Congress, Olso 13

14. August 12, 2004 IAML - IASA 2004 Congress, Olso 14

15. August 12, 2004 IAML - IASA 2004 Congress, Olso 15

16. August 12, 2004 IAML - IASA 2004 Congress, Olso 16

17. August 12, 2004 IAML - IASA 2004 Congress, Olso 17

18. August 12, 2004 IAML - IASA 2004 Congress, Olso 18

19. August 12, 2004 IAML - IASA 2004 Congress, Olso 19 What’s the use… Increased collections Quantity Variety Help in organizing Cataloguing and indexing Help in finding Search and retrieval Contribute knowledge Metadata, indexation Relevance Users Getting involved ISMIR annual conferences www.ismir.net www.ismir.net Be informed Publish, speak Review Organize