1. Introduction to Music Information Retrieval (MIR)
J.-S. Roger Jang (張智星)
MIR Lab, CSIE Dept., National Taiwan Univ.
2018/9/11

2. How to Search for a Song? Content-based search Melody Mood Genre
Instrument
Chords
Cover song
島谷ひとみ / 亜麻色の髪の乙女

3. Types of Search of MIR Systems
Metadata-based  Easier
Song title, artist, tags, composer, …
Query input: text or speech
Content-based  Harder
Melody, lyrics, mood, genre, chord, instruments, …
Query input:
Symbolic: notes, chord, text, …
Acoustic: Singing, humming, whistling, tapping, speech, recording of exact example, beatboxing…

4. Types of Acoustic Inputs for MIR
Singing/humming
Query by humming (usually “ta” or “da”)
Query by singing
Whistling
Query by whistling
Wolf whistle
Hand whistle
Fingerless whistle
Leaf whistle
Tapping
Query by tapping (at the onsets of notes)
Speech
Query by speech (for lyrics or meta-data)
Exact but noisy example
Query by example (noisy version of original clips)
Beatboxing

5. Types of Contents for Comparison
Melody
Query by humming
Query by singing
Query by whistling
Note onsets
Query by tapping (at the onsets of notes)
Meta-data
Query by speech
Audio contents
Query by examples (noisy versions of original clips)
Drum patterns
Query by beatboxing

6. Questions or comments for MIR?

7. Introduction to QBSH QBSH: Query by Singing/Humming Progression
Input: Singing or humming from microphone
Output: A ranking list retrieved from the song database
Progression
First paper: Around 1994
Extensive studies since 2001
State of the art: QBSH tasks at ISMIR/MIREX, since 2006

8. Challenges in QBSH Systems
Reliable pitch tracking for acoustic input
Input from mobile devices or noisy karaoke bar
Song database preparation
MIDIs, singing clips, or audio music
Efficient/effective retrieval
Karaoke machine: ~10,000 songs
Internet music search engine: ~500,000,000 songs

10. Two Types of Processing for QBSH
Offline processing
Collect pitch vectors for DB
Pure vocals
MIDI files
Pitch obtained from polyphonic music
Label the anchor position
Identify repeated patterns
Online processing
Compute pitch from the user’s query
Convert pitch into note sequence (optional)
Compare with DB
List the ranked result

11. Flowchart of QBSH On-line processing Off-line processing
Microphone input
Filtering
Pitch tracking
Pitch vector
smoothing
Frame-based
representation
Similarity comparison
Query results
(Ranked song list)
Melody track
extraction
MIDI files
Off-line processing

12. Short Latency and Strategies
Goal: To retrieve songs effectively within a given response time, say 5 seconds or so
Our strategies
Multi-stage progressive filtering
Indexing for different comparison methods
Repeating pattern identification
Platform upgrade: GPU is the way to go!