1. Demos for QBSH J.-S. Roger Jang ( 張智星 ) jang@cs.nthu.edu.tw http://mirlab.org/jang CSIE Dept, National Taiwan University

2. Intro. to QBSH zQBSH: Query by Singing/Humming zChallenges yRobust pitch tracking yKey transposition yCollection of song databases yEfficient comparison xKaraoke box: ~10000 songs xInternet: 500M songs, 12M albums (www.jogli.com)www.jogli.com

3. Efficient Retrieval in QBSH zMethods for efficient retrieval yMulti-stage progressive filtering yIndexing for different comparison methods yMusic phrase identification yRepeating pattern identification yDistributed & parallel computing zOur focus yParallel computing via GPU

4. MIRACLE zMIRACLE yMusic Information Retrieval Acoustically via Clustered and paralleL Engines zDatabase (~20K songs) yMIDI files ySolo vocals (<100) yMelody extracted from polyphonic music (<100) zComparison methods yLinear scaling yDynamic time warping zTop-10 Accuracy y~75% zPlatform ySingle CPU+GPU

5. MIRACLE (II) zReferences (full list)full list yJ.-S. Roger Jang and Ming-Yang Gao, "A Query-by-Singing System based on Dynamic Programming", International Workshop on Intelligent Systems Resolutions (the 8th Bellman Continuum), PP. 85-89, Hsinchu, Taiwan, Dec 2000. yJyh-Shing Roger Jang, Jiang-Chun Chen, Ming-Yang Kao, "MIRACLE: A Music Information Retrieval System with Clustered Computing Engines", International Symposium on Music Information Retrieval (ISMIR) 2001 y… yChung-Che Wang and Jyh-Shing Roger Jang, “Acceleration of Query by Singing/Humming Systems on GPU: Compare from Anywhere”, International Conference on Acoustics, Speech, and Signal Processing (ICASSP), 2012

6. MIRACLE Before Oct. 2011 zClient-server distributed computing zCloud computing via clustered PCs Master server Clients Clustered servers PC PDA/Smartphone Cellular Slave Master server Slave servers Request: pitch vector Response: search result Database size: ~12,000

7. Current MIRACLE zSingle server with GPU yNVIDIA 560 Ti, 384 cores (speedup factor = 10) Master server Clients Single server PC PDA/Smartphone Cellular Master server Request: pitch vector Response: search result Database size: ~13,000

8. MIRACLE in the Future zMulti-modal retrieval ySinging, humming, speech, audio, tapping… Master server Clients Clustered servers PC PDA/Smartphone Cellular Slave Master server Slave servers Request: feature vector Response: search result

9. QBSH for Various Platforms zPC yWeb version zEmbedded systems yKaraoke machines zSmartphones yiPhone/Android zToysToys y16-bit micro- controller

10. QBSH Prototype in MATLAB z To create a QBSH prototype in MATLAB yGet familiar with audio processing in MATLAB xSee audio signal processingaudio signal processing yTry the programming contests on xPitch trackingPitch tracking xQBSHQBSH Run exampleProgram/goDemo.m to test drive the QBSH prototype in MATLAB!

11. QBSH Demos zQBSH demos by our lab yQBSH on the web: MIRACLEMIRACLE yQBSH on toysQBSH on toys zExisting commercial QBSH systems ywww.midomi.comwww.midomi.com ywww.soundhound.comwww.soundhound.com

12. Returned Results zTypical results of MIRACLE

13. 13 Online Karaoke Synchronized lyrics Calory consumption Real-time score Recording Live broadcast Real-time pitch display Automatic key adjustment

14. Future Work zMulti-modal music retrieval yQuery by user’s inputs: Singing, humming, whistling, speech, tapping, beatboxing yQuery by exact examples: Audio clips zSpeedup schemes yRepeating pattern id., DTW indexing zDatabase preparation yPolyphonic audio music as database  The ultimate challenge!