1. Student: Mike Jiang Advisor: Dr. Ras, Zbigniew W. Music Information Retrieval

2.  Pitch - fundamental frequency ◦ Melody  Temporal- duration ◦ rhythmic  Timbral* ◦ tone color Facets of Music Information

3.  Aural Queries ◦ Query By Humming (QBH) systems  Input: aural melody  matches melody, rhythm  Indexing for Aural Queries ◦ melodies are extracted from the source ◦ Translated into text representations of intervals, pitch  Legal ◦ Is any passage from this piece sampled or copied from one of ours? possible Applications

4.  Music education ◦ Music performance analysis ◦ Searching music by instruments for Quintet practicing.  Music therapy ◦ Help doctors identify efficient musical pieces. string quartet piano sonata

5. Data sourceorganizationvolumeTypeQuality Traditional datastructuredmodestdiscrete, categorical clean Audio dataUnstructuredVery largeContinuous, Numeric noise The nature and types of raw data

6. IDAgeoccup ation SalaryCity 118Stude nt lowAtlant a 230Work er medi um Cleve land 343teach er medi um Rich mond 450profe ssor highBosto n 540bank er highNew York..

7.  Binary File  PCM : ◦ Sampling Rate 44K Hz 16 bits 2,646,000 int/min.

8. Feature Database traditional pattern recognition Feature Extraction lower level raw data form Object/Pattern detection Higher level representations classificationclusteringregression Pattern Database Energy values at each sample point manageable, (nearly) homogeneous subset of objects

9.  organizing large collections of music  create MusicMaps ◦ Automatic description of digital audio files by sound features ◦ visualize the similarity of songs and artists ◦ Similarity search in music collection MusicMiner

10.  Low level features extraction-400  high level features-60  feature selection  Clustering MusicMiner- numerical measure of perceptual music similarity

11.  A query by whistling/humming system for melody retrieval  A collection of approx. 2000 melodies and classical themes notify! Whistle

12.  Note extraction process ◦ Thresholding ◦ Signal splitting ◦ Fourier analysis ◦ Quantization to MIDI-Note level notify! Whistle

14.  Collection provided by user; music archives  Query by Example, Audio File  audio is indexed and feature vectors are store in vector file  interactive exploration  similarity-based search PlaySOM

15.  Matching Description ◦ Features(Rhythm Patterns) are passed to a self- organizing map ◦ retrieves similar music by creating paths on the map PlaySOM

16.  For each audio file, generate reproducible landmarks ◦ –Each landmark occurs at a time offset  For each landmark, generate a “fingerprint” tag that characterizes its location Shazam-Industry leader in audio fingerprinting

17.  Do same for sample  Generate list of matching fingerprints  timedb–timesample= Constant Shazam-Industry leader in audio fingerprinting

18. Shazam-no match

19. Shazam-match

20.  Input the melody  Match the note sequence and get the answer on composer, title, notes that matched C-Brahms Retrieval Engine for Melody Searching

21.  A Java applet records the audio signal.  Then its fundamental frequency is analyzed.  Adaptive preprocessing reduces the influence of background noise on the succeeding steps. A Java-based online QBH system

22.  Query by Example  probabilistic matching ◦ probabilistic models  Clustered dataset ◦ tree structure ◦ match the query following the paths GUIDO

23.  Query by Humming,Query by Example  Multimodal Adaptive Recognition System ◦ also takes into account speech and phonetic content  comparing hummed queries to other hummed queries http://www.midomi.com/ http://www.midomi.com/ Midomi

24.  43 MIR systems  Most are pitch estimation-based melody and rhythm match  Is there MIR system based on timbre match existed? summary

25.  Auto indexing system for musical instruments Auto indexing system for musical instruments  intelligence query answering system for music instruments intelligence query answering system for music instruments WWW.MIR.UNCC.EDU

26. Flow chart of MIR with sound separation. Polyphonic Sound Polyphonic Sound Get frame FFT Feature extraction Classifier Pitch Estimation Pitch Estimation Get Instrument Sound separation Power Spectrum New spectrum

27. Hierarchical Classification Strings Violin Music Brass Trumpet Cello Percussion Wood Winds Piano Flute Guitar English Horn Viola Bass Flute Oboe Bass Clarinet French Horn Harp

28. Feature Extraction Feature Extraction Features Classifier instrumentconfidence Candidate 170% Candidate 2 50%...... Candidate N10% 40ms

29. MIR with new strategy. Polyphonic Sound Polyphonic Sound Get frame FFT Feature extraction Higher level Classifier Get Family lower level Classifier Get Instrument Candidates Get Instrument Candidates Finish all the Frames estimation Voting process Get Final winners