Looking for New, Not Known Music Only : Music Retrieval by Melody Style Fang-Fei Kuo Dept. of Computer Science and Information Engineering National Chiao Tung UniversityHsinChu, Taiwan, ROC Man-Kwan ShanDept. of Computer Science National Cheng Chi UniversityTaipei, Taiwan, ROC Proceedings of the 4th ACM/IEEE-CS joint conference on Digital libraries

Outline Introduction Introduction Methodology Methodology Experiments Experiments Conclusions Conclusions

Introduction New music object or “feel like” another music style New music object or “feel like” another music style Humming, singing, or tapping is helpless Humming, singing, or tapping is helpless Text-based metadata Text-based metadata Content-based music retrieval by melody style Content-based music retrieval by melody style

Introduction(con) Issues Issues (1) Specification query style (1) Specification query style (2) Determine music feature and representation (2) Determine music feature and representation (3) Description of melody style (3) Description of melody style (4) Relevance between the music object and (4) Relevance between the music object and the query style the query style

Methodology Query Specification Query Specification (1) Query-by-music-group (QBMG) (1) Query-by-music-group (QBMG) (2) Query-by-music-example (QBME) (2) Query-by-music-example (QBME) (3) Query-by-taxonomic-style (QBTS) (3) Query-by-taxonomic-style (QBTS) (4) Query-by-taxonomic-style-combinations (4) Query-by-taxonomic-style-combinations (QBTSC) (QBTSC)

Methodology(con) Query Specification (con) Query Specification (con)

Methodology(con) Feature Extraction Feature Extraction (1) melody extraction from MIDI (1) melody extraction from MIDI (2) chords extraction from melody (2) chords extraction from melody (3) sixty common chords are chosen (3) sixty common chords are chosen (4) decides length of the sampling unit (4) decides length of the sampling unit (5) chord or chord-set (5) chord or chord-set

Methodology(con) Feature Representation Feature Representation (1) set of chord-sets (1) set of chord-sets EX:{(1.2),(1),(3)} EX:{(1.2),(1),(3)} (2) set of bigrams (2) set of bigrams EX:{((1.2),1),(1,3)} EX:{((1.2),1),(1,3)} (3) sequence of chord-sets (3) sequence of chord-sets EX: ({1.2} 1 3) EX: ({1.2} 1 3)

Methodology(con) Characterization for chord-set or bi-grams Characterization for chord-set or bi-grams (1) Frequent pattern mining technique (1) Frequent pattern mining technique (2) Item-set support > minimum support (2) Item-set support > minimum support (3) Frequent item-set (3) Frequent item-set EX:{{I}, {V, Ⅵ m7}, {V}} EX:{{I}, {V, Ⅵ m7}, {V}}

Methodology(con) Characterization for sequence chord-sets Characterization for sequence chord-sets (1) Sequence data mining technique (1) Sequence data mining technique (2) support > minimum support (2) support > minimum support (3) Frequent consecutive sequence pattern (3) Frequent consecutive sequence pattern EX:({I}, {V, Ⅵ m7}, {V}, {I, III, Vim7}) EX:({I}, {V, Ⅵ m7}, {V}, {I, III, Vim7}) ({V, Ⅵ m7}, {V}, {I, III, Vim7}) ({V, Ⅵ m7}, {V}, {I, III, Vim7}) ({V, Ⅵ m7}, {I, III, Vim7}) ({V, Ⅵ m7}, {I, III, Vim7})

Methodology(con) Discrimination Discrimination melody style rule r = l --> y melody style rule r = l --> y l : characterization of y y : music group l : characterization of y y : music group melody style pattern set = melody style pattern set = moldy style classification algorithm moldy style classification algorithm

Methodology(con) Ranking Function Ranking Function (1) For QBMG QBTS QBTSC (1) For QBMG QBTS QBTSC (2) For QBME (2) For QBME

Methodology(con) Ranking function for QBMG QBTS QBTSC Ranking function for QBMG QBTS QBTSC If first matched rule does not belong to the selected group, the music is not a qualified answer If first matched rule does not belong to the selected group, the music is not a qualified answer

Methodology(con) Music object : { Ⅱ 7 ⅤⅢⅡⅤⅠⅦ } pattern style rule : Music object : { Ⅱ 7 ⅤⅢⅡⅤⅠⅦ } pattern style rule : Set : { Ⅰ, Ⅲ, Ⅳ 7}, conf = 0.9 Set : { Ⅰ, Ⅲ, Ⅳ 7}, conf = 0.9 Bigram : {( ⅤⅠ ), ( Ⅴ 7 Ⅶ )}, conf = 0.75 Bigram : {( ⅤⅠ ), ( Ⅴ 7 Ⅶ )}, conf = 0.75 Sequence : ( Ⅴ Ⅲ Ⅱ Ⅴ Ⅰ ), conf = 0.6 Sequence : ( Ⅴ Ⅲ Ⅱ Ⅴ Ⅰ ), conf = 0.6 Bigram : {( Ⅰ Ⅱ ), ( Ⅳ 7 Ⅴ ), ( Ⅱ Ⅵ )}, Bigram : {( Ⅰ Ⅱ ), ( Ⅳ 7 Ⅴ ), ( Ⅱ Ⅵ )}, conf = 0.57 conf = 0.57 Default_class, conf = 0.52 Default_class, conf = 0.52

Methodology(con) Ranking function for QBME Ranking function for QBME Similarity between two chord-sets Similarity between two chord-sets

Methodology(con) Ranking function for QBME(con) Ranking function for QBME(con) similarity constraint :δ similarity constraint :δ U={u1,u2,u3…..uM} V={v1,v2,v3….vN} U={u1,u2,u3…..uM} V={v1,v2,v3….vN} Mapping relation : Mapping relation :

Methodology(con) Ranking function for QBME(con) Ranking function for QBME(con) Similarity between two set of chord-sets Similarity between two set of chord-sets U={u1,u2,u3…..uM} V={v1,v2,v3….vN} U={u1,u2,u3…..uM} V={v1,v2,v3….vN}

Methodology(con) Example: Example: U={u1,u2,u3,u4} V={v1,v2,v3} U={u1,u2,u3,u4} V={v1,v2,v3} similarity constraint δ=0.4 similarity constraint δ=0.4 chord-set whose similarity ≧ δ consist of chord-set whose similarity ≧ δ consist of (u1,v1),(u1,v2),(u2,v2),(u2,v3),(u3,v1)(v4,u1) (u1,v1),(u1,v2),(u2,v2),(u2,v3),(u3,v1)(v4,u1) are ½, 1/√6, 1/ √3, 1/ √3, 1, 1/ √6 are ½, 1/√6, 1/ √3, 1/ √3, 1, 1/ √6 similarity= (½ + 1/ √ 6+1/ √ 3+1/ √ 3+1+1/ √ 6) / √ 12= similarity= (½ + 1/ √ 6+1/ √ 3+1/ √ 3+1+1/ √ 6) / √ 12= 0.986

Experiments

Experiments(con)

Experiments(con) Precision=N_retrieved_relevant / N_retrievedAverage_score=∑score/N_retrieved

Experiments(con)

Conclusions Proposed an approach for melody style retrieval Proposed an approach for melody style retrieval Proposed four types of query specification Proposed four types of query specification Proposed query processing Proposed query processing Future work : provide other query methods Future work : provide other query methods query by selecting multiple styles query by selecting multiple styles query by style example music query by style example music