1. 1 The Representation, Indexing and Retrieval of Music Data at NTHU Arbee L.P. Chen National Tsing Hua University Taiwan, R.O.C. http://www.cs.nthu.edu.tw/~alpchen

2. 2 Outline Content-based media data retrieval Music data retrieval Features of music data Feature indexing and matching Prototypes Reference

3. 3 Content-based Media Data Retrieval Representation of media contents features Feature extraction from media data Feature indexing Query interface

4. 4 Content-based Media Data Retrieval Matching query features against the feature index approximate/partial matching similarity measure precision: how many of the answers are in fact correct recall: how many of the correct answers are in fact retrieved relevance feedback

5. 5 Music Data Retrieval: System Architecture

6. 6 Features of Music Data

7. 7 Static music information The intrinsic music characteristics of music objects Key, beat, and tempo E.g., the Beethoven Symphony No. 5, Op. 67, C minor, 4/4, Allegro con brio Acoustical features Loudness, pitch, duration, bandwidth and brightness Can be computed and represented as numerical values

8. 8 Features of Music Data Thematic features Themes, melodies, rhythms, and chords Can be derived from the staff information of a music object Melody The melody of a song is the sequence of the pitches of all notes in the songs E.g., the melody of the theme of the Beethoven ’ s Symphony No.5 is “ sol – sol – sol – mi – fa – fa – fa - re ”

9. 9 Features of Music Data Rhythm The rhythm of a song is the sequence of the durations of all notes in the songs E.g., the rhythm of the theme of the Beethoven ’ s Symphony No.5 is “ 1/2-1/2-1/2-2-1/2-1/2-1/2-4 ” Chord A chord consists of three (root, third, and fifth) or more notes which sound together in harmony

10. 10 Features of Music Data Coding scheme: a music object  a sequence of music segments music segment = (segment type, segment duration, segment pitch) four segment types: ┌┐ (type A), └┘ (type B), ┌┘ (type C), and └┐ (type D)

11. 11 Features of Music Data For example, the sequence of music segments: (B,3,-3) (A,1,+1) (D,3,-3) (B,1,-2) (C,1,+2) (C,1,+2) (C,1,+1)

12. 12 music segment = (type, duration, pitch)

13. 13 Features of Music Data Repeating Pattern A sequence of notes appearing more than once in the music object Efficient content-based retrieval Semantics-rich representation Extracting repeating patterns Tree-based approach Matrix-based approach

14. 14 Features of Music Data Experiment 1

15. 15 Features of Music Data Dissimilarity of melody strings

16. 16 Features of Music Data Dissimilarity of repeating patterns

17. 17 Features of Music Data Experiment 2

18. 18 Features of Music Data Validity of classes

19. 19 Finding Repeating Patterns: Tree-based Approach Construct an RP-tree for RP ’ s with lengths 2 n, n  0, 1,... S = “ ABCDEFGHABCDEFGHIJABC ”

20. 20 Finding Repeating Patterns: Tree-based Approach Length 1 {A, 3, (1, 9, 19)} {B, 3, (2, 10, 20)} {C, 3, (3, 11, 21)} {D, 2, (4, 12)} {E, 2, (5, 13)} {F, 2, (6, 14)} {G, 2, (7, 15)} {H, 2, (8, 16)}

21. 21 Finding Repeating Patterns: Tree-based Approach Length 2 {AB, 3, (1, 9, 19)} = {A, 3, (1, 9, 19)}  0 {B, 3, (2, 10, 20)} {BC, 3, (2, 10, 20)} = {B, 3, (2, 10, 20)}  0 {C, 3, (3, 11, 21)} {CD, 2, (3, 11)} = {C, 3, (3, 11, 21)}  0 {D, 2, (4, 12)} …

22. 22 Finding Repeating Patterns: Tree-based Approach Length 4 {ABCD, 2, (1, 9)} = {AB, 3, (1, 9, 19)}  0 {CD, 2, (3, 11)} {BCDE, 2, (2, 10)} = {BC, 2, (2, 10, 20)}  0 {DE, 2, (4, 12)} … Length 8 {ABCDEFGH, 2, (1, 9)} = {ABCD, 2, (1, 9)}  0 {EFGH, 2, (5, 13)}

23. 23 Finding Repeating Patterns: Tree-based Approach

24. 24 Finding Repeating Patterns: Tree-based Approach Prune trivial patterns of length 2 n, n = 0, 1, … Let X be an RP of S, Y a substring of X, and Z a substring of Y If freq(X) = freq(Z), Y is trivial

25. 25 Finding Repeating Patterns: Tree-based Approach Length 1 {ABCDEFGH, 2, (1, 9)} {ABCD, 2, (1, 9)} {BCDE, 2, (2, 10)} {CDEF, 2, (3, 11)} {DEFG, 2, (4, 12)} {EFGH, 2, (5, 13)} {AB, 3, (1, 9, 19)} {BC, 3, (2, 10, 20)} {CD, 2, (3, 11)} {DE, 2, (4, 12)} {EF, 2, (5, 13)} {FG, 2, (6, 14)} {GH, 2, (7, 15)}

26. 26 Finding Repeating Patterns: Tree-based Approach Length 2 {ABCDEFGH, 2, (1, 9)} {ABCD, 2, (1, 9)} {BCDE, 2, (2, 10)} {CDEF, 2, (3, 11)} {DEFG, 2, (4, 12)} {EFGH, 2, (5, 13)} {AB, 3, (1, 9, 19)} {BC, 3, (2, 10, 20)}

27. 27 Finding Repeating Patterns: Tree-based Approach Length 4 {ABCDEFGH, 2, (1, 9)} {AB, 3, (1, 9, 19)} {BC, 3, (2, 10, 20)}

28. 28 Finding Repeating Patterns: Tree-based Approach Generate all patterns of lengths  2 n, n  0, 1,... {ABCDEFGH, 2, (1, 9)} {AB, 3, (1, 9, 19)} {BC, 3, (2, 10, 20)} {ABC, 3, (1, 9, 19)} order-1 string-join AB  1 BC = ABC

29. 29 Finding Repeating Patterns: Tree-based Approach Prune all trivial patterns {ABCDEFGH, 2, (1, 9)} {ABC, 3, (1, 9, 19)}

30. 30 Feature Indexing and Matching 1D-List PAT-Tree L-Tree Augmented Suffix Tree Grid-Twin Suffix Tree

31. 31 Feature Indexing and Matching: 1D-List There are two music objects M1 and M2 M1: ” sol-mi-mi-fa-re-re-do-re-mi-fa-sol-sol- sol ” M2: ” do-mi-sol-sol-re-mi-fa-fa-do-re-re-mi ” The melody string of the music query Q: ” do-re-mi ” Problem: to find whether M1 and M2 contain the melody string Q

32. 32

33. 33

34. 34 Feature Indexing and Matching: PAT-Tree Example, songs in chord strings Song1 : Am F2 Dm Am Song2 : C C F C Song3 : G E1 C D Song4 : E1 G Am Bm

35. 35 Feature Indexing and Matching: PAT-Tree

36. 36 Prototype 1

37. 37 Prototype 1

38. 38 Prototype 2

39. 39 Prototype 2

40. 40 Prototype 2

41. 41 References (http://db.nthu.edu.tw) Chen, A.L.P., M. Chang, J. Chen, J.L. Hsu, C.H. Hsu, and S.Y.S. Hua, “ Query by Music Segments:An Efficient Approach for Song Retrieval, ” in Proc. of IEEE Intl. Conference on Multimedia and Expo, 2000. Chen, J.C.C. and A.L.P. Chen, “ Query by Rhythm:An Approach for Song Retrieval in Music Database, ” in Proc. of IEEE Intl. Workshop on Research Issues in Data Engineering, 1998. Chou, T.C., A.L.P. Chen, and C.C. Liu, “ Music Databases: Indexing Techniques and Implementation, ” in Proc. of IEEE Intl. Workshop on Multimedia Data Base Management System, 1996. Hsu, J.L., C.C. Liu, and A.L.P. Chen, “ Efficient Repeating Pattern Finding in Music Databases, ” in Proc. of ACM Intl. Conference on Information and Knowledge Management, 1998.

42. 42 References (http://db.nthu.edu.tw) Lee, W and A.L.P. Chen, “ Efficient Multi-Feature Index Structure for Music Data Retrieval, ” in Proc. of SPIE Conference on Storage and Retrieval for Image and Video Databases, 2000. Liu, C.C., J.L. Hsu, and A.L.P. Chen, “ An Approximate String Matching Algorithm for Content-Based Music Data Retrieval, ” in Proc. of IEEE Intl.Conference on Multimedia Computing and Systems, 1999. Liu, C.C., J.L. Hsu, and A.L.P. Chen, “ Efficient Theme and Non- Trivial Repeating Pattern Discovering in Music Databases, ” in Proc. of IEEE Intl. Conference on Data Engineering, 1999.