1. LAM: Musical Audio Similarity Michael Casey Centre for Cognition, Computation and Culture Department of Computing Goldsmiths College, University of London

2. Overview Machine Music Understanding Features / Classes / Clusters Real-Time Audio Matching Feature Extraction Feature Similarity (Indexing / Retrieval) PD/MSP Tools Music Similarity Applications Sound object matching Texture matching

3. Sound Understanding Signal ProcessingSound Understanding

4. Feature Extraction

10. p( | ) * P( ) Statistical Learning for Decision Making Decision boundary Partitioning of feature space P( | )= p( ) Music Speech

11. MPEG-7 Audio Tools Audio

12. MPEG-7 Audio Tools Log Frequency Spectrogram Audio AudioSpectrumEnvelopeD

13. MPEG-7 Audio Tools Log Frequency Spectrogram Audio Log Amplitude Decorrelating Transform / Dimension Reduction AudioSpectrumEnvelopeD AudioSpectrumProjectionD

14. SoundModelStatePathD State Path Use estimated state sequence as a feature

15. MPEG-7 Audio Tools Log Frequency Spectrogram Audio Log Amplitude Decorrelating Transform / Dimension Reduction AudioSpectrumEnvelopeD AudioSpectrumProjectionD Hidden Markov Model SoundModelDS

16. MPEG-7 Audio Strings Acoustic Lexicons Log Frequency Spectrogram Audio Log Amplitude Decorrelating Transform / Dimension Reduction AudioSpectrumEnvelopeD AudioSpectrumProjectionD Hidden Markov Model SoundModelDS State Path ? 7 1 V 7 1 0 1... SoundModelStatePathD SYMBOL STRING

18. State Symbol Sequence (40 State Model) ?71V7101...

19. State Symbol Sequence (40 State Model) ?71V7101...

20. State Symbol Sequence (40 State Model) ?71V7101...

21. State Symbol Sequence (40 State Model) ?71V7101...

22. SoundModelStateHistogramD seconds state index 0.01s Frames

23. Self-Similarity Matrix

26. a

27. a b

28. a b

30. S-Matrix

31. Efficient Storage / Retrieval Real-Time Access Large Databases Distributed Databases

32. PostgreSQL Database Representation of State Path “Strings” and Histograms

33. Similarity Compute distance between feature pairs Features == SoundModelStateHistogramD Similarity Metric dist(a,b) >= 0 dist(a,b)== 0 iff a==b dist(a,b) + dist(b,c) >= dist(a,c) Vector Dot Product

34. Similarity of Feature Trajectories

35. Dynamic Time Warping

36. Acousticon Strings Distance Metric –String Edit Distance (Levenschtein) Scalable to Large Databases –PostgreSQL Implementation –Can use built-in Index Structures Scalable to Real-Time Implementation –matching and audio streaming (< 20ms )

37. Information Retrieval for Creativity Utilize sound extant database for new material Take the structure of a music clip but replace the content. New interfaces for music creativity.

38. Audio Information Retrieval MPEG-7 Database A pre-indexed Collection of Sounds

39. Audio Query Extract MPEG-7 Database SegmentMatch Result List A Sound or Scene or List of Sounds Audio Information Retrieval

40. Audio Query Extract MPEG-7 Database SegmentMatch Result List Feature extraction from audio. Audio Information Retrieval

41. Audio Query Extract MPEG-7 Database SegmentMatch Result List Partitioning of audio into chunks. Audio Information Retrieval

42. Audio Query Extract MPEG-7 Database SegmentMatch Result List Find similar chunks of Audio Audio Information Retrieval

43. Real-Time Matching

44. Musaics Real-Time Matching

45. Musaics Real-Time Matching

46. Musaics Real-Time Matching

47. Musaics Real-Time Matching

48. Musaics Real-Time Matching

49. Musaics Real-Time Matching

50. Musaics Real-Time Matching

51. Musaics Real-Time Matching

52. Musaics Real-Time Matching

53. Musaics Real-Time Matching