1. Wedneday, January 21st, 2008 Comp. Sci. Colloquium The Problem with Music: Modeling Distance Distributions of Large Music Collections Prof. Michael Casey Program in Digital Musics Dartmouth College, Hanover, NH

2. a.k.a. The Problem with Multimedia: Music Music Videos Videos Images

3. Scalable Similarity 8M tracks in commercial collection 8M tracks in commercial collection 6B Images on WWW 6B Images on WWW Require scalable nearest-neighbor methods Require scalable nearest-neighbor methods Increase scale, decrease search complexity Increase scale, decrease search complexity

4. Example: Hattogate

5. Example: Remixing / Sampling in Yahoo! Music Original Track Original Track Remix 1 Remix 1 Remix 2 Remix 2 Remix 3 Remix 3

6. Example: 3B Images in Flickr

7. Specificity Partial document (sub-track) retrieval Partial document (sub-track) retrieval Alternate versions: remix, cover, live, album Alternate versions: remix, cover, live, album Task is mid-high specificity Task is mid-high specificity

8. Machine Listening

9. Feature Extraction

15. Audio Shingles, concatenate l frames of m dimensional features A shingle is defined as: Shingles provide contextual information about features Originally used for Internet search engines: Andrei Z. Broder, Steven C. Glassman, Mark S. Manasse, Geoffrey Zweig:Steven C. GlassmanMark S. ManasseGeoffrey Zweig “Syntactic Clustering of the Web”. Computer Networks 29(8-13): 1157-1166 (1997)Computer Networks 29 Related to N-grams, overlapping sequences of features Applied to audio domain by Casey and Slaney : Casey, M. Slaney, M. “The Importance of Sequences in Musical Similarity”, in Proc. IEEE Int. Conf. onIEEE Int. Conf. on Acoustics, Speech and Signal Processing, 2006. ICASSP 2006Acoustics, Speech and Signal Processing, 2006. ICASSP 2006

16. Audio Shingle Similarity

17. , a query shingle drawn from a query track {Q}, database of audio tracks indexed by (n), a database shingle from track n Shingles are normalized to unit vectors, therefore: For shingles with M dimensions (M=l.m); m=12, 20; l=30,40

18. AudioDB: Shingle Nearest Neighbor Search

19. Whole-track similarity Often want to know which tracks are similar Often want to know which tracks are similar Similarity depends on specificity of task Similarity depends on specificity of task Distortion / filtering / re-encoding (high) Distortion / filtering / re-encoding (high) Remix with new audio material (mid) Remix with new audio material (mid) Cover song: same song, different artist (mid) Cover song: same song, different artist (mid)

20. Whole-track resemblance: radius-bounded search Compute the number of shingle collisions between two tracks:

21. Whole-track resemblance: radius-bounded search Compute the number of shingle collisions between two tracks: Requires a threshold for considering shingles to be related Need a way to estimate relatedness (threshold) for data set

22. SCALE Mazurkas: 10,000 tracks 10-100ms features Mazurkas: 10,000 tracks 10-100ms features 3s clips (30 – 300 frames per vector) 3s clips (30 – 300 frames per vector) 12d – 20d features (360 – 600d vectors) 12d – 20d features (360 – 600d vectors) Yahoo! Music Yahoo! Music 6M tracks 6M tracks 1000 vectors per track 1000 vectors per track (6M x 1k)^2 search for near neighbours (6M x 1k)^2 search for near neighbours

23. LSH

24. Approximate Near Neighbor Matching

25. Approximate near neighbors In many applications we need only near neghbors In many applications we need only near neghbors We can exploit this by allowing a degree of approximation in retrieval We can exploit this by allowing a degree of approximation in retrieval

26. Space partitioning

27. Curse of dimensionality d=4d=8 d=1024 dist. Pr(dist)‏

28. Border effects in high d

29. ε-NN : approximate near neighbors

30. Setting the range

31. Hashing Types of hashes Types of hashes String : put Bash vs Bush in different bins String : put Bash vs Bush in different bins Locality sensitive : close matches in same bin Locality sensitive : close matches in same bin High-dimensional and probabilistic High-dimensional and probabilistic Nearest Neighbor implementations Nearest Neighbor implementations Pair-wise distance computation Pair-wise distance computation 1,000,000,000,000 comparisons in 2M song database 1,000,000,000,000 comparisons in 2M song database Hash bucket collisions Hash bucket collisions 1,000,000,000 hash projections 1,000,000,000 hash projections

32. Exact matching via hashing Audio fingerprinting Audio fingerprinting Shazzam, etc. Shazzam, etc. Make the feature robust Make the feature robust Use exact matching on integer hash Use exact matching on integer hash Find a sequence of hashes to identify specific recording or image Find a sequence of hashes to identify specific recording or image Drawback: only exact matches possible Drawback: only exact matches possible

33. Locality-Sensitive Hashing (Indyk-Motwani’98)‏ Hash functions are locality-sensitive, if, for a random hash random function h, for any pair of points p,q we have: –Pr[h(p)=h(q)] is “high” if p is “close” to q –Pr[h(p)=h(q)] is “low” if p is”far” from q

34. Locality Sensitive Hashing

35. Random Projections Random projections estimate distance Random projections estimate distance Multiple projections improve estimate Multiple projections improve estimate

36. h’s are locality-sensitive Pr[h(p)=h(q)]=(1-D(p,q)/d) k Pr[h(p)=h(q)]=(1-D(p,q)/d) k We can vary the probability by changing k We can vary the probability by changing k k=1k=2 distance Pr

38. LSH Random Projections 3d to 2d

39. Statistical approaches to modeling distance distributions

40. Distribution of minimum distances Database: 1.4 million shingles. The left bump is the minimum between 1000 randomly selected query shingles and this database. The right bump is a small sampling (1/98 000 000) of the full histogram of all distances.

41. Radius-bounded retrieval performance: cover song (opus task) Performance depends critically on xthresh, the collision threshold Want to estimate xthresh automatically from unlabelled data

42. Order Statistics Minimum-value distribution is analytic Minimum-value distribution is analytic Estimate the distribution parameters Estimate the distribution parameters Substitute into minimum value distribution Substitute into minimum value distribution Define a threshold in terms of FP rate Define a threshold in terms of FP rate This gives an estimate of xthresh This gives an estimate of xthresh

43. Estimating xthresh from unlabelled data Use theoretical statistics Use theoretical statistics Null Hypothesis: Null Hypothesis: H 0 : shingles are drawn from unrelated tracks H 0 : shingles are drawn from unrelated tracks Assume elements i.i.d., normally distributed Assume elements i.i.d., normally distributed M dimensional shingles, d effective degrees of freedom: M dimensional shingles, d effective degrees of freedom: Squared distance distribution for H 0 Squared distance distribution for H 0

44. ML for background distribution Likelihood for N data points (distances squared) d = effective degrees of freedom M = shingle dimensionality

45. Background distribution parameters Likelihood for N data points (distances squared) d = effective degrees of freedom M = shingle dimensionality

46. Minimum value over N samples

47. Minimum value distribution of unrelated shingles

48. Estimate of xthresh, false positive rate

49. Unlabelled data experiment Unlabelled data set Unlabelled data set Known to contain: Known to contain: cover songs (same work, different performer) cover songs (same work, different performer) Near duplicate recordings (misattribution, encoding) Near duplicate recordings (misattribution, encoding) Estimate background distance distribution Estimate background distance distribution Estimate minimum value distribution Estimate minimum value distribution Set xthresh so FP rate is <= 1% Set xthresh so FP rate is <= 1% Whole-track retrieval based on shingle collisions Whole-track retrieval based on shingle collisions

50. Misattributions Joyce Hatto: 100% of known misattributions in first rank Joyce Hatto: 100% of known misattributions in first rank Sergie Fiorentino Sergie Fiorentino Eleven out of twenty-six Mazurkas performances on another Concert Artists/Fidelio disc, issued under the name of Sergio Fiorentino, are in fact copies of recordings by other artists. This is the first time that such practices have been found in the Concert Artist‘ Fidelio recordings issued other than under the name of Joyce Hatto, and prompts speculation as to how much more misattributed material remains to be found in the Concert Artists/Fidelio catalogue. Click here for further details. Eleven out of twenty-six Mazurkas performances on another Concert Artists/Fidelio disc, issued under the name of Sergio Fiorentino, are in fact copies of recordings by other artists. This is the first time that such practices have been found in the Concert Artist‘ Fidelio recordings issued other than under the name of Joyce Hatto, and prompts speculation as to how much more misattributed material remains to be found in the Concert Artists/Fidelio catalogue. Click here for further details.Click hereClick here

52. Cover song retrieval

53. Scaling Locality sensitive hashing Locality sensitive hashing Trade-off approximate NN for time complexity Trade-off approximate NN for time complexity 3 to 4 orders of magnitude speed-up 3 to 4 orders of magnitude speed-up No noticeable degradation in performance No noticeable degradation in performance For optimal radius threshold For optimal radius threshold

54. Remix retrieval via LSH

55. Open source: google: “audioDB” Open source: google: “audioDB” Management of tracks, sequences, salience Management of tracks, sequences, salience Automatic indexing parameters Automatic indexing parameters OMRAS2, Yahoo!, AWAL, CHARM, more… OMRAS2, Yahoo!, AWAL, CHARM, more… Web-services interface (SOAP / JSON) Web-services interface (SOAP / JSON) Implementation of LSH for large N ~ 1B Implementation of LSH for large N ~ 1B 1-10 ms whole-track retrieval from 1B vectors 1-10 ms whole-track retrieval from 1B vectors AudioDB: Shingle Nearest Neighbor Search

56. Current deployment Large commercial collections Large commercial collections AWAL ~ 100,000 tracks AWAL ~ 100,000 tracks Yahoo! 2M+ tracks, related song classifier Yahoo! 2M+ tracks, related song classifier Flickr 1B+ Images Flickr 1B+ Images AudioDB: open-source, international consortium of developers AudioDB: open-source, international consortium of developers Google: “audioDB” Google: “audioDB”

57. Conclusions Radius-bounded retrieval model for tracks Radius-bounded retrieval model for tracks Shingles preserve temporal information, high d Shingles preserve temporal information, high d Implements mid-to-high specificity search Implements mid-to-high specificity search Optimal radius threshold from order statistics Optimal radius threshold from order statistics null hypothesis: shingles are drawn from unrelated tracks null hypothesis: shingles are drawn from unrelated tracks LSH requires radius bound, automatic estimate LSH requires radius bound, automatic estimate Scales to 1B shingles+ using LSH Scales to 1B shingles+ using LSH

58. Thanks Malcolm Slaney, Yahoo! Research Inc. Malcolm Slaney, Yahoo! Research Inc. Christophe Rhodes, Goldsmiths, U. of London Christophe Rhodes, Goldsmiths, U. of London Michela Magas, Goldsmiths, U. of London Michela Magas, Goldsmiths, U. of London Funding: EPSRC: EP/E02274X/1 Funding: EPSRC: EP/E02274X/1