1. Audio Processing Mitch Parry

2. Resource!

3. Sound Waves and Harmonic Motion

4. Properties of Sine Waves

5. Resonance as Harmonic Frequencies

6. Nonsinusoidal Waves

7. Sampling and Aliasing

8. Quantization

9. Nyquist and Aliasing

10. Spectral Domain Audio can be represented in the time, frequency, or time-frequency domain

11. Time Domain Generate  Silence Audio Track  Set Rate  Other  2048 Hz. Generate  Tone

12. Time Domain 100 Hz Sine wave @ 2048 Hz sampling rate

13. Frequency Domain… Click here to select all

14. Frequency Domain Select all audio from the track Analyze  Plot Spectrum… Try this: – Function: Rectangular Window – One spike at 100 Hz.

15. Spectrogram… Click “Audio Track”  Spectrum Edit  Preferences  Spectrograms  check “show the spectrum using grayscale colors.”

16. Spectrogram (Time-frequency)

17. Audio Mixing Free Multitrack Downloads http://www.cambridge-mt.com/ms-mtk.htm

18. “Stop Messing with Me” by Sven Bornemark Steinberg Grand Piano Acoustic Guitar Bass Drums Overhead Electric Guitar Electric Guitar Ambience Kick Drum Vocal

19. Audacity: Mixing Tutorial Mixing Tutorial

20. Simple Unmixing Left: Drums + 0.5 * Vocal Right: Guitar + 0.5 * Vocal Remove vocals: – Karaoke track = Left – Right = Drums – Guitar

21. Audacity: Let’s try it. Real example: Norah Jones

22. Removing Hiss Hiss_*.wav

23. Removing Clicks

24. Short-Time Fourier Transform Spectrogram Each frame contributes one column of spectrogram FFT

25. Audacity: Let’s try it.

26. Changing Speed Downsample – Shorten the clip – Increase its pitch

27. Changing Tempo Change length of clip without changing pitch Split into frames, repeat or remove frames

28. Changing Pitch Change pitch without changing length – Increase pitch: Repeat frames and downsample – Decrease pitch: Remove frames and upsample

29. Audacity: Beat Detection Drum track

30. Audacity Audacity Manual More Effects and Analyzers

31. Echo Nest Echo Nest API Analyze Documentation

33. Musical Features Visualizing Structure Rhythm/ Tempo Melody Timbre

34. Visualizing Structure Compute any features Choose similarity metric Visualize self-similarity Foote & Cooper. ICMC 2001.

35. Visualizing Structure High-level segmentation based on novelty score

36. Tempo Diagonal Sums Autocorrelation Foote & Cooper. ICMC 2001 Beat Spectrum

37. Identifying Identical Audio Segmentation – 0.37 second frames – Overlapping by 31/32 FFT – Band Division – Energy computed for 33 non-overlapping logarithmically spaced frequency bands (300-2000Hz) – E(n,m) = energy of band m of frame n. Haitsma & Kalker. ISMIR 2002.

38. Identifying Identical Audio 2 32-bit sub-fingerprint represents increase/decrease between neighboring frequency bands and frame F(n,m) = [E(n-1, m+1) + E(n,m)] -[E(n-1,m) + E(n,m+1)] > 0 n-1 n mm+1… … -+ -+ 257 33 Time (Frames) Frequency Bands Haitsma & Kalker. ISMIR 2002.

39. Identifying Identical Audio 3 Similarity is the bit error rate (BER) between two fingerprints Approximately 3 seconds of audio 256 X 32-bit = 1KB per fingerprint. Haitsma & Kalker. ISMIR 2002.

40. Timbre Similarity Timbre = “Color” of sound Timbre = Type of instrument, voice Similarity decreases in order: – Same recording – Same artist – Same genre Useful for finding different live performances of the same song by an artist Aucouturier & Klapuri. ISMIR 2002.

41. Timbre Similarity 2 Timbre Features – Low-order MFCCs account for timbre. – Hi-order MFCCs account for pitch. – Only use first 8 MFCCs (out of 13). Feature Extraction: – Segment signal into 0.05 sec. non-overlapping frames – Compute first 8 MFCCs for each frame. – Yields ~3600 features (28,800 scalars) per song Aucouturier & Klapuri. ISMIR 2002.

42. Timbre Similarity 3 Gaussian Mixture Model (GMM) – Approximates the distribution of features as the sum of M Gaussian distributions – M = 3 Learn timbre model for each song Timbre similarity between song A and song B is the likelihood that the model for song A generated the features in song B. Aucouturier & Klapuri. ISMIR 2002.

43. Timbre Similarity Examples http://www.csl.sony.fr/~jj/Timbre/timbre.html

44. Audio Textures Generate new audio given examples Analysis – Segment into frames – Extract MFCCs – Similarity Window Weighted Cosine Distance – Transition probabilities proportional to exponential similarity – Segment into sub-clips according to novelty score Lu et. al. ICASSP 2002

45. References Aucouturier, J-J., and Klapuri, A. (2002). "Music Similarity Measures: What's the Use?". Proc. of Int'l Conference on Music Information Retrieval, 3, (pp. 157-163). PDFPDF Foote, J. and Cooper, M. (2001). "Visualizing Musical Structure and Rhythm via Self- Similarity." Proc. of Int'l Computer Music Conference, 27, (pp. 419-422). PDFPDF Haitsma, J. and Kalker, T. (2002). "A Highly Robust Audio Fingerprinting System." Proc. of Int'l Conference on Music Information Retrieval, 3, (pp. 107-115). PDFPDF Lu, L., Li, S., Liu, W., AND Zhang, H. (2002). “Audio Textures.” Proc. of IEEE Int’l Conference on Acoustics, Speech and Signal Processing. PDFPDF Paulus, J. & Klapuri, A. (2002). Measuring the Similarity of Rhythmic Patterns. Proc. of the International Conference on Music Information Retrieval, 3, (pp. 150-156). Paris: IRCAM Centre Pompidou. PDFPDF Scheirer, E. (1998). "Tempo and Beat Analysis of Acoustic Musical Signals.” Journal of the Acoustical Society of America, 103(1), 588-601. PDFPDF Tzanetakis, G., Essl, G., & Cook, P. (2001). Audio Analysis using the DiscreteWavelet Transform. Proc. of WSES International Conference on Acoustics and Music: Theory and Applications. PDFPDF Tzanetakis, G., Ermolinskiy, A. and Cook, P. (2002). "Pitch Histograms in Audio and Symbolic Music Information Retrieval." Proc. of Int'l Conference on Music Information Retrieval, 3, (pp. 31-38). PDFPDF