Recent Research in Musical Timbre Perception James W. Beauchamp University of Illinois at Urbana-Champaign Andrew B. Horner Hong University of Science and Technology Michael D. Hall James Madison University, Harrisonburg, VA James W. Beauchamp University of Illinois at Urbana-Champaign Andrew B. Horner Hong University of Science and Technology Michael D. Hall James Madison University, Harrisonburg, VA

Starting Point Timbre experiments are based on musical instrument sounds.

Starting Point Timbre experiments are based on musical instrument sounds. Perform short-time spectral analysis. Timbre experiments are based on musical instrument sounds. Perform short-time spectral analysis.

Starting Point Timbre experiments are based on musical instrument sounds. Perform short-time spectral analysis. Identify parameters of ST spectrum: Timbre experiments are based on musical instrument sounds. Perform short-time spectral analysis. Identify parameters of ST spectrum:

Starting Point Timbre experiments are based on musical instrument sounds. Perform short-time spectral analysis. Identify parameters of ST spectrum:  Partial (harmonic) amplitudes - Time variation - Spectral envelope (centroid, irregularity, etc.) Timbre experiments are based on musical instrument sounds. Perform short-time spectral analysis. Identify parameters of ST spectrum:  Partial (harmonic) amplitudes - Time variation - Spectral envelope (centroid, irregularity, etc.)

Starting Point Timbre experiments are based on musical instrument sounds. Perform short-time spectral analysis. Identify parameters of ST spectrum:  Partial (harmonic) amplitudes - Time variation - Spectral envelope (centroid, irregularity, etc.)  Partial (harmonic) frequencies - Time variation - Inharmonicity Timbre experiments are based on musical instrument sounds. Perform short-time spectral analysis. Identify parameters of ST spectrum:  Partial (harmonic) amplitudes - Time variation - Spectral envelope (centroid, irregularity, etc.)  Partial (harmonic) frequencies - Time variation - Inharmonicity

Methods for Studying Timbre Stimuli Preparation In Freq. Domain –Simplification –Perturbation –Normalization

Methods for Studying Timbre Listener Experiments –Discrimination (pairs) –Timbral Distance Estimation –Classification –Identification Stimuli Preparation In Freq. Domain –Simplification –Perturbation –Normalization

Methods for Studying Timbre Listener Experiments –Discrimination (pairs) –Timbral Distance Estimation –Classification –Identification Stimuli Preparation In Freq. Domain –Simplification –Perturbation –Normalization Data Processing/Presentation –Discrimination (sensitivity) scores/plots –Multidimensional Scaling –Correspondence (R 2 ) Measurements

Studies Reviewed 1999 Discrimination Study 2006 Discrimination Study 2006 Multidimensional Scaling (MDS) Study 2009 Discrimination/Classification Study

1999 Discrimination Study (McAdams, Beauchamp, Meneguzzi, JASA) Seven reference sounds –clarinet, flute, oboe, trumpet, violin, harpsichord, marimba

1999 Discrimination Study (McAdams, Beauchamp, Meneguzzi, JASA) Seven reference sounds –clarinet, flute, oboe, trumpet, violin, harpsichord, marimba Equalize F 0, loudness, and duration.

1999 Discrimination Study (McAdams, Beauchamp, Meneguzzi, JASA) Seven reference sounds –clarinet, flute, oboe, trumpet, violin, harpsichord, marimba Equalize F 0, loudness, and duration. Test sounds: Apply six spectrotemporal simplifications.

1999 Discrimination Study (McAdams, Beauchamp, Meneguzzi, JASA) Seven reference sounds –clarinet, flute, oboe, trumpet, violin, harpsichord, marimba Equalize F 0, loudness, and duration.. Test sounds: Apply six spectrotemporal simplifications. Subjects discriminate between original and simplified sounds.

1999 Discrimination Study Results Spectral envelope smoothing96% Spectral flux elimination91% Amplitude envelopes smoothing66% Frequency envelopes smoothing 70% Freq. envs. harmonic locking69% Frequency variations elimination71% Discrim Score

2006 Discrimination Study Horner, Beauchamp, and So JAES Eight sustained musical instrument tones –bassoon, clarinet, flute, horn, oboe, alto sax, trumpet, violin

2006 Discrimination Study Horner, Beauchamp, and So JAES Eight sustained musical instrument tones –bassoon, clarinet, flute, horn, oboe, alto sax, trumpet, violin Modified by fixed random transfer function

2006 Discrimination Study Horner, Beauchamp, and So JAES Eight sustained musical instrument tones –bassoon, clarinet, flute, horn, oboe, alto sax, trumpet, violin Modified by fixed random transfer function – F 0, loudness, duration, centroid preserved

2006 Discrimination Study Horner, Beauchamp, and So JAES Eight sustained musical instrument tones –bassoon, clarinet, flute, horn, oboe, alto sax, trumpet, violin Modified by fixed random transfer function – F 0, loudness, duration, centroid preserved Typical spectral envelopes:

2006 Discrimination Study Horner, Beauchamp, and So JAES Objective: To discover which metrics based on the time-varying harmonic amplitudes give the best correspondence to discrimination between original and modified tones.

2006 Discrimination Study Horner, Beauchamp, and So JAES Objective: To discover which metrics based on the time-varying harmonic amplitudes give the best correspondence to the discrimination data. Best results: obtained by relative-amplitude (harmonic) spectral error:

2006 Discrimination Study Horner, Beauchamp, and So JAES R 2 =0.81 Discrimination vs. error level (  ):

2006 Discrimination Study Discrimination vs. rel-amp spec error: R 2 =0.90 for a=1.0 Horner, Beauchamp, and So JAES

2006 MDS Study Ten sustained musical instrument tones –bassoon, cello, clarinet, flute, horn, oboe, recorder, alto sax, trumpet, violin Beauchamp, Horner, Koehn, and Bay (ASA Honolulu)

2006 MDS Study Ten sustained musical instrument tones –bassoon, cello, clarinet, flute, horn, oboe, recorder, alto sax, trumpet, violin F 0, loudness, duration, attack & decay times, and average spectral centroid are equalized. Beauchamp, Horner, Koehn, and Bay (ASA Honolulu)

2006 MDS Study Ten sustained musical instrument tones –bassoon, cello, clarinet, flute, horn, oboe, recorder, alto sax, trumpet, violin F 0, loudness, duration, attack & decay times, and average spectral centroid are equalized. Two types of tones: static (flux removed) and dynamic (flux retained). Beauchamp, Horner, Koehn, and Bay (ASA Honolulu)

2006 MDS Study Ten sustained musical instrument tones –bassoon, cello, clarinet, flute, horn, oboe, recorder, alto sax, trumpet, violin F 0, loudness, duration, attack & decay times, and average spectral centroid are equalized. Two types of tones: static (flux removed) and dynamic (flux retained). Subjects estimate timbral dissimilarity between instruments. Beauchamp, Horner, Koehn, and Bay (ASA Honolulu)

2006 MDS Study Ten sustained musical instrument tones –bassoon, cello, clarinet, flute, horn, oboe, recorder, alto sax, trumpet, violin F 0, loudness, duration, attack & decay times, and average spectral centroid are equalized. Two types of tones: static (flux removed) and dynamic (flux retained). Subjects estimate timbral dissimilarity between instruments. Data processed by two multi-dimensional scaling (MDS) programs (SPSS & Matlab). Beauchamp, Horner, Koehn, and Bay (ASA Honolulu)

2006 MDS Study Beauchamp, Horner, Koehn, and Bay (ASA Honolulu) Acoustical Correlates to Test: Even/Odd: Ratio of even and odd harmonic rms amplitudes.

2006 MDS Study Beauchamp, Horner, Koehn, and Bay (ASA Honolulu) Acoustical Correlates to Test: Even/Odd: Ratio of even and odd harmonic rms amplitudes Spectral IRregularity: Degree of jaggedness of a spectrum.

2006 MDS Study Beauchamp, Horner, Koehn, and Bay (ASA Honolulu) Acoustical Correlates to Test: Even/Odd: Ratio of even and odd harmonic rms amplitudes Spectral IRregularity: Degree of jaggedness of a spectrum. Spectral Centroid Variation: Standard deviation of the spectral centroid normalized by average value. For Dynamic Tones Only:

2006 MDS Study Beauchamp, Horner, Koehn, and Bay (ASA Honolulu) Acoustical Correlates to Test: Even/Odd: Ratio of even and odd harmonic rms amplitudes Spectral IRregularity: Degree of jaggedness of a spectrum. Spectral Centroid Variation: Standard deviation of the spectral centroid normalized by average value. Spectral INcoherence: Degree of spectral change relative to the average spectrum (same as flux). For Dynamic Tones Only:

2006 MDS Study Beauchamp, Horner, Koehn, and Bay (ASA Honolulu) 2D MDS Results: Static Tone Case SPSS algorithm Correlations: E/O:R=0.78 SIR:R=0.69 Stress=0.12

2006 MDS Study Beauchamp, Horner, Koehn, and Bay (ASA Honolulu) 2D MDS Results: Static Tone Case Matlab algorithm Correlations: E/O:R=0.79 SIR:R=0.75 Stress=0.12

2006 MDS Study Beauchamp, Horner, Koehn, and Bay (ASA Honolulu) 2D MDS Results: Dynamic Tone Case SPSS algorithm Correlations: E/O:R=0.71 SCV:R=0.68 SIN:R=0.56 SIR:R=0.39 Stress=0.17

2006 MDS Study Beauchamp, Horner, Koehn, and Bay (ASA Honolulu) 2D MDS Results: Dynamic Tone Case Matlab algorithm Correlations: E/O:R=0.69 SCV:R=0.68 SIN:R=0.53 SIR:R=0.40 Stress=0.15

2009 Study Hall and Beauchamp (Canadian Acoustics) Goals/Purpose Exp. 1. Relative importance of spectral vs. temporal cues: Compare listener discrimination and classification performance for interpolations between two (impoverished) instruments with respect to spectral envelope and amplitude-vs.-time envelope.

2009 Study Hall and Beauchamp (Canadian Acoustics) Goals/Purpose Exp. 1. Relative importance of spectral vs. temporal cues: Compare listener discrimination and classification performance for interpolations between two (impoverished) instruments with respect to spectral envelope and amplitude-vs.-time envelope. Exp. 2. Relative importance of spectral envelope (formant) structure vs. spectral centroid: Compare discrimination/classification performance for interpolated tones vs. tones obtained by filtration which matches the centroids of the interpolated tones.

2009 Study Hall and Beauchamp (Canadian Acoustics) Experiment 1 Method Reference stimuli: Impoverished (static) violin and trombone sounds. (Each sound has a fixed spectrum and a single amplitude-vs.-time envelope.)

2009 Study Hall and Beauchamp (Canadian Acoustics) Experiment 1 Method Reference stimuli: Impoverished (static) violin and trombone sounds. (Each sound has a fixed spectrum and a single amplitude-vs.-time envelope.) Test stimuli: A 4  4 array of sounds is created based on interpolations of the spectral envelope and the amplitude-vs- time envelope between the violin and trombone timbres. VnI 01 I 02 I 03 I 10 I 11 I 12 I 13 I 20 I 21 I 22 I 23 I 30 I 31 I 32 Tr Temporal Spectral

2009 Study Hall and Beauchamp (Canadian Acoustics) Experiment 1 Method Reference stimuli: Impoverished (static) violin and trombone sounds. (Each sound has a fixed spectrum and a single amplitude-vs.-time envelope.) Test stimuli: A 4  4 array of sounds is created based on interpolations of the spectral envelope and the amplitude-vs- time envelope between the violin and trombone timbres. Interpolation steps: Test tones differ from reference (original) tones by 1, 2, or 3 steps along either the spectral envelope or amplitude envelope dimension or both (3 steps gives the opposite timbre.).

2009 Study Hall and Beauchamp (Canadian Acoustics) Experiment 1 Method Reference stimuli: Impoverished (static) violin and trombone sounds. (Each sound has a fixed spectrum and a single amplitude-vs.-time envelope.) Test stimuli: A 4  4 array of sounds is created based on interpolations of the spectral envelope and the amplitude-vs- time envelope between the violin and trombone timbres. Interpolation steps: Test tones differ from reference (original) tones by 1, 2, or 3 steps along either the spectral envelope or amplitude envelope dimension or both (3 steps gives the opposite timbre.). Subjects’ tasks: - 1) to discriminate tone pairs. - 2) to classify tones as ‘violin’, ‘trombone’, or ‘other’.

2009 Study Hall and Beauchamp (Canadian Acoustics) Experiment 1 Results Discrimination: Note: Low sensitivity to temporal changes. High sensitivity to spectral changes. reference stimuli:

2009 Study Hall and Beauchamp (Canadian Acoustics) Experiment 1 Results Classification: Note: Low sensitivity to temporal changes. High sensitivity to spectral changes.

2009 Study Hall and Beauchamp (Canadian Acoustics) Experiment 2 Method Reference stimulus: Original impoverished violin.

2009 Study Hall and Beauchamp (Canadian Acoustics) Experiment 2 Method Reference stimulus: Original impoverished violin. Test stimuli: -1) 3 violin tones interpolated with respect to spectral envelope (steps 1-3) (original violin amp env kept).

2009 Study Hall and Beauchamp (Canadian Acoustics) Experiment 2 Method Reference stimulus: Original impoverished violin. Test stimuli: -1) 3 violin tones interpolated with respect to spectral envelope (steps 1-3) (original violin amp env kept). -2) 3 violin tones low-pass filtered in steps to match the spectral centroids of the 3 interpolated tones of 1).

2009 Study Hall and Beauchamp (Canadian Acoustics) Experiment 2 Method Reference stimulus: Original impoverished violin. Test stimuli: -1) 3 violin tones interpolated with respect to spectral envelope (steps 1-3) (original violin amp env kept). -2) 3 violin tones low-pass filtered in steps to match the spectral centroids of the 3 interpolated tones of 1). Subjects’ tasks: Discrimination and classification as in Exp. 1. (Which has the greater effect? Interpolation or filtration?)

2009 Study Hall and Beauchamp (Canadian Acoustics) Experiment 2 Results Discrimination: Classification:

Conclusion Summary 1999 discrimination study: –Spectral envelope detail and spectral flux are important for dynamic musical sounds, and they are more important than temporal detail.

Conclusion Summary 1999 discrimination study: –Spectral envelope detail and spectral flux are important for dynamic musical sounds, and they are more important than temporal detail discrimination study: –The ability to hear differences between dynamic tones with matched spectral centroids and randomly altered spectra correlates strongly with relative spectral- amplitude differences.

Conclusions 2006 MDS study: –Using centroid and attack/decay normalized tones, there is strong evidence that even/odd ratio and other spectral envelope details are important for timbral differences of impoverished (static) and dynamic musical instrument tones.

Conclusions 2006 MDS study: –Using centroid and attack/decay normalized tones, there is strong evidence that even/odd ratio and other spectral envelope details are important for timbral differences of impoverished (static) and dynamic musical instrument tones discrimination/classification study: –Using spectral interpolation with respect to both spectral and temporal dimensions on impoverished violin and trombone tones: 1)Spectral differences were found to be more important than temporal differences. 2)Detailed spectral differences were much more important than mere spectral centroid differences.