2. C enter for A uditory and A coustic R esearch Representation of Timbre in the Auditory System Shihab A. Shamma Center for Auditory and Acoustic Research Institute for Systems Research Electrical and Computer Engineering University of Maryland, College Park

3. C enter for A uditory and A coustic R esearch

4. C enter for A uditory and A coustic R esearch sound Central Auditory Stages Collicular Stages Midbrain Nuclei Early Auditory Stages Attributes of Complex Sounds NLL LL TB Anatomy of the Auditory System DCN PVCN AVCN LocationTimbrePitch Spatial maps Computing pitch Harmonic templates ILD, ITD Spectral cues The auditory spectrum IC MGB

5. C enter for A uditory and A coustic R esearch

6. C enter for A uditory and A coustic R esearch

7. C enter for A uditory and A coustic R esearch 4000 2000 1000 500 250 Time(ms) 60 average response Auditory-Nerve Response Patterns to Two-Tone Stimulus

8. C enter for A uditory and A coustic R esearch

9. C enter for A uditory and A coustic R esearch Sound Estimated stimulus spectrum 60 Time (msec) Basilar membrane vibrations Time (msec) 500 A’ B’ C’ Cochlear Analysis Hair cells along the tonotopic axis Characteristic Frequency Axis (CF) Auditory-nerve fibers Lateral Inhibition

10. C enter for A uditory and A coustic R esearch Normal Down-Shift Compress Dilate

11. C enter for A uditory and A coustic R esearch

12. C enter for A uditory and A coustic R esearch

13. C enter for A uditory and A coustic R esearch

14. C enter for A uditory and A coustic R esearch Three envelopes of modulation: Slow (< 30 Hz) Intemediate (< 500 Hz) Fast (< 4 kHz) /come/ /home/ /right/ /away/

15. C enter for A uditory and A coustic R esearch Decomposing a Spectrogram into Dynamic Ripples 

16. C enter for A uditory and A coustic R esearch

17. C enter for A uditory and A coustic R esearch w  (Hz)  = 0.8 cyc/oct Time (ms) w  = 12 Hz  (cyc/oct) Time (ms) | F { }| |TF(,  )| 0  0 T 0 X t (ms) STRF(t,x) B frequency -ww w A

18. C enter for A uditory and A coustic R esearch

19. C enter for A uditory and A coustic R esearch

20. C enter for A uditory and A coustic R esearch Multiscale Cortical Representation of a Spectrogram Frequency Rate (Hz)

21. C enter for A uditory and A coustic R esearch Scale-Rate Decomposition Reconstruction

22. C enter for A uditory and A coustic R esearch MUSICAL TIMBRE

23. C enter for A uditory and A coustic R esearch

24. C enter for A uditory and A coustic R esearch Patterns of Musical Timbre

25. C enter for A uditory and A coustic R esearch

26. C enter for A uditory and A coustic R esearch Timbre Metric for Musical Instruments Guitar Harp Violin Pizz. Violin Bowed Bass Synth A Synth B Oboe Clarinet Flute Horn Trumpet Guitar Harp Violin Pizz. Violin Bowed Bass Synth A Synth B Oboe Clarinet Flute Horn Trumpet Guitar Harp Violin Pizz. Violin Bowed Bass Synth A Synth B Oboe Clarinet Flute Horn Trumpet Guitar Harp Violin Pizz. Violin Bowed Bass Synth A Synth B Oboe Clarinet Flute Horn Trumpet Subjects (1-24) Spectral cues Temporal cues Spectro-temporal cues

27. C enter for A uditory and A coustic R esearch Mapping musical instruments Guitar Trumpet Trumpar ACE Chord A Melody with the Trumpar

28. C enter for A uditory and A coustic R esearch Speech Analysis & Assessment of Inteligibility

29. C enter for A uditory and A coustic R esearch Three envelopes of modulation: Slow (< 30 Hz) Intemediate (< 500 Hz) Fast (< 4 kHz) /come/ /home/ /right/ /away/

30. C enter for A uditory and A coustic R esearch

31. C enter for A uditory and A coustic R esearch Human versus Ferret Sensitivity to Spectrotemporal Modulations

32. C enter for A uditory and A coustic R esearch

33. C enter for A uditory and A coustic R esearch

34. C enter for A uditory and A coustic R esearch Auditory Scene Analysis & Pitch Extraction

35. C enter for A uditory and A coustic R esearch Relevance to Auditory Scene Analysis: Streaming and grouping Frequency Rate (Hz) Working Hypotheses Streaming : Any consistently isolated feature in the multiscale representation can be streamed e.g., spectral patterns (tones or average vocal tract spectra) repetitive temporal dynamics (modulated noise or sinusoidal FM tones) - transients as segmenters Grouping: Harmonicity and its linearly interpolated extensions (pitch extraction and segregation, regular patterns) Shared dynamics (Common onsets and modulations)

36. C enter for A uditory and A coustic R esearch 020406080100120140 0 2 4 6 8 10 12 14 16 250 500 1000 2000 4000 0.5 1.0 2.0 4.0 Cortical Representation of Harmonic & Shifted Spectra Auditory Spectrum Multiscale Representation Scale Frequency Reduced Representation Scale Shifted Spectra are also grouped although they are inharmonic

37. C enter for A uditory and A coustic R esearch

38. C enter for A uditory and A coustic R esearch

39. C enter for A uditory and A coustic R esearch

40. C enter for A uditory and A coustic R esearch Voice Morphing

41. C enter for A uditory and A coustic R esearch

42. C enter for A uditory and A coustic R esearch Morphing Voices

43. C enter for A uditory and A coustic R esearch Auditory Speech and Music Processing Tai Chi, Mounya El-Hilali, Powen Ru Cortical Physiology and Auditory Computations Didier Depireux, Jonathan Fritz, David Klein Jonathan Simon Acknowledgment Supported by: MURI # N00014-97-1-0501 from the Office of Naval Research # NIDCD T32 DC00046-01 from the NIDCD # NSFD CD8803012 from the National Science Foundation