2. Meena Ramani 04/12/06 EEL 6586 Automatic Speech Processing

3. Topics to be covered The incredible sense of hearing 1 Lecture 1: The incredible sense of hearing 1 Anatomy Perception of Sound The incredible sense of hearing 2 Lecture 2: The incredible sense of hearing 2 Psychoacoustics Hearing aids and cochlear implants

4. The incredible sense of hearing-2 “ Behind these unprepossessing flaps lie structures of such delicacy that they shame the most skillful craftsman" -Stevens, S.S. [Professor of Psychophysics, Harvard University]

5. How do we hear?

6. Threshold of Hearing

7. Equal loudness curves

8. The Bass Loss Problem Rock music Too low  no bass Too high  too much bass

9. Threshold variation with age

10. The Audiogram

11. The Audiogram (contd.) Pure tone audiogram [250 500 1K 2K 4K 6k] Hz <20 dB HL is Normal Hearing

12. Loudness Growth Curve

13. Otoacoustic emissions The ear produces some sounds! –OHC-outer hair cell Used to test hearing for infants & check if patient is feigning a loss

14. Monoaural beats If two tones are presented monaurally with a small frequency difference, a beating pattern can be heard 500 & 502 Hz500 & 520 Hz Interaction of the two tones in the same auditory filter Waveform: 150 Hz + 170 Hz

15. Beating can also be heard when the tones are presented to different ears! Beating arises from neural interaction Only perceived if the tones are sufficiently close in frequency 500 Hz - left520 Hz - rightbinaural Binaural beats

16. The case of the missing fundamental Telephone BW: 300-3400 Hz How do we know the pitch? Primary Auditory cortex Pitch sensitive neurons [Bendor and Wang, Nature 2005] Neuron responds to fundamental and harmonics What are the I/Ps to these neuron? How do spikes represent periodic, temporal and spectral information?

17. Matlab code available Feed it a wav file Spits out PSTH Auditory-periphery model (Zhang et al. ~2001)

18. Critical bands Equally loud, close in frequency Same IHCs Slightly louder Equally loud, separated in freq. Different IHCs Twice as loud Psychoacoustic experiments

19. Critical Band (cont.) Proposed by Fletcher How to measure? –S/N ratio vs noise BW CB ~= 1.5mm spacing on BM 24 such band pass filters BW of the filters increases with f c Logarithmic relationship –Weber’s law example Bark scale Center FreqCritical BW 10090 20090 500110 1000150 2000280 5000700 100001200

20. Critical bands for HI

21. “You know I can't hear you when the water is running!” MASKINGMASKING

22. Frequency Masking Masking occurs because two frequencies lie within a critical band and the higher amplitude one masks the lower amplitude signal Masking can be because of broad band, narrowband noise, pure and complex tones Low frequency broad band sounds mask the most –Eg. Truck on road, water flowing Masking threshold –Amount of dB for test tone to be just audible in presence of noise

23. Temporal Aspects of Masking Simultaneous Masking Pre-Stimulus/Backward/Premasking –1 st test tone 2 nd Masker Poststimulus/Forward/Postmasking –1 st Masker 2 nd test tone

24. Simultaneous masking –Duration >200ms constant test tone threshold –Assume hearing system integrates over a period of 200ms Postmasking –Decay in effect of masker for 100ms –More dominant Premasking –Takes place 20ms before masker is on!! –Each sensation is not instantaneous, requires build-up time Quick build up for loud maskers Slower build up for softer maskers –Less dominant effect Temporal Aspects of Masking (contd.)

25. Temporal masking for HI

26. Meena Ramani 04/14/06 EEL 6586 Automatic Speech Processing

27. Normal Hearing Sensorineural Hearing Loss Mild to Severe Loss [10 20 30 60 80 90] dB HL What do the hearing impaired hear?

28. Facts on Hearing Loss in Adults One in every ten (28 million) Americans has hearing loss. The vast majority of Americans (95% or 26 million) with hearing loss can have their hearing loss treated with hearing aids. Only 6 million use HAs Millions of Americans with hearing loss could benefit from hearing aids but avoid them because of the stigma.

29. Types of Hearing aids Behind The ear In the Ear In the Canal Completely in the canal

30. Anatomy of a Hearing Aid Microphone Tone hook Volume control On/off switch Battery compartment

31. Ear Mold Measurements Hearing Aid Fitting

32. Acclimatization effect Auditory cortex  brain plasticity Time for the HI to reuse the HF information: Acclimatization effect How does this affect HA fitting? –Multiple fitting sessions –Initial fitting should be optimum one

33. So doc, what is the fitting methodology employed by the hearing aid company to compensate for my hearing loss? Not-so-average Joe (PhD EE/Speech person) CONFIDENTIAL ?

34. So, do you want your HA to: 1)Always be comfortably loud 2)Equalize loudness across frequencies 3)Normalize loudness …? ? best Which fitting methodology is the best ?

35. Existing HL compensation algorithms Rationale  Adhoc: Half Gain, POGO  Make speech comfortable: NAL-R  Loudness normalization: IHAFF, Fig 6  Loudness equalization: DSL

36. Sensorineural hearing loss [10 20 30 60 80 90] dB HL Speech level= 65 dBA Spectrograms and sound files Normal hearing Hearing impairedHI with Linear gain HI with DSL gain HI with RBC gain Section Two

37. Speech Intelligibility Objective Measures AI, STI Speech Quality Objective Measures PESQ Subjective Measures MOS Speech Intelligibility (SI): The degree to which speech can be understood Performance metrics Subjective Measures HINT Speech Quality: “Does the speech match your expectations?”

38. Performance metrics (contd.) Objective speech quality measure –Perceptual Evaluation of Subjective Quality (PESQ) Subjective speech quality measure –Mean Opinion Score (MOS) Subjective speech intelligibility measure –Hearing In Noise Test (HINT) Reference signal Comparison signal Score

39. Hearing In Noise Test (HINT)

40. Subjective listening experiments Audiograms of the HI patients Location: Shands speech & hearing clinic (sound proof booth) Subjects: 15 HI people –PTA: 40-70 dB HL 15 normal hearing people Tools used: Matlab HINT and MOS GUIs

41. Subjective HINT and MOS scores for RBC: hearing impaired, cell phone speech RBC has a 7 dB improvement in SI when compared to DSL MOS scores reveal that RBC has a quality rating of ‘Good’

42. Subjective HINT and MOS scores for RBC: normal hearing, cell phone speech RBC has a 12 dB improvement in SI when compared to DSL MOS scores reveal that RBC has a quality rating of ‘Good’

43. Cochlear Implants The first fully functional Brain Machine Interface (BMI) Definition: A device that electrically stimulates the auditory nerve of patients with severe-to- profound hearing loss to provide them with sound and speech information

44. Who is a candidate? Severe-to profound sensorineural hearing loss Hearing loss did not reach severe-to- profound level until after acquiring oral speech and language skills Limited benefit from hearing aids

45. Worldwide: –Over 100,000 multi-channel implants At Univ of Florida: –Implanted first patient in 1985 –Currently follow over 400 cochlear patients CI statistics

46. Technical and Safety Issues Magnetic Resonance Imaging Surgical issues

47. How does the Cochlea encode frequencies?

50. Example: New Freedom

52. CI characteristics 1.Electrode design –Number of electrodes, electrode configuration 2. Type of stimulation –Analog or pulsatile 3. Transmission link –Transcutaneous or percutaneous 4. Signal processing –Waveform representation or feature extraction

53. Signal processing Compressed Analog (CA) Continuous Interleaved Sampling (CIS) Multiple Peak (MPEAK ) Spectral Maxima Sound Processor (SMSP) Spectral Peak (SPEAK)

54. Compressed Analog (CA) approach

55. CA activation signals

56. Continuous Interleaved Sampling (CIS)

57. CIS activation signals

58. Multiple Peak (MPEAK)

59. MPEAK activated electrodes

60. Spectral Maxima Sound Processor (SMSP)

61. SMSP activated electrodes

62. Spectral Peak (SPEAK)

63. SPEAK activated electrodes

64. Outcomes for Post-lingual Adults Wide range of success Most score 90-100% on AV sentence materials Majority score > 80% on high context Performance more varied on single word tests

65. Auditory Brainstem Implant Approved October 20, 2000 Uses the Nucleus 24 system processors Plate array with 21 electrodes

66. Review-1 Pinna: ITDs,IIDs: Horizontal localization Reflections: Vertical localization Ear canal: ¼ wave resonance 1-3 kHz Middle ear: Amplification by lever action and by area Stapedius reflex Cochlea: IHCs/OHCs: convert mechanical to electrical Place theory: frequency analysis Missing fundamental

67. Review-2 Adaptation: AN firing sensitive to changes Otoacoustic emissions: Produced by movement of OHCs Beats: Monaural & binaural Measurement of hearing: Audiogram: threshold of hearing Threshold variation with age Equal loudness curves Bass loss problem: discrimination against LFs

68. Review-3 Critical bands: used for efficient encoding Bark scale Masking: Frequency: LFs mask more Temporal: simultaneous, pre and post Hearing impairment: Hearing aids: external to cochlea Cochlear implants: inside cochlea