1. pcpandey@ee.iitb.ac.in EE Dept., IIT Bombay Workshop “AICTE Sponsored Faculty Development Programme on Signal Processing and Applications", Dept. of Electrical Engineering, VJTI, Mumbai, Feb 23-27, 2015, Coordinator: Prof. Alice N. Cheeran. Session: Feb 27, Friday, 10:00 a.m. to 11:30 a.m. ============================================================================ Signal processing for persons with sensorineural hearing loss: Challenges and some solutions P. C. Pandey IIT Bombay

2. pcpandey@ee.iitb.ac.in EE Dept., IIT Bombay 2/15 Outline A.Speech & Hearing B.Sliding-band Dynamic Range Compression (Ref: N. Tiwari & P. C. Pandey, NCC 2014, Paper No.1569847357) C.Automated modification of consonant-vowel ratio of stops (Ref: A. R. Jayan & P. C. Pandey, Int. J. Speech Technology, vol. 18, pp. 113–130, 2015) Outline A.Speech & Hearing B.Sliding-band Dynamic Range Compression (Ref: N. Tiwari & P. C. Pandey, NCC 2014, Paper No.1569847357) C.Automated modification of consonant-vowel ratio of stops (Ref: A. R. Jayan & P. C. Pandey, Int. J. Speech Technology, vol. 18, pp. 113–130, 2015)

3. pcpandey@ee.iitb.ac.in EE Dept., IIT Bombay 3/15 Part A Speech & Hearing P. C. Pandey, "Signal processing for persons with sensorineural hearing loss: Challenges and some solutions,” AICTE Sponsored Faculty Development Programme on Signal Processing and Applications, Dept. of Electrical Engineering, VJTI, Mumbai, Feb. 23-27, 2015. ============================================================================

4. pcpandey@ee.iitb.ac.in EE Dept., IIT Bombay 4/15 Speech Production Excitation source & filter model Excitation: voiced/unvoiced glottal, frication Filtering: vocal tract filter

5. pcpandey@ee.iitb.ac.in EE Dept., IIT Bombay 5/15 Speech segments Words Syllables Phonemes Sub-phonemic segments Phonemes: basic speech units Vowels: Pure vowels, Diphthongs Consonants: Semivowels, Stops, Fricatives, Affricates, Nasals /aba/ /apa/ /aga/ /ada/

6. pcpandey@ee.iitb.ac.in EE Dept., IIT Bombay 6/15 Phonemic features Modes of excitation Glottal: Unvoiced (constriction at the glottis), Voiced (glottal vibration) Frication: Unvoiced (constriction in vocal tract), Voiced (constriction in v.t. & glottal vibration) Movement of articulators Continuant (steady-state v.t. configuration): vowels, nasal stops, fricatives Non-continuant (changing v.t.): diphthongs, semivowels, oral stops (plosives) Place of articulation (place of maximum constriction in v.t.) Bilabial, Labio-dental, Linguo-dental, Alveolar, Palatal, Velar, Gluttoral Changes in voicing frequency (Fo) Supra-segmental features: Intonation, Rhythm

7. pcpandey@ee.iitb.ac.in EE Dept., IIT Bombay 7/15 Hearing Mechanism Peripheral auditory system External ear: sound collection ○ Pinna ○ Auditory canal Middle ear: impedance matching ○ Ear drum ○ Middle ear bones Inner ear (cochlea): analysis & transduction Auditory nerve: transmission of neural impulses Central auditory system Information processing & interpretation

8. pcpandey@ee.iitb.ac.in EE Dept., IIT Bombay 8/15 Tonotopic map of cochlea Auditory system

9. pcpandey@ee.iitb.ac.in EE Dept., IIT Bombay 9/15 Hearing Impairment Types of hearing losses Conductive Sensorineural Central Functional Sensorineural hearing loss Associated with abnormalities in the cochlear hair cells or the auditory nerve. Causes: aging, excessive noise exposure, infection, adverse effect of medicines, congenital. Hearing Impairment Types of hearing losses Conductive Sensorineural Central Functional Sensorineural hearing loss Associated with abnormalities in the cochlear hair cells or the auditory nerve. Causes: aging, excessive noise exposure, infection, adverse effect of medicines, congenital.

10. pcpandey@ee.iitb.ac.in EE Dept., IIT Bombay 10/15 Effects of sensorineural hearing loss Elevated hearing thresholds: inaudibility of low-level sounds Reduced dynamic range & loudness recruitment (abnormal loudness growth): distortion of loudness relationship among speech components Increased temporal masking: poor detection of acoustic landmarks Increased spectral masking (widening of auditory filters): reduced ability to sense spectral shapes >> Poor intelligibility and degraded perception of speech, particularly in noisy environment. Effects of sensorineural hearing loss Elevated hearing thresholds: inaudibility of low-level sounds Reduced dynamic range & loudness recruitment (abnormal loudness growth): distortion of loudness relationship among speech components Increased temporal masking: poor detection of acoustic landmarks Increased spectral masking (widening of auditory filters): reduced ability to sense spectral shapes >> Poor intelligibility and degraded perception of speech, particularly in noisy environment.

11. pcpandey@ee.iitb.ac.in EE Dept., IIT Bombay 11/15 Signal Processing in Hearing Aids Currently available techniques Frequency selective amplification: improves audibility but not necessarily intelligibility Automatic volume control: not effective in improving intelligibility Multichannel dynamic range compression (with settable attack & release times, compression ratios): effectiveness reduced due to processing artifacts Signal Processing in Hearing Aids Currently available techniques Frequency selective amplification: improves audibility but not necessarily intelligibility Automatic volume control: not effective in improving intelligibility Multichannel dynamic range compression (with settable attack & release times, compression ratios): effectiveness reduced due to processing artifacts

12. pcpandey@ee.iitb.ac.in EE Dept., IIT Bombay 12/15 Techniques under development Noise suppression Distortion-free dynamic range compression Techniques for reducing the effects of increased spectral masking o Binaural dichotic presentation o Spectral contrast enhancement o Multi-band frequency compression Improvement of consonant-to-vowel ratio (CVR): for reducing the effects of increased temporal masking Techniques under development Noise suppression Distortion-free dynamic range compression Techniques for reducing the effects of increased spectral masking o Binaural dichotic presentation o Spectral contrast enhancement o Multi-band frequency compression Improvement of consonant-to-vowel ratio (CVR): for reducing the effects of increased temporal masking

13. pcpandey@ee.iitb.ac.in EE Dept., IIT Bombay 13/15 Analog Hearing Aids Pre-amp → AVC → Freq. Response → Amp. Digital Hearing Aids Pre-amp & AVC → ADC → Multi-band Amplitude Compr. & Freq. Resp. → DAC & Amp. Existing Problems Poor intelligibility in noisy environment & reverberation Distortions due to multiband amplitude compression Poor speech perception due to increased spectral & temporal masking Visit to audiologist for change of settings Analog Hearing Aids Pre-amp → AVC → Freq. Response → Amp. Digital Hearing Aids Pre-amp & AVC → ADC → Multi-band Amplitude Compr. & Freq. Resp. → DAC & Amp. Existing Problems Poor intelligibility in noisy environment & reverberation Distortions due to multiband amplitude compression Poor speech perception due to increased spectral & temporal masking Visit to audiologist for change of settings

14. pcpandey@ee.iitb.ac.in EE Dept., IIT Bombay 14/15 Proposed Hearing Aids Distortion-free dynamic range compression & adjustable frequency response Noise suppression & de-reverberation Processing for reducing the effects of increased spectral masking Processing for reducing the effects of increased temporal masking Implementation of signal processing using a low-power DSP chip with acceptable signal delay (< 60 ms) User selectable settings Proposed Hearing Aids Distortion-free dynamic range compression & adjustable frequency response Noise suppression & de-reverberation Processing for reducing the effects of increased spectral masking Processing for reducing the effects of increased temporal masking Implementation of signal processing using a low-power DSP chip with acceptable signal delay (< 60 ms) User selectable settings

15. pcpandey@ee.iitb.ac.in EE Dept., IIT Bombay 15/15 Some Solutions for Improving Speech Perception by Listeners with Moderate-to- severe Sensorineural Loss Sliding-band dynamic range compression as a solution to the problem posed by loudness recruitment Automated modification of consonant-vowel ratio of stop consonants as a solution to the problem posed by increased intraspeech spectral and temporal masking. Implementation using a 16-bit fixed-point DSP processor & testing for satisfactory operation. Some Solutions for Improving Speech Perception by Listeners with Moderate-to- severe Sensorineural Loss Sliding-band dynamic range compression as a solution to the problem posed by loudness recruitment Automated modification of consonant-vowel ratio of stop consonants as a solution to the problem posed by increased intraspeech spectral and temporal masking. Implementation using a 16-bit fixed-point DSP processor & testing for satisfactory operation.

16. pcpandey@ee.iitb.ac.in EE Dept., IIT Bombay To be continued in Part B

17. pcpandey@ee.iitb.ac.in EE Dept., IIT Bombay Workshop: AICTE Sponsored Faculty Development Programme on Signal Processing and Applications", Dept. of Electrical Engineering, VJTI, Mumbai, Feb 23-27, 2015, Coordinator: Prof. alice N. Cheeran. Speaker: Prof. P. C. Pandey, EE Dept, IIT Bombay Topic: Signal processing for persons with sensorineural hearing loss: Challenges and some solutions Abstract Sensorineural hearing loss is caused by abnormalities in the cochlear hair cells or the auditory nerve. It occurs due to aging, excessive exposure to noise, infection, or congenital abnormalities. It is generally associated with elevated hearing thresholds, reduced dynamic range and loudness recruitment, and increased temporal and spectral masking, leading to degraded perception of speech, particularly in noisy environment. To address these problems, several signal processing techniques have been reported. Most of these techniques are not suited for use in hearing aids due to distortions caused by processing related artifacts, computational complexities in implementing the technique for real-time processing using a low-power processor, or excessive signal delay which may interfere with lipreading. We have investigated two novel techniques: (i) a sliding-band dynamic range compression as a solution to the problem posed by loudness recruitment [1], and (ii) automated modification of consonant-vowel ratio of stop consonants as a solution to the problem posed by increased intraspeech spectral and temporal masking [2]. Persons with sensorineural loss generally have a highly reduced dynamic range of hearing, with a significant frequency- dependent elevation of hearing threshold levels without corresponding increase in the upper comfortable listening levels. To present the sounds comfortably within the limited dynamic range of the listener, analog hearing aids generally use single-band compression with the gain being dependent on the time-varying signal level. As the power is mostly contributed by the low-frequency components, the high frequency components may become inaudible and distortions in temporal envelope may get introduced. In multiband compression available in most digital hearing aids, the spectral components of the input signal are divided in multiple bands and the gain for each band is calculated on the basis of signal power in that band. This type of processing can introduce spurious spectral distortions. Use of a large number of bands reduces spectral contrasts and the modulation depth of speech, resulting in an adverse effect on the perception of certain speech cues. Further, the frequency response of a multiband compression system has a time-varying magnitude response without corresponding variation in the phase response, which can cause audible distortions, particularly for non-speech audio. These distortions may partly offset the advantages of dynamic range compression for the hearing-impaired listener. In order to significantly reduce the temporal and spectral distortions associated with the currently used single-band and multiband compressions in hearing aids, a "sliding- band compression" has been developed. It involves calculating a frequency-dependent gain function, in which the gain for each spectral sample is determined by the short-time power in an auditory critical band centered at it. The gain calculation takes into account the specified hearing thresholds, compression ratios, and attack and release times. Unlike single-band compression, it does not result in any significant temporal distortions because the effect of short-time energy of a spectral component on other spectral components is limited to those located within a critical bandwidth. Due to use of sliding critical bands for calculating the power spectrum, formant transitions do not result in discontinuities in the processed output. The technique is realized using an FFT-based analysis-synthesis method which masks phase related discontinuities.

18. pcpandey@ee.iitb.ac.in EE Dept., IIT Bombay Increasing the level of the consonant segments relative to the nearby vowel segments, known as consonant-vowel ratio (CVR) modification, is reported to be effective in improving speech intelligibility for listeners in noisy backgrounds and for hearing impaired listeners. A technique for real-time CVR modification of stops using the rate of change of spectral centroid for detection of spectral transitions is presented. Its effectiveness in improving the recognition of consonants in the presence of speech spectrum shaped noise is evaluated by conducting listening tests on normal-hearing subjects. At lower values of SNR, there was an increase of 7 - 21% in recognition scores and an equivalent SNR advantage of 3 dB. Both the techniques have been implemented using a 16-bit fixed-point DSP processor with on-chip FFT hardware and have been tested for satisfactory real-time operation. They can be integrated with other FFT-based signal processing techniques in hearing aids. References [1] N. Tiwari and P. C. Pandey, A sliding-band dynamic range compression for use in hearing aids, Proc. National Conference on Communications 2014 (NCC 2014), Kanpur, Feb. 28 - Mar. 2, 2014, paper no. 1569847357. [2] A. R. Jayan & P. C. Pandey, Automated modification of consonant-vowel ratio of stops for improving speech intelligibility, Int. J. Speech Technology, vol. 18, pp. 113–130, 2015. DOI: 10.1007/s10772-014-9254-4. Dr. Prem C. Pandey Dr. Pandey is a Professor in Electrical Engineering at IIT Bombay. He received B.Tech. in electronics engineering from Banaras Hindu University in 1979, M.Tech. in electrical engineering from IIT Kanpur in 1981, and Ph.D. in electrical & biomedical engineering from the University of Toronto (Canada) in 1987. In 1987, he joined the University of Wyoming (USA) as an assistant professor and later joined IIT Bombay in 1989. His research interests include speech & signal processing; biomedical signal processing & instrumentation; electronic instrumentation & embedded system design. The focus of his R&D efforts has been in the areas of impedance cardiography and aids for persons with speech and hearing impairment.