1. Acousteen Herman J.M. Steeneken Subjective Intelligibility Assessment Dr. Herman J.M. Steeneken

2. Acousteen Herman J.M. Steeneken 2 Signal-to-Noise ratio !!!

3. Acousteen Herman J.M. Steeneken 3 Research Questions Intelligibility versus Quality assessment Evaluation of a system or application Ranking of the performance of a number of systems Diagnostic assessment Prediction of system performance during design

4. Acousteen Herman J.M. Steeneken 4 Assessment Methods Subjective assessment with subjects (speakers and listeners): representative, limited reproduction, non diagnostic, laborious Objective assessment based on physical properties (measurements): reproducible, diagnostic, fast Prediction of system performance: design tool

5. Acousteen Herman J.M. Steeneken 5 SUBJECTIVE INTELLIGIBILITY

6. Acousteen Herman J.M. Steeneken 6 Subjective Intelligibility methods Phoneme level (nonsense words, rhyme words, consonants, vowels) Word level (meaningful words, nonsense words, phonetically balanced PB, equally balanced Eqb) Sentence level (Mean Opinion Score MOS, Speech Reception Threshold SRT)

7. Acousteen Herman J.M. Steeneken 7 Methodology I Response categories: Open response (e.g., nonsense words) Closed response (Rhyme tests, e.g., MRT, DRT) Scaling (MOS, five point scale: excellent - bad) Ranking (e.g., pair-wise comparison)

8. Acousteen Herman J.M. Steeneken 8 Methodology II Test design: Words embedded in carrier phrase Reference conditions (e.g. MNRU, …) Speakers (gender, number, non-native, …) Listeners ( number of speaker-listener pairs) Learning effects

9. Acousteen Herman J.M. Steeneken 9 Listening test with four subjects

10. Acousteen Herman J.M. Steeneken 10

11. Acousteen Herman J.M. Steeneken 11 Embedded CVC words: versta des over en nu fijs uit het woord zek einde noteer lal punt “Semi random”combination of: 17 initial consonants 15 vowels 11 final consonants

12. Acousteen Herman J.M. Steeneken 12 Methodology III Scoring, data analysis: Phone-word scores Confusion matrices Effective gain (e.g. effective SNR) Statistics (Anova, scaling, multiple regression,...)

13. Acousteen Herman J.M. Steeneken 13 Relation Consonants-Vowels

14. Acousteen Herman J.M. Steeneken 14 How to calculate average word scores Subject responses may require to use the median

15. Acousteen Herman J.M. Steeneken 15 Example relation MOS-CVC

16. Acousteen Herman J.M. Steeneken 16 Relation between methods and Qualification

17. Acousteen Herman J.M. Steeneken 17 Test-retest variability Cronbach α based on split of speaker- listener pairs

18. Acousteen Herman J.M. Steeneken 18 Common Intelligibility scale (IEC60849) After Barnett and Knight 1994 CIS not linear with SNR = STI  = 100 - ALcons x = AI  = PB words (256 words)  = Short Sentences = PB words (1000 words)  = 1000 syllables Barnett and Knight (1995)

19. Acousteen Herman J.M. Steeneken 19 CVC scores (%) of realistic conditions malefemale Wide band90.389.3 Telephone band89.585.3 White noise SNR 0 dB58.044.1 Speech noise SNR +3 dB71.360.7 Speech noise SNR -3 dB 43.040.6

20. Acousteen Herman J.M. Steeneken 20 Example of consonant confusions pbfvmnRw p1068621244002 b112100200117050 f4419151930000 v6433773900243 m1500106811316 n0000111108102 R12020211613 w63113307251065

21. Acousteen Herman J.M. Steeneken 21 Two dimensional display of confusions

22. Acousteen Herman J.M. Steeneken 22 Introduction of phoneme specific frequency weighting Four groups of phonemes (SAMPA notation: Fricatives (f, s, v, z) Plosives (b, d, x, p, t, k) Vowel-like consonants (m, n, l, R, j, w, …) Vowels (aa, a, ee, e, o, oo, u, uu, au, …)

23. Acousteen Herman J.M. Steeneken 23 Phoneme group specific spectra

24. Acousteen Herman J.M. Steeneken 24 Phoneme group specific spectra

25. Acousteen Herman J.M. Steeneken 25 Frequency weighting (fricatives)

26. Acousteen Herman J.M. Steeneken 26 Frequency weighting (plosives)

27. Acousteen Herman J.M. Steeneken 27 Frequency weighting (vowel-like cons)

28. Acousteen Herman J.M. Steeneken 28 Frequency weightings (vowels)

29. Acousteen Herman J.M. Steeneken 29 Frequency weightings (CVC words)

30. Acousteen Herman J.M. Steeneken 30 Prediction of (CVC) word score by a weighted combination of phoneme group probabilities (DUTCH) C i = 0.294 fric + 0.294 plo + 0.412 Cvo V = V (no weighting) C f = 0.273 fric + 0.273 plo + 0.454 Cvo CVC score = C i * V * C f * 100 %

31. Acousteen Herman J.M. Steeneken 31 CVC-word prediction (male) S.d.= 4.11% Male speech

32. Acousteen Herman J.M. Steeneken 32 CVC-word prediction (female) S.d. = 3.63% Female speech

33. Acousteen Herman J.M. Steeneken 33 ISO: Ergonomics – Assessment of speech communication (ISO 9921 DIS)

34. Acousteen Herman J.M. Steeneken 34 Qualification table