1. Catia Cucchiarini Quantitative assessment of second language learners’ fluency in read and spontaneous speech Radboud University Nijmegen

2. Context Research on automatic assessment of oral proficiency in Dutch as a second language

3. Fluency Important construct in evaluation of second language proficiency Also relevant for pathological speech

4. Fluency Frequently applied notion, but not clearly defined Various interpretations Overall language proficiency Oral command of a language Temporal aspect of oral proficiency

5. Two experiments Exp1: read speech Exp2: spontaneous speech Human fluency judgements related to Objective temporal measures (CSR)

6. Aim of these experiments To explore the relationship between objective properties of speech and perceived fluency in read and spontaneous speech, with a view to determining whether such quantitative measures can be used to develop objective fluency tests.

7. Method: speakers Exp1: 60 non-native speakers 3 proficiency levels beginner (PL1) intermediate (PL2) advanced (PL3) different mother tongues different gender

8. Method: speakers Exp2: 60 non-native speakers 2 proficiency levels beginner level (BL) intermediate level (IL) different mother tongues different gender

9. Method: speech material Exp1: 2 sets of 5 phonetically rich sentences read over the telephone

10. Method: speech material Exp2: existing test of Dutch as a second language (DSL) è Profieltoets 8 items from BL version short tasks, 15 s to answer candidates can answer immediately 8 items from IL version long tasks, 30 s to answer candidates have to reflect to provide motivations

11. Method: raters Exp1: 3 phoneticians (PH) 3 speech therapists (ST1) 3 speech therapists (ST2) Exp2: 5 DSL teachers for BL (RBL) 5 DSL teachers for IL (RIL)

12. Method: automatic scoring Speech orthographically transcribed CSR: 38 monophones + lexicon Viterbi alignment of speech signals and orthographic transcriptions Segmentation at phone level

13. Method: some definitions silent pause: a stretch of silence of no less than 200 ms dur1 = duration speech without pauses (s) dur2 = duration speech with pauses (s)

14. Method: objective measures Primary variables art = # phones / dur1 ros = # phones / dur2 ptr = 100% * dur1 / dur2 mlr = mean # phones between 2 pauses mlp = mean length silent pauses dsp = tot. dur. sil. pauses / (dur2 / 60) # sp = # sil. pauses / (dur2 / 60)

15. Method: objective measures Secondary variables # fp = # filled pauses / (dur2 / 60) # disf =# disfluencies / (dur2 / 60)

16. Method: fluency ratings Sentences scored on fluency on the basis of a ten-point scale Raters received no special training

17. Method: rating procedure Exp1: each group of raters judged speakers of different proficiency levels Exp2: each group of raters judged speakers of the same proficiency level

18. Results: reliability

19. Results: raw fluency ratings

20. Results: objective measures

21. Results: disfluencies Repetitions: exact repetitions of words Repairs: corrections Restarts: repetitions initial parts of words

22. Results: disfluencies

23. Results: correlations

25. Discussion Reliable fluency scoring is possible Fluency scores related to task performed Role objective variables in rs and ss similarities: weak relation sec. var. / fluency differences: varying roles prim. var.

26. Discussion Read speech: strong relation: art, ros, ptr, #sp, dsp, mlr weaker relation: mlp  for perceived fluency pause freq. more important than pause length  two factors important fluency rs: articulation rate pause frequency

27. Discussion Spontaneous speech : strong relation: ros, ptr, #sp, dsp, mlr weaker relation: art, mlp  possibly higher freq. pauses effaces importance art  fluency in ss particularly related to var. that contain info on pause freq.

28. Conclusions Reliable fluency scoring by human raters is possible Objective fluency scoring is possible Fluency scores vary with speech type Fluency scores vary with task performed

29. Conclusions Read speech: fluency scores strongly related to art and pause frequency Spontaneous speech: fluency scores strongly related to pause frequency and distribution Expert fluency ratings can be predicted more accurately on the basis of objective measures in rs than in ss

30. Conclusions Temporal measures of fluency may be used to develop objective fluency tests Selection of variables to be employed should be dependent on material investigated and task performed