Developing rating instruments for the assessment of Academic Writing and Speaking at Austrian University English Departments Language Testing in Austria: Towards a Research Agenda Dr Helen Heaney Dr Armin Berger Department of English University of Vienna

Chapter outline Introduction Background to the project The design process Establishing design principles Componential analysis Operational analysis and benchmarking The products of the process Conclusion Developing rating instruments for the assessment of Academic Writing and Speaking at Austrian University English Departments Klagenfurt 19.10.2018

Descriptor for Lexico-Grammatical Resources and Fluency C1.2 Has a good command of a very broad [C2: VOCABULARY RANGE: 112] range of language to express him/herself clearly [C1: GENERAL LINGUISTIC RANGE: 110] and appropriately Generally maintains a high degree of lexical and grammatical control, as appropriate for the genre, where errors are rare and mostly insignificant [C1: GRAMMATICAL ACCURACY: 114] Can express him/herself fluently and [...] almost effortlessly [C1: SPOKEN FLUENCY: 129] (i.e. rarely pauses for reasons of grammar or word choice, reducing pauses by using appropriate fillers; reformulations occur mainly for reasons of expressing ideas fully) Black: original CEFR descriptor Red: initial change to CEFR descriptor Orange: CEFR descriptor added at a later point Turquoise: new descriptor element Developing rating instruments for the assessment of Academic Writing and Speaking at Austrian University English Departments Klagenfurt 19.10.2018

An expanded model for rating scale development (based on Galaczi et al. 2011) Chronological dimension: establishing design principles componential analysis operational analysis; Functional dimension: consultation develompent research Methodological dimension: theory-informed activities (boxes shaded in grey) qualitative procedures (dotted background) quantitative methods (checked pattern)

The way forward Revision and/or fine-tuning of descriptors for writing More recent research, e.g. Core Inventory for General English (North, Ortega & Sheehan 2011), English Profile Descriptive scheme of CEFR (CoE 2001), CEFR Companion Volume (CoE 2017) Systematic performance analysis Corpus analysis Post hoc validation of writing scale Investigating the use of the scales in different contexts Rater training Developing rating instruments for the assessment of Academic Writing and Speaking at Austrian University English Departments Klagenfurt 19.10.2018

Bibliography Berger, Armin (2015). Validating analytic rating scales: A multi-method approach to scaling descriptors for academic speaking. Frankfurt am Main: Peter Lang. CoE/Council of Europe (2001). Common European Framework of Reference for Languages, https://rm.coe.int/1680459f97. (02.10.2018) CoE/Council of Europe (2017). Common European Framework of Reference for Languages: Learning, Teaching, Assessment. Companion Volume with New Descriptors. (Provisional Edition). https://rm.coe.int/cefr-companion-volume-with-new-descriptors- 2018/1680787989. (02.10.2018) English Profile (n.d.), http://www.englishprofile.org. (02.10.2018) Galaczi, Evelina; ffrench, Angela; Hubbard, Chris; Green, Anthony (2011). “Developing assessment scales for large-scale speaking tests: A multiple-method approach”. Assessment in Education: Principles, Policy & Practice 18(3), 217-237. Heaney, Helen (2013). “Fleshing out CEFR descriptors at C1 and above for the assessment of academic writing in departments of English at Austrian universities”. Dutch Journal of Applied Linguistics 2(1), 43-56. LTC/Language Testing Center (2009). ELTT Mission Statement, https://www.aau.at/en/language-testing-centre/archive/austrian- university-eltt/. (02.10.2018) North, Brian, Ortega, Angeles; Sheehan, Susan (2011). A Core Inventory for General English. London: British Council/EAQUALS, http://englishagenda.britishcouncil.org/sites/default/files/attachments/books-british-council-eaquals-core-inventory.pdf. (02.10.2018) Developing rating instruments for the assessment of Academic Writing and Speaking at Austrian University English Departments Klagenfurt 19.10.2018