1. Developing rating instruments for the assessment of Academic Writing and Speaking at Austrian University English Departments
Language Testing in Austria: Towards a Research Agenda
Dr Helen Heaney
Dr Armin Berger
Department of English
University of Vienna

2. Chapter outline Introduction Background to the project
The design process
Establishing design principles
Componential analysis
Operational analysis and benchmarking
The products of the process
Conclusion
Developing rating instruments for the assessment of Academic Writing and Speaking at Austrian University English Departments Klagenfurt

3. Descriptor for Lexico-Grammatical Resources and Fluency C1.2
Has a good command of a very broad [C2: VOCABULARY RANGE: 112] range of language to express him/herself clearly [C1: GENERAL LINGUISTIC RANGE: 110] and appropriately
Generally maintains a high degree of lexical and grammatical control, as appropriate for the genre, where errors are rare and mostly insignificant [C1: GRAMMATICAL ACCURACY: 114]
Can express him/herself fluently and [...] almost effortlessly [C1: SPOKEN FLUENCY: 129] (i.e. rarely pauses for reasons of grammar or word choice, reducing pauses by using appropriate fillers; reformulations occur mainly for reasons of expressing ideas fully)
Black: original CEFR descriptor
Red: initial change to CEFR descriptor
Orange: CEFR descriptor added at a later point
Turquoise: new descriptor element
Developing rating instruments for the assessment of Academic Writing and Speaking at Austrian University English Departments Klagenfurt

4. An expanded model for rating scale development
(based on Galaczi et al. 2011)
Chronological dimension:
establishing design principles
componential analysis
operational analysis;
Functional dimension:
consultation
develompent
research
Methodological dimension:
theory-informed activities (boxes shaded in grey)
qualitative procedures (dotted background)
quantitative methods (checked pattern)

5. The way forward Revision and/or fine-tuning of descriptors for writing
More recent research, e.g. Core Inventory for General English (North, Ortega & Sheehan 2011), English Profile
Descriptive scheme of CEFR (CoE 2001), CEFR Companion Volume (CoE 2017)
Systematic performance analysis
Corpus analysis
Post hoc validation of writing scale
Investigating the use of the scales in different contexts
Rater training
Developing rating instruments for the assessment of Academic Writing and Speaking at Austrian University English Departments Klagenfurt

6. Bibliography
Berger, Armin (2015). Validating analytic rating scales: A multi-method approach to scaling descriptors for academic speaking. Frankfurt am Main: Peter Lang. CoE/Council of Europe (2001). Common European Framework of Reference for Languages, ( ) CoE/Council of Europe (2017). Common European Framework of Reference for Languages: Learning, Teaching, Assessment. Companion Volume with New Descriptors. (Provisional Edition) / ( ) English Profile (n.d.), ( ) Galaczi, Evelina; ffrench, Angela; Hubbard, Chris; Green, Anthony (2011). “Developing assessment scales for large-scale speaking tests: A multiple-method approach”. Assessment in Education: Principles, Policy & Practice 18(3), Heaney, Helen (2013). “Fleshing out CEFR descriptors at C1 and above for the assessment of academic writing in departments of English at Austrian universities”. Dutch Journal of Applied Linguistics 2(1), LTC/Language Testing Center (2009). ELTT Mission Statement, university-eltt/. ( ) North, Brian, Ortega, Angeles; Sheehan, Susan (2011). A Core Inventory for General English. London: British Council/EAQUALS, ( )
Developing rating instruments for the assessment of Academic Writing and Speaking at Austrian University English Departments Klagenfurt