1. Rating scale validation for the assessment of spoken English at tertiary level
Language Testing in Austria: Towards a Research Agenda
Dr Armin Berger
Department of English
University of Vienna

2. Chapter outline Introduction
The role of rating scales in performance assessment
Rating scales as operationalisations of test constructs
Controversy about rating scales
Rating scale validation
Research questions and design
Data analysis and results
Discussion and conclusions
Rating scale validation for the assessment of spoken English at tertiary level Klagenfurt

3. Research questions
To what extent do the descriptors of the ELTT speaking scales (presentation and interaction) define a continuum of increasing speaking proficiency?
Does the validation methodology have an effect on the hierarchy of descriptor units?
Which rating scale descriptors are the most effective ones?
Do the revised rating scales function adequately in terms of (a) the discrimination between candidates, (b) rater spread and consistency, and (c) rating scale properties?
What are raters’ perceptions of the revised speaking scales?
Rating scale validation for the assessment of spoken English at tertiary level Klagenfurt

4. Research design Descriptor sorting Descriptor calibration
Descriptor-performance matching
Operational trial
Participants
21 university teachers
16 students (trial group)
78 students
8 university teachers
55 students
8 university teachers
Instruments
Sorting task
2 task prompts
153 video performances
Rating sheets
Rater guidelines
55 video performances
Revised rating scales
Semi-structured interview
Procedures
Descriptor sorting
Descriptor scaling
Task development
Mock exams
Performance matching
Performance rating
Retrospective group interviews
Analyses
Correlations
Multi-faceted Rasch
Content analysis

5. Conclusions
The revised scales function adequately in operational settings.
Evidence-based scale validation is fundamental.
The project has shown the benefits of the symbiosis between intuitive, qualitative and quantitative scale development methods (cf. Galaczi et al. 2011).
Validation methods are complementary rather than interchangeable.
The utility of scale validation methods should be evaluated in terms of their combined effect.
Rating scale validation for the assessment of spoken English at tertiary level Klagenfurt

6. Bibliography
Berger, Armin Validating analytic rating scales: A multi-method approach to scaling descriptors for academic speaking. Frankfurt am Main: Peter Lang. CoE/Council of Europe Common European Framework of Reference for Languages: Learning, Teaching, Assessment. Companion Volume with New Descriptors. (Provisional Edition) / ( ) Galaczi, Evelina; ffrench, Angela; Hubbard, Chris; Green, Anthony “Developing assessment scales for large-scale speaking tests: A multiple-method approach”. Assessment in Education: Principles, Policy & Practice 18(3),
Rating scale validation for the assessment of spoken English at tertiary level Klagenfurt