1. Language proficiency evaluation: Raters
Henry Emery
PRICESG Linguistic Sub-Group

2. Session structure Define terms Operational and linguistic raters
The need for adequate training
Rater competencies
A mechanism for international standardisation of language proficiency assessment:
The Association of Language Raters

3. An examiner: delivers a direct language test
is trained in a specific test format
elicits speech
interacts with the candidate/candidates (an interlocutor)
may examine and rate simultaneously

4. A rater: assesses language proficiency
applies the Rating Scale to speech
requires familiarity with the test format
may examine and rate simultaneously
may rate ‘live’ speech
may rate a pre-recorded sample

5. Who is an operational expert?
What is the role of an operational expert?
“to assess operational integrity and technical accuracy” (ICAO doc. 9835)
When should an operational be employed?
“test prompts should not be intended to evaluate specific technical knowledge concerning operations” (
In a test of plain English is there a role for an operational rater?
Operational experts: tests which aim to assess both phraseology and plain English proficiency

6. Operational raters subjective impression ‘pass’ or ‘no pass’
operational integrity
technical accuracy
overall effectiveness
Holistic Descriptors ?
Who has the final say, the linguistic or the operational expert?

7. Linguistic raters more complex judgement
judge components of performance separately
less subjectivity
justify assessment
identify specific aspects of performance
analytic Rating Scale

8. The complexities of the ICAO Rating Scale
Six key features

9. Feature 1
The six operational levels of the rating scale are not of equal intervals
Levels three and four are particularly broad

10. Level descriptors for Structure
Errors in basic grammatical structures rarely interfere with meaning 4
Errors in in basic grammatical structures frequently interfere with meaning 3
Limited control of a few simple memorized grammatical structures and sentence patterns 2 1
Performs below the elementary level

11. Feature 2
A candidate is unlikely to fully ‘fit’ a descriptor at any level
A candidate’s speech may demonstrate features of different levels at the same time

12. Level descriptors for Interactions6
Interacts with ease in nearly all situations. Is sensitive to verbal and non-verbal cues, and responds to them appropriately
Responses are immediate, appropriate, and informative. Manages the speaker/listener relationship effectively
Responses are usually immediate, appropriate, and informative. Initiates and maintains exchanges even when dealing with an unexpected turn of events. Deals adequately with apparent misunderstandings by checking, confirming, or clarifying 5 4

13. Feature 3
A candidate with a ‘flat’ profile across the six profiles in the Rating Scale will have a ‘jagged’ proficiency

14. Extracts from level 4 descriptors
Vocabulary … sufficient to communicate effectively
...occasional loss of fluency... does not prevent effective communication
Errors ... (in basic structures) … rarely interfere with meaning
Pronunciation, stress, rhythm, and intonation … sometimes interfere with ease of understanding

15. Feature 4 Pronunciation is notoriously difficult to assess
What does good pronunciation of a second language mean?
Standardise comprehensibility?
“Can we be certain that two or more ... speakers will find the utterance of a foreign speaker equally comprehensible…?”
(Harris 1977:81)

16. Feature 5 Degree of ‘overlap’ between profiles
Linguistic behaviours described in one profile appear in other profiles
Lack of proficiency in one profile will affect performance in another

17. Feature 6
Rating scale includes both ‘positive’ and ‘negative’ descriptors
Positive descriptor:
Can often paraphrase successfully...
Negative descriptor:
...may not vary speech flow...

18. Level 6 Level 5 Level 4 Level 3 Level 2 Level 1 Positive descriptors
Increase in positive descriptions moving from Levels 1-6
Decrease in negative descriptions moving from Levels 1-6
Negative descriptors

19. Reliable and accurate rating requires thorough training in the application of the ICAO Rating Scale
It is not enough to say “ I can apply the Scale”. Raters must demonstrate this ability

20. Linguistic Rater competencies:
Qualified and experienced teacher of English to speakers of other languages?
Experience in language assessment?
Experience in teaching English for aviation?
Native speaker/certificate of language proficiency?
Background in aviation operations?
Familiar with R/T phraseology?

21. Standardisation: Making level 4 level 4
Intra-rater reliability
Inter-rater reliability:
a. same test format
b: different tests
c: international standardisation
Standardisation of rating procedure

22. ICAO Document 9835, Section 6.7.4
Raters must
understand the criteria and the context in which the criteria occur
agree upon the standardised procedures for the implementation of the criteria
inspire trust and gain the confidence of the community and the candidates

23. “A body of well-informed and experienced raters must be formed”
ICAO Document 9835, Section 6.7.4
“A body of well-informed and experienced raters must be formed”

24. ALR
The Association of
Language Raters

25. Speech samples are gathered from organisations around the world
Regional ALR representatives (aviation SMEs and linguists) convene to:
rate the speech samples and write rationales for the ratings
organise the rated speech samples into ‘assessment packs’
produce an association ‘Code of Practice’
formalise guidance material for language raters

26. Language Rater Accreditation Service
A prospective language rater applies to ALR for Rater Accreditation
The Association sends an assessment pack to the rater
The rater rates speech samples contained in the assessment pack
If the ratings agree with the ALR standard, an official Certificate of Accreditation is issued

27. o Governmental and commercial aviation organisations
Funding:
o Governmental and commercial aviation organisations
o Accreditation Service Fees
ALTE
ICAO
Executive committee
Association of Language Raters (ALR)
ILTA
Cyclical presidency
STANDARDISATION
Regional Representatives
(Language Raters and Operations Personnel)
Regional Representatives
(Language Raters and Operations Personnel) F E D B A C K
Team Leaders (In-country)
Team Leaders (In-country)
Local Rater
Local Rater
Local Rater
Local Rater

28. Rater Accreditation Certificates valid for two years
On expiry, the rater must re-apply for ALR accreditation
ALR-accredited raters for assessments for licensing purposes

29. Regional ALR representatives will convene every two years to
Rate fresh speech samples for ‘assessment packs’ for the new accreditation period
Review existing policy for rater accreditation
Review feedback gathered by Team Leaders
Produce a published bi-annual report for the international aviation community

30. Long-term activity
Language Test Accreditation
Inspection and auditing of language testing organisations
Training programs
Regional seminars

31. Comments and questions: henry@maycoll.co.uk
ALR
The Association of
Language Raters
Comments and questions: