1. ICAO-Eurocontrol European Seminar on Language Proficiency for Aeronautical Communications, Eurocontrol Brussels October 27th-28th 2005 1 Standards for selection and development of language tests Dr Jeremy Mell Head of Language Studies ENAC, Toulouse, France jeremy.mell@enac.fr

2. ICAO-Eurocontrol European Seminar on Language Proficiency for Aeronautical Communications, Eurocontrol Brussels October 27th-28th 2005 2 Options for assessing language proficiency for aviation informal observation s –« line checks », « inspections » –classroom assessments formal language tests –direct/semi-direct/indirect –simulated language use most direct integrative: performance samples are matched to rating scales –paper and pencil OR screen and mouse tasks most indirect discrete items/skills: favour numerical scores but « juggling with only one ball »….

3. ICAO-Eurocontrol European Seminar on Language Proficiency for Aeronautical Communications, Eurocontrol Brussels October 27th-28th 2005 3 Language tests Definition structured events or procedures to elicit performances as samples of test-taker ’s language skills in a standardised way enabling reliable inferences to be made concerning his/her level of competence and possibility of reproducing those skills at that level of competence consistently over time –adapted from Carrol (1968) and Douglas (2000)

4. ICAO-Eurocontrol European Seminar on Language Proficiency for Aeronautical Communications, Eurocontrol Brussels October 27th-28th 2005 4 Language testing A well-developed domain of intellectual activity: –academic body of research, cross-disciplinary links, international associations,... codes of ethics, codes of practice institutions of excellence, « chapels », « gurus », feuds –commercial international testing service providers general, academic, business language varieties Attempts to link different rating scales/test results

5. ICAO-Eurocontrol European Seminar on Language Proficiency for Aeronautical Communications, Eurocontrol Brussels October 27th-28th 2005 5 Language test qualities general qualities –validity –face validity –reliability –positive washback –practicality –security conflicting qualities developement procedures –needs analysis (language corpora, surveys,….) –design/item-writing team (operational, linguistic, testing expertise) –trialling (representative populations, controlled conditions, data retrieval) –statistical tools –test documentation –examiner training, accreditation

6. ICAO-Eurocontrol European Seminar on Language Proficiency for Aeronautical Communications, Eurocontrol Brussels October 27th-28th 2005 6 Types of language tests entry placement diagnostic progress/achievement proficiency

7. ICAO-Eurocontrol European Seminar on Language Proficiency for Aeronautical Communications, Eurocontrol Brussels October 27th-28th 2005 7 Entry/placement tests Purposes –ab initio training: recruitment form level groups –recurrent training: benchmarking a population form level groups Characteristics –items chosen to cover a broad range of levels of general language –formats reflect previous education –scores tend to form a « bell curve » –stakes high (recruitment) mid (benchmarking)

8. ICAO-Eurocontrol European Seminar on Language Proficiency for Aeronautical Communications, Eurocontrol Brussels October 27th-28th 2005 8 Diagnostic tests Purposes –identify specific areas of skill/knowledge for improvement in subsequent training –individualisation of training programmes Characteristics –each item chosen to represent a single significant area of knowledge/skill –overall score less important than analysis of right/wrong responses - focus on errors –stakes: low

9. ICAO-Eurocontrol European Seminar on Language Proficiency for Aeronautical Communications, Eurocontrol Brussels October 27th-28th 2005 9 Progress/achievement tests Purposes –to measure effectiveness of a phase of learning –to allow access by learner to next phase of learning Characteristics –items chosen to closely reflect content and methodology of preceding training phase –scores will often be interpreted with regard to average or pre-established norm –may refer to intermediate levels (3,5; 4,5;…) –stakes: low to medium

10. ICAO-Eurocontrol European Seminar on Language Proficiency for Aeronautical Communications, Eurocontrol Brussels October 27th-28th 2005 10 Proficiency tests Purposes –to establish the competence of candidate to exercise language skills in operational conditions Characteristics –items chosen to resemble real-world tasks –overall scores are holistic: YES/NO –stakes: very high –a “blunt” instrument but must be highly robust: proven validity proven reliability security

11. ICAO-Eurocontrol European Seminar on Language Proficiency for Aeronautical Communications, Eurocontrol Brussels October 27th-28th 2005 11 Testing needs of aviation Ab initio populations –entry selection local education vs aviation training policies –progress/achievement intermediate training objectives continuation/curtailment –initial proficiency licensing Qualified populations –placement/diagnostic benchmarking populations individual training needs acceptance by takers –(progress/achievement) degree and speed –proficiency renewal recurrent

12. ICAO-Eurocontrol European Seminar on Language Proficiency for Aeronautical Communications, Eurocontrol Brussels October 27th-28th 2005 12 Test design specifications (ICAO) “As of 5 March 2008, aeroplane and helicopter pilots, air traffic controllers and aeronautical station operators shall demonstrate the ability to speak and understand the language used for radiotelephony communications to the level specified in the language proficiency requirements in the Appendix.” (Annex 1, 1.2.9.4) FORMAL EVALUATION “An individual must demonstrate proficiency at level 4 in all categories in order to receive a level 4 score.” (Doc 9835, Manual on the Implementation of ICAO LPRs, 2.8.4) PROFILE EVALUATION

13. ICAO-Eurocontrol European Seminar on Language Proficiency for Aeronautical Communications, Eurocontrol Brussels October 27th-28th 2005 13 Institutional context possibility of independent international oversight to ensure: –validity of testing procedures used –comparability of testing procedures and outcomes testing outcomes are linked with personnel licensing –testing may not be in the sole hands of the “language” community –partnerships with CAA, service provider, airline,... system cannot tolerate a high failure rate –waste of expensive training investment –staffing levels must ensure continuity of service prior knowledge of test procedures for positive “washback” on training –trainee motivation and attitude, reassurance of candidates –development of appropriate training systems –BUT unsuitability of “cramming”, test item banks,...

14. ICAO-Eurocontrol European Seminar on Language Proficiency for Aeronautical Communications, Eurocontrol Brussels October 27th-28th 2005 14 A common test for ATC and pilots? pilots and controllers are partners in R/T communication but they are set apart by: –different interactive roles –complementary passive/active competencies –opportunity to use other job-related language uses to extend speech sample: controller: telephone co-ordinations, report to supervisor,... pilot: pre-flight, intra-cockpit, cabin announcements, ground staff, …. a possible solution? –common core tests of knowledge –job-specific components for testing competence

15. ICAO-Eurocontrol European Seminar on Language Proficiency for Aeronautical Communications, Eurocontrol Brussels October 27th-28th 2005 15 Need for well-planned testing services time/space constraints –dispersed locations (recurrent) –irregular schedules and limited availability of test-takers (recurrent) –availability of technologies –test/re-test test components –single event, integrative? –different item designs to test separate skills? standardisation of results –extensive trialling –comparable conditions of test administration –examiner training/auditing based on speech samples interlocutors raters –paired/multiple rating test security –secure storage and transfer of test materials –multiple parallel versions

16. ICAO-Eurocontrol European Seminar on Language Proficiency for Aeronautical Communications, Eurocontrol Brussels October 27th-28th 2005 16 Appropriate technologies voice-only –telephone, 2-way radio, language laboratories, training simulators,... input delivery –analogue players, computer screen (multimedia) performance storage and access –analogue recorders, computer sound files (rapid access) speech recognition –standardise perception of intelligibility, save rating costs –BUT needs to filter all possible speech variations « interactions » need to be human-human –initiate and maintain exchanges –deal with misunderstandings

17. ICAO-Eurocontrol European Seminar on Language Proficiency for Aeronautical Communications, Eurocontrol Brussels October 27th-28th 2005 17 Rater/interlocutor qualifications 3 basic qualifications –language level: level 6 (language raters) level 5/6 (interlocutors, SME raters) –aviation background + familiarity with ICAO LPRs –principles of language proficiency and language testing suitable candidates –aviation personnel –language trainers L1 background –native speakers(NS)/non-native speakers (NNS) –risk of familiarity with a given form of NNS spoken English

18. ICAO-Eurocontrol European Seminar on Language Proficiency for Aeronautical Communications, Eurocontrol Brussels October 27th-28th 2005 18 Assessment of level 6 NOT a safety issue, but raises concerns about: resources ( no re-test) NS/NNS distinctions Test-takers may be: mother-tongue non-mother tongue –intranational use –non-intranational use Candidacy on basis of biographical criteria citizenship, educational background, residency,... Protocols for assessment –admissible evidence –assessor qualification –appropriate documentation Subsequent checking and oversight must address: –adherence to standard ICAO phraseologies –ICAO LPRs for intelligibility appropriateness

19. ICAO-Eurocontrol European Seminar on Language Proficiency for Aeronautical Communications, Eurocontrol Brussels October 27th-28th 2005 19 Points of view within the aviation English profession « We need to find workable and practical solutions to the training and testing challenges of the next few years. Those solutions may not be able to wait for the « perfect » aviation English proficiency test.» ICAEA_WORLD discussion group, 29th May 2004. “I have some considerable concerns about… an individualistic approach being adopted by airlines / ATC administrations / civil aviation authorities around the world. … my view is that it is essential that testing be subject to ‘external standards’ and be ‘standardised and universal to some degree’. ICAEA_WORLD discussion group, 14th April 2004

20. ICAO-Eurocontrol European Seminar on Language Proficiency for Aeronautical Communications, Eurocontrol Brussels October 27th-28th 2005 20 Informal evaluations Within the initial or continuing training process, language trainers can identify 3 categories of trainee: -well-below level 4 -approximately at level 4 (confirm by formal evaluation) -comfortably above above level 4 …thus enabling informed decisions to be made on possible access to professional functions further training required benchmarking of populations

21. ICAO-Eurocontrol European Seminar on Language Proficiency for Aeronautical Communications, Eurocontrol Brussels October 27th-28th 2005 21 Equivalences? ICAO level 4 is specific to proficiency requirements of aeronautical communications There are no validated equivalences with bands on other rating scales or scores in general purpose tests..... however scales/scores relating to listening and speaking could be used to identify « optimal points of transition » to ICAO LPRs for selection purposes, eg : –IELTS test: bands 5/6 –Common European Framework: levels B2/C1 –Cambridge Testing Suite: FCE (high pass)/CAE (low pass)

22. ICAO-Eurocontrol European Seminar on Language Proficiency for Aeronautical Communications, Eurocontrol Brussels October 27th-28th 2005 22 Formal tests for ICAO level 4 Placement (entry) –must relate to ALL 6 skills in ICAO scale –must measure distances above and below level 4 –may have diagnostic capability –tasks/topics/input materials focus on general or job-related skills –development requires some statistical validation Proficiency (exit) –must relate to ALL 6 skills in ICAO scale –must characterise test-takers as BELOW or AT levels 4, 5 or 6 –tasks/topics/input material must reflect radiotelephony language competencies (including standardised phraseology) –development requires strong statistical validation (high stakes)

23. ICAO-Eurocontrol European Seminar on Language Proficiency for Aeronautical Communications, Eurocontrol Brussels October 27th-28th 2005 23 Detailed specifications (proficiency) provide a representative range of intelligible international accents as input for comprehension; provide a professionally relevant format for candidates to display comprehension; elicit an adequate continuous speech sample to test fluency/pronunciation; provide a voice-only setting for “diadic” (2-person) interactions; provide examples of routine and unexpected events in a work-related context; allow the candidate to use basic grammatical structures creatively; allow the candidate to demonstrate ability to paraphrase; allow the candidate to change between rehearsed/formulaic speech and spontaneous interaction; simulate unexpected events to create opportunities for misunderstanding. –EUROCONTROL/ENAC preliminary feasibility study 2004

24. ICAO-Eurocontrol European Seminar on Language Proficiency for Aeronautical Communications, Eurocontrol Brussels October 27th-28th 2005 24 Some existing specific tests Controllers –ab initio entry: EPT/FEAST(Eurocontrol) progress: APRO (ENAC, F) proficiency: PELA (Eurocontrol) –recurrent placement: TNP (DGAC, F) proficiency: ELPAC (Eurocontrol) – under development for 2007 Pilots –ab initio proficiency: FCL 1.200 (DGAC, F) Other tests –TOEFA (Peru), –TELPA (IAES, Korea) –TELAP (CA Flight University,China) –RELTA (RMIT, Australia) –TEA (Mayflower, UK) –….

25. ICAO-Eurocontrol European Seminar on Language Proficiency for Aeronautical Communications, Eurocontrol Brussels October 27th-28th 2005 25 TNP (DGAC, France) developed by national language and subject-matter experts (controllers and ATC management) as a placement/diagnostic tool 3 papers cover professional language content: a written multiple choice test of language knowledge a tape-mediated written test of listening comprehension (RT messages) an individual oral interview using paper-mediated graphic and text input expresses results in terms of the ICAO level chart used since 1999 with qualified air traffic controllers: benchmarking to establish national training needs formulating periodic individualised language development and maintenance programmes in order to meet national qualification renewal requirements

26. ICAO-Eurocontrol European Seminar on Language Proficiency for Aeronautical Communications, Eurocontrol Brussels October 27th-28th 2005 26 PELA (Eurocontrol) developed as a proficiency testing tool as part of European ATM harmonisation; 3 papers cover professional language content: a tape-mediated written test of listening comprehension (RT and telephone co- ordination exchanges) a tape-mediated test of interaction (short responses) in routine ATC situations a live interaction with trained interlocutor simulating a non-standard ATC situation and report to supervisor used in Eurocontrol/ECAC states with ab initio controller trainees at end of basic training; recently revised by language and subject-matter experts to conform with ICAO language proficiency requirements computer-based versions of Papers 1 and 2 currently under development

27. ICAO-Eurocontrol European Seminar on Language Proficiency for Aeronautical Communications, Eurocontrol Brussels October 27th-28th 2005 27 Further reading Bachman, L. and Palmer, A. 1996. Language Testing in Practice. Oxford: Oxford University Press. Carroll, J. B. (1968). The psychology of language testing. In A. Davies (Ed.), Language Testing Symposium: A Psycholinguistic Perspective (pp. 46-69). Oxford: Oxford University Press. Douglas, D. 2000. Assessing Languages for Specific Purposes. Cambridge: Cambridge University Press.