ICAO Aviation English Language Test Endorsement Jean-Marc Guillemette Project Manager ICAO Aviation English Language Test Endorsement

Outline Purpose: Topics: Review the work underway to establish an ICAO aviation English test endorsement process Topics: Project purpose and structure Test endorsement process Current activities Next steps

Purpose of Test Endorsement Ensure that the design, development and administration of aviation English language tests reflect, the principles and practices of effective language test construction and administration the constraints and framework established by the ICAO language provisions Support the application of globally harmonized language testing criteria See the document “About ICAO’s Aviation English Language Test Endorsement Service” for additional detail on the process and its implementation. The first two questions in that document provide additional detail for this slide.

Participation Initial workgroup with representatives from ILTA, IFALPA, IFATCA, ICAEA developed the endorsement process. Expanded the group in January to increase test service provider representation: ILTA, IFALPA, IFATCA, ICAEA Representatives from China, Latin America, Eastern Europe, Eurocontrol Participatory approach based on extensive use of communication technology. The diagram in this slide illustrates using a cyclical approach to work through discussions and decision making. While the diagram implies completing steps only once, the workgroup consistently followed an iterative cycle of discussion, review and decision making throughout the project. This was therefore a participatory approach based on equal contributions and collaboration. The workgroup worked mostly at a distance using various computer-based tools. For example, a webinar platform, document sharing services and VOIP communication (e.g. Skype). Thanks to everyone’s flexibility and willingness to work through some technological problems, the workgroup was able to complete the project nearly entirely at distance.

Project Overview Activities Sept-Dec 2010 Jan-Apr 2011 May-Jun 2011 Post start-up Develop a test endorsement process Develop test evaluation criteria Pilot the endorsement process Implement / manage process This slide is intended as broad project timeline to provide the audience with a sense of how the project evolved over time. The project currently remains essentially on time, with some slippage conducting the pilot. More about the pilot on slides 10 and 11 of this presentation. The step to “Implement / manage process” consists of setting up a more permanent test endorsement service within ICAO to ensure ongoing improvement of what we now have. ICAO intends to continue working closely with professional associations and other stakeholders to keep the endorsement process as practical, effective and efficient as possible.

Endorsement: Main Parts Evaluate original tests (created by TSP) - test construction / ICAO constraints & framework (formative process) Test Endorsement Review and confirm endorsed tests continue meeting set criteria (3 years) Test re-endorsement There are three main parts to the endorsement process: - to evaluate original tests from Test Service Providers (i.e. tests created by TSP submitting it for endorsement; - to review and re-endorsed tests every three years; and to provide up-to-date information on tests endorsed to States and members of the public. Test endorsement and re-endorsement remain team efforts in which language testing and operational experts will together determine whether or not a test can be endorsed. Working in cooperation with associations like IFALPA, ICAO will set up and manage a pool of test evaluators with appropriate expertise. The first part is the most demanding for all concerned, in terms of time and effort. Test Service Providers will be required to provide detailed information on test design, development and use to be considered for endorsement. Confirm test administration data – assist States with language proficiency oversight Test Administration

Endorsement: TSP Perspective This diagram shows what TSP will do to complete the endorsement process. Create an account for identification and security purposes Complete an online pre-assessment questionnaire to help confirm their test is ready for evaluation and endorsement. TSP complete the questionnaire online and receive feedback immediately after highlighting possible test problems. TSP can then decide to stop-and-fix or continue as desired. Once ready to continue, TSP pay a non-refundable endorsement fee. Once payment is confirmed, TSP receive additional instruction to prepare and submit more detailed test documentation to be reviewed by an evaluation team. TSP with an endorsed test will receive a certificate confirming ICAO has reviewed and confirmed the test is in good standing. Endorsed tests will be identified in the public area of the endorsement website.

Test Endorsement Website ICAO is creating a dedicated website to act as the main communication channel between all those involved in the endorsement process. The site will reflect the parts of the process previously identified. The website is therefore essentially divided in 4 parts: For the system administrator (system caretaker) For TSP: with the tools and information needed to submit tests for endorsement For evaluators to learn about and become test evaluators, and track evaluation assignments To display a list of endorsed tests to the public. Parts of the website for the administrator, TSP and test evaluators will be secure and require login. The public area will be open to all.

Test Endorsement Website This image shows the home page of the test endorsement website. As indicated, it is primarily intended as a communication channel between those involved in the process and is not intended for broader public use. It currently remains a simple tool to be refined over time as ICAO learns from the experience of offering the service.

Test Endorsement Pilot Just starting Includes 3 existing tests from different TSP Working with the project workgroup as evaluators Opportunity to confirm: The endorsement process works as intended and is feasible over time Test evaluation criteria is reliable Where and how to improve all aspects of the process How best to manage implementation over time

Next Steps Complete the pilot: review / improve the endorsement process, criteria and tools Set up more ‘permanent’ process Ensure full integration of test endorsement and other processes within ICAO Build a capacity to continually work with users to improve the system Operate regular test endorsement service 2nd bullet refers to setting up a more permanent test endorsement service as indicated in slide 5, above. 3rd bullet refers to other endorsement services being considered by ICAO. For example, to endorse training programs (technical, not language) or other types of tests. While there are currently no set plans to develop these services, there is growing interest in applying the experience of this project in others.

Thank you! Jean-Marc Guillemette Workplace Learning Architect Project Manager ICAO Aviation English Language Test Endorsement