1. ICAO Aviation English Language Test Endorsement
Jean-Marc Guillemette
Project Manager
ICAO Aviation English Language Test Endorsement

2. Outline Purpose: Topics:
Review the work underway to establish an ICAO aviation English test endorsement process
Topics:
Project purpose and structure
Test endorsement process
Current activities
Next steps

3. Purpose of Test Endorsement
Ensure that the design, development and administration of aviation English language tests reflect,
the principles and practices of effective language test construction and administration
the constraints and framework established by the ICAO language provisions
Support the application of globally harmonized language testing criteria
See the document “About ICAO’s Aviation English Language Test Endorsement Service” for additional detail on the process and its implementation.
The first two questions in that document provide additional detail for this slide.

4. Participation
Initial workgroup with representatives from ILTA, IFALPA, IFATCA, ICAEA developed the endorsement process.
Expanded the group in January to increase test service provider representation:
ILTA, IFALPA, IFATCA, ICAEA
Representatives from China, Latin
America, Eastern Europe, Eurocontrol
Participatory approach based on
extensive use of communication
technology.
The diagram in this slide illustrates using a cyclical approach to work through discussions and decision making. While the diagram implies completing steps only once, the workgroup consistently followed an iterative cycle of discussion, review and decision making throughout the project. This was therefore a participatory approach based on equal contributions and collaboration.
The workgroup worked mostly at a distance using various computer-based tools. For example, a webinar platform, document sharing services and VOIP communication (e.g. Skype). Thanks to everyone’s flexibility and willingness to work through some technological problems, the workgroup was able to complete the project nearly entirely at distance.

5. Project Overview Activities Sept-Dec 2010 Jan-Apr 2011 May-Jun 2011
Post start-up
Develop a test endorsement process
Develop test evaluation criteria
Pilot the endorsement process
Implement / manage process
This slide is intended as broad project timeline to provide the audience with a sense of how the project evolved over time.
The project currently remains essentially on time, with some slippage conducting the pilot. More about the pilot on slides 10 and 11 of this presentation.
The step to “Implement / manage process” consists of setting up a more permanent test endorsement service within ICAO to ensure ongoing improvement of what we now have. ICAO intends to continue working closely with professional associations and other stakeholders to keep the endorsement process as practical, effective and efficient as possible.

6. Endorsement: Main Parts
Evaluate original tests (created by TSP) - test construction / ICAO constraints & framework (formative process)
Test Endorsement
Review and confirm endorsed tests continue meeting set criteria (3 years)
Test re-endorsement
There are three main parts to the endorsement process:
- to evaluate original tests from Test Service Providers (i.e. tests created by TSP submitting it for endorsement;
- to review and re-endorsed tests every three years; and
to provide up-to-date information on tests endorsed to States and members of the public.
Test endorsement and re-endorsement remain team efforts in which language testing and operational experts will together determine whether or not a test can be endorsed. Working in cooperation with associations like IFALPA, ICAO will set up and manage a pool of test evaluators with appropriate expertise.
The first part is the most demanding for all concerned, in terms of time and effort. Test Service Providers will be required to provide detailed information on test design, development and use to be considered for endorsement.
Confirm test administration data – assist States with language proficiency oversight
Test Administration

7. Endorsement: TSP Perspective
This diagram shows what TSP will do to complete the endorsement process.
Create an account for identification and security purposes
Complete an online pre-assessment questionnaire to help confirm their test is ready for evaluation and endorsement. TSP complete the questionnaire online and receive feedback immediately after highlighting possible test problems. TSP can then decide to stop-and-fix or continue as desired.
Once ready to continue, TSP pay a non-refundable endorsement fee.
Once payment is confirmed, TSP receive additional instruction to prepare and submit more detailed test documentation to be reviewed by an evaluation team.
TSP with an endorsed test will receive a certificate confirming ICAO has reviewed and confirmed the test is in good standing.
Endorsed tests will be identified in the public area of the endorsement website.

8. Test Endorsement Website
ICAO is creating a dedicated website to act as the main communication channel between all those involved in the endorsement process. The site will reflect the parts of the process previously identified.
The website is therefore essentially divided in 4 parts:
For the system administrator (system caretaker)
For TSP: with the tools and information needed to submit tests for endorsement
For evaluators to learn about and become test evaluators, and track evaluation assignments
To display a list of endorsed tests to the public.
Parts of the website for the administrator, TSP and test evaluators will be secure and require login. The public area will be open to all.

9. Test Endorsement Website
This image shows the home page of the test endorsement website. As indicated, it is primarily intended as a communication channel between those involved in the process and is not intended for broader public use. It currently remains a simple tool to be refined over time as ICAO learns from the experience of offering the service.

10. Test Endorsement Pilot
Just starting
Includes 3 existing tests from different TSP
Working with the project workgroup as evaluators
Opportunity to confirm:
The endorsement process works as intended and is feasible over time
Test evaluation criteria is reliable
Where and how to improve all aspects of the process
How best to manage implementation over time

11. Next Steps
Complete the pilot: review / improve the endorsement process, criteria and tools
Set up more ‘permanent’ process
Ensure full integration of test endorsement and other processes within ICAO
Build a capacity to continually work with users to improve the system
Operate regular test endorsement service
2nd bullet refers to setting up a more permanent test endorsement service as indicated in slide 5, above.
3rd bullet refers to other endorsement services being considered by ICAO. For example, to endorse training programs (technical, not language) or other types of tests. While there are currently no set plans to develop these services, there is growing interest in applying the experience of this project in others.

12. Thank you! Jean-Marc Guillemette Workplace Learning Architect
Project Manager
ICAO Aviation English Language Test Endorsement