RATERS: SELECTING AND TRAINING FOR BENCHMARKING AND PROFICIENCY EVALUATION PURPOSES ICAO WORKSHOP ON LANGUAGE PROFICIECY Baku, Azerbaijan, December 7-9, 2005 Evgenia Povaliy, Aviation English Instructor Sergey Melnichenko, Deputy Director Moscow ATC Training Center

CASE STUDY Accidents where language was a causal or contributing factor Accidents where language was a causal or contributing factor Phraseologies only are not enough Phraseologies only are not enough Plain language is needed in non- routine and urgent situations Plain language is needed in non- routine and urgent situations

English as an Instrument of Communications 70 per cent of 28,000 safety reports were connected with communications 70 per cent of 28,000 safety reports were connected with communications 70 per cent of those who communicate are non-native English speakers 70 per cent of those who communicate are non-native English speakers No other existing live or artificial language is either better or worse than English No other existing live or artificial language is either better or worse than English English has become a “lingua franca” English has become a “lingua franca”

Manual on the implementation of ICAO Language Proficiency Requirements (Doc 9835) Licensing authorities should require specialized evaluation of those who do not demonstrate Level 6 proficiency Licensing authorities should require specialized evaluation of those who do not demonstrate Level 6 proficiency Phraseologies-only testing is not appropriate Phraseologies-only testing is not appropriate Demonstration of actual speaking and listening ability is required Demonstration of actual speaking and listening ability is required Until 5 March 2008 States may continue to use procedures they currently have in place to assess English proficiency of flight crews and controllers Until 5 March 2008 States may continue to use procedures they currently have in place to assess English proficiency of flight crews and controllers

Requirements for raters Higher education in Linguistics Higher education in Linguistics Level 6 proficiency according to the ICAO Scale Level 6 proficiency according to the ICAO Scale Highly developed moral standards Highly developed moral standards Aviation background Aviation background Relevant training, preferably in an English-speaking country Relevant training, preferably in an English-speaking country

THE SCALE Who has seen it? Who has seen it? Who is happy with it? Who is happy with it? Who has used it for benchmarking evaluation? Who has used it for benchmarking evaluation? Who has been taught to use the Scale? Who has been taught to use the Scale?

TESTS Entry Entry Placement Placement Diagnosis Diagnosis Progress Progress Proficiency Proficiency

ENTRY TEST Enables Airline Operators and ANS providers to avoid new personnel that is likely to have problems in 2008 because of poor knowledge of English at present

PLACEMENT TEST Enables Airline Operators and ANS providers to map out the level of existing personnel

DIAGNOSIS TEST Enables linguists to identify the needs of individuals (problem areas, course duration and its content)

PROGRESS TEST Enables Airline Operators, ANS providers and linguists to mark progress/no progress/regress and to use other training path when needed

PROFICIENCY TEST Shows the actual level of English language proficiency according to ICAO Scale

Available Tests Chinese Chinese Korean Korean American (Oral Proficiency Interview) American (Oral Proficiency Interview) British (Mayflower College) British (Mayflower College) Russian (Moscow ATC Center) Russian (Moscow ATC Center) Eurocontrol (ELPAC – July 2007) Eurocontrol (ELPAC – July 2007)

Drawbacks Computer – no interaction Computer – no interaction Oral Proficiency – time and manpower- consuming Oral Proficiency – time and manpower- consuming THERE IS NO IDEAL TESTING TODAY THERE IS NO IDEAL TESTING TODAYBUT… “Any test at all” is not better than “no test” (Doc 9835) “Any test at all” is not better than “no test” (Doc 9835)

Steps to Make Introduce ICAO qualification requirements into national legislation Introduce ICAO qualification requirements into national legislation Select (through independent linguistic institutions) and train raters Select (through independent linguistic institutions) and train raters Evaluate current level of pilots and controllers Evaluate current level of pilots and controllers Ensure they are adequately taught General and Aviation English Ensure they are adequately taught General and Aviation English Evaluate proficiency level of pilots and controllers in 2008 Evaluate proficiency level of pilots and controllers in 2008

LAST BUT NOT LEAST Code of ethics

Ethical Principles (ILTA Code of Ethics) Raters shall have respect for humanity and dignity of each of their test takers Raters shall have respect for humanity and dignity of each of their test takers Raters shall have all information in confidence and use professional judgment in sharing it Raters shall have all information in confidence and use professional judgment in sharing it Raters should adhere to all relevant ethical principles embodied in national and international guidelines when undertaking any trial, experiment or other research activity Raters should adhere to all relevant ethical principles embodied in national and international guidelines when undertaking any trial, experiment or other research activity Raters shall not allow the misuse of their professional knowledge or skills Raters shall not allow the misuse of their professional knowledge or skills Raters shall continue to develop their professional knowledge, sharing it with colleagues Raters shall continue to develop their professional knowledge, sharing it with colleagues Raters shall share the responsibility of upholding the integrity of language testing profession Raters shall share the responsibility of upholding the integrity of language testing profession Raters shall strive to improve the quality of language testing, assessment and teaching services Raters shall strive to improve the quality of language testing, assessment and teaching services Raters shall be mindful of their obligations to the society Raters shall be mindful of their obligations to the society Raters shall regularly consider the potential effects on all stakeholders, reserving the right to withhold their professional services on the grounds of conscience Raters shall regularly consider the potential effects on all stakeholders, reserving the right to withhold their professional services on the grounds of conscience

THANK YOU! Evgenia Povaliy, Aviation English Instructor Sergey Melnichenko, Deputy Director Moscow ATC Training Center Evgenia Povaliy, Aviation English Instructor Sergey Melnichenko, Deputy Director Moscow ATC Training Center