Presentation is loading. Please wait.

Presentation is loading. Please wait.

Raters Training as a Safety Concern

Similar presentations


Presentation on theme: "Raters Training as a Safety Concern"— Presentation transcript:

1 Raters Training as a Safety Concern
Associate Member Raters Training as a Safety Concern Sergey MELNICHENKO CompLang Aviation Training Centre (Moscow), Deputy Director ICAO LPR Implementation Workshop Russia, St-Petersburg, May 24-26, 2011

2 Raters Training as a Safety Concern
Associate Member ICAO LPRI Workshop, St-Petersburg, Russia, 2011 - RATER AS A DECISION MAKER - SELECTION OF PERSONNEL - AB INITIO TRAINING - RATERS CERTIFICATION - INTER-RATER RELIABILITY - INTRA-RATER RELIABILITY - RASCH ANALYSIS - RECURRENT TRAINING - EVALUATION OF RATER’S ADEQUACY Raters Training as a Safety Concern

3 RATER AS A DECISION MAKER
Associate Member ICAO LPRI Workshop, St-Petersburg, Russia, 2011 RATER AS A DECISION MAKER SELECTION OF PERSONNEL AB INITIO TRAINING RATERS CERTIFICATION INTER-RATER RELIABILITY INTRA-RATER RELIABILITY RASCH ANALYSIS RECURRENT TRAINING EVALUATION OF RATER’S ADEQUACY Raters Training as a Safety Concern

4 Aviation English test results are felt around the globe
Associate Member ICAO LPRI Workshop, St-Petersburg, Russia, 2011 Aviation English test results are felt around the globe Raters Training as a Safety Concern

5 Inadequacy in Level Evaluation
Associate Member ICAO LPRI Workshop, St-Petersburg, Russia, 2011 Inadequacy in Level Evaluation Level 3 instead of Level 4: Loss of job and/or occupation Financial losses for AO or ANSP Market losses Redistribution of air transportation services Lack of confidence for test-takers Raters Training as a Safety Concern

6 Inadequacy in Level Evaluation
Associate Member ICAO LPRI Workshop, St-Petersburg, Russia, 2011 Inadequacy in Level Evaluation Level 4 instead of Level 3: Aviation safety is compromised Degradation to Level 2 Overconfidence for test-takers Raters Training as a Safety Concern

7 SELECTION OF PERSONNEL
Associate Member ICAO LPRI Workshop, St-Petersburg, Russia, 2011 RATER AS A DECISION MAKER SELECTION OF PERSONNEL AB INITIO TRAINING RATERS CERTIFICATION INTER-RATER RELIABILITY INTRA-RATER RELIABILITY RASCH ANALYSIS RECURRENT TRAINING EVALUATION OF RATER’S ADEQUACY Raters Training as a Safety Concern

8 ICAO recommendations on interlocutors and raters qualifications
Associate Member ICAO LPRI Workshop, St-Petersburg, Russia, 2011 ICAO recommendations on interlocutors and raters qualifications Raters Training as a Safety Concern

9 Familiarity with ICAO documentation Annex 1; Doc 9835;
Associate Member ICAO LPRI Workshop, St-Petersburg, Russia, 2011 ICAO recommendations on interlocutors and raters qualifications (Doc 9835) Familiarity with ICAO documentation Annex 1; Doc 9835; ICAO Rated Speech Samples CD; ICAO Circ 318 Raters Training as a Safety Concern

10 Familiarity with ICAO documentation
Associate Member ICAO LPRI Workshop, St-Petersburg, Russia, 2011 ICAO recommendations on interlocutors and raters qualifications (Doc 9835) Familiarity with ICAO documentation Operational and/or language specialist expertise Level 5 or Level 6 proficiency Familiarity with Aviation English RATERS CODE OF ETHICS Raters Training as a Safety Concern

11 AB INITIO TRAINING Raters Training as a Safety Concern
Associate Member ICAO LPRI Workshop, St-Petersburg, Russia, 2011 RATER AS A DECISION MAKER SELECTION OF PERSONNEL AB INITIO TRAINING RATERS CERTIFICATION INTER-RATER RELIABILITY INTRA-RATER RELIABILITY RASCH ANALYSIS RECURRENT TRAINING EVALUATION OF RATER’S ADEQUACY Raters Training as a Safety Concern

12 Theory of language testing
Associate Member ICAO LPRI Workshop, St-Petersburg, Russia, 2011 Theory of language testing ICAO documents: Annex 1, Annex 10, Doc 9432, Doс 9835, Circ 318 The ICAO Proficiency Rating Scale and holistic descriptors Correlating speech samples with the ICAO LPR Scale Raters Training as a Safety Concern

13 Raters Training as a Safety Concern
Associate Member ICAO LPRI Workshop, St-Petersburg, Russia, 2011 Raters training course should be aimed at experienced non-native and native English language teaching professionals and subject-matter experts who are currently teaching Aviation English and who want to be able to rate according to the ICAO Language Proficiency Scale Raters Training as a Safety Concern

14 RATERS CERTIFICATION Raters Training as a Safety Concern
Associate Member ICAO LPRI Workshop, St-Petersburg, Russia, 2011 RATER AS A DECISION MAKER SELECTION OF PERSONNEL AB INITIO TRAINING RATERS CERTIFICATION INTER-RATER RELIABILITY INTRA-RATER RELIABILITY RASCH ANALYSIS RECURRENT TRAINING EVALUATION OF RATER’S ADEQUACY Raters Training as a Safety Concern

15 Associate Member ICAO LPRI Workshop, St-Petersburg, Russia, 2011 Certification by national Civil Aviation Authority or by international aviation organization is mandatory Raters Training as a Safety Concern

16 Is geographical principle necessary?
Associate Member ICAO LPRI Workshop, St-Petersburg, Russia, 2011 Is geographical principle necessary? Raters Training as a Safety Concern

17 Are quotas necessary? Raters Training as a Safety Concern
Associate Member ICAO LPRI Workshop, St-Petersburg, Russia, 2011 Are quotas necessary? Raters Training as a Safety Concern

18 INTER-RATER RELIABILITY
Associate Member ICAO LPRI Workshop, St-Petersburg, Russia, 2011 RATER AS A DECISION MAKER SELECTION OF PERSONNEL AB INITIO TRAINING RATERS CERTIFICATION INTER-RATER RELIABILITY INTRA-RATER RELIABILITY RASCH ANALYSIS RECURRENT TRAINING EVALUATION OF RATER’S ADEQUACY Raters Training as a Safety Concern

19 Different raters but same results
Associate Member ICAO LPRI Workshop, St-Petersburg, Russia, 2011 Different raters but same results Raters Training as a Safety Concern

20 INTRA-RATER RELIABILITY
Associate Member ICAO LPRI Workshop, St-Petersburg, Russia, 2011 RATER AS A DECISION MAKER SELECTION OF PERSONNEL AB INITIO TRAINING RATERS CERTIFICATION INTER-RATER RELIABILITY INTRA-RATER RELIABILITY RASCH ANALYSIS RECURRENT TRAINING EVALUATION OF RATER’S ADEQUACY Raters Training as a Safety Concern

21 Different times but same results
Associate Member ICAO LPRI Workshop, St-Petersburg, Russia, 2011 Different times but same results Raters Training as a Safety Concern

22 RASCH ANALYSIS Raters Training as a Safety Concern
Associate Member ICAO LPRI Workshop, St-Petersburg, Russia, 2011 RATER AS A DECISION MAKER SELECTION OF PERSONNEL AB INITIO TRAINING RATERS CERTIFICATION INTER-RATER RELIABILITY INTRA-RATER RELIABILITY RASCH ANALYSIS RECURRENT TRAINING EVALUATION OF RATER’S ADEQUACY Raters Training as a Safety Concern

23 Evidence of the validity of the responses
Associate Member ICAO LPRI Workshop, St-Petersburg, Russia, 2011 Evidence of the validity of the responses Raters Training as a Safety Concern

24 Evidence of the validity of the responses
Associate Member ICAO LPRI Workshop, St-Petersburg, Russia, 2011 Evidence of the validity of the responses Correctional refresher course Raters Training as a Safety Concern

25 Evidence of the validity of the responses
Associate Member ICAO LPRI Workshop, St-Petersburg, Russia, 2011 Evidence of the validity of the responses Correctional refresher course Administrative measures to marginal raters Raters Training as a Safety Concern

26 RECURRENT TRAINING Raters Training as a Safety Concern
Associate Member ICAO LPRI Workshop, St-Petersburg, Russia, 2011 RATER AS A DECISION MAKER SELECTION OF PERSONNEL AB INITIO TRAINING RATERS CERTIFICATION INTER-RATER RELIABILITY INTRA-RATER RELIABILITY RASCH ANALYSIS RECURRENT TRAINING EVALUATION OF RATER’S ADEQUACY Raters Training as a Safety Concern

27 Annual refresher course for raters
Associate Member ICAO LPRI Workshop, St-Petersburg, Russia, 2011 Annual refresher course for raters Raters Training as a Safety Concern

28 Annual refresher course for raters New materials
Associate Member ICAO LPRI Workshop, St-Petersburg, Russia, 2011 Annual refresher course for raters New materials Raters Training as a Safety Concern

29 Annual refresher course for raters New materials Practical training
Associate Member ICAO LPRI Workshop, St-Petersburg, Russia, 2011 Annual refresher course for raters New materials Practical training Raters Training as a Safety Concern

30 EVALUATION OF RATER’S ADEQUACY
Associate Member ICAO LPRI Workshop, St-Petersburg, Russia, 2011 RATER AS A DECISION MAKER SELECTION OF PERSONNEL AB INITIO TRAINING RATERS CERTIFICATION INTER-RATER RELIABILITY INTRA-RATER RELIABILITY RASCH ANALYSIS RECURRENT TRAINING EVALUATION OF RATER’S ADEQUACY Raters Training as a Safety Concern

31 Rater is not a lifetime profession RATER TRAINING IS A SAFETY CONCERN
Associate Member ICAO LPRI Workshop, St-Petersburg, Russia, 2011 Rater is not a lifetime profession RATER TRAINING IS A SAFETY CONCERN Raters Training as a Safety Concern

32 Thank you Raters Training as a Safety Concern
Associate Member ICAO LPRI Workshop, St-Petersburg, Russia, 2011 Thank you Raters Training as a Safety Concern


Download ppt "Raters Training as a Safety Concern"

Similar presentations


Ads by Google