Download presentation
Presentation is loading. Please wait.
1
Raters Training as a Safety Concern
Associate Member Raters Training as a Safety Concern Sergey MELNICHENKO CompLang Aviation Training Centre (Moscow), Deputy Director ICAO LPR Implementation Workshop Russia, St-Petersburg, May 24-26, 2011
2
Raters Training as a Safety Concern
Associate Member ICAO LPRI Workshop, St-Petersburg, Russia, 2011 - RATER AS A DECISION MAKER - SELECTION OF PERSONNEL - AB INITIO TRAINING - RATERS CERTIFICATION - INTER-RATER RELIABILITY - INTRA-RATER RELIABILITY - RASCH ANALYSIS - RECURRENT TRAINING - EVALUATION OF RATER’S ADEQUACY Raters Training as a Safety Concern
3
RATER AS A DECISION MAKER
Associate Member ICAO LPRI Workshop, St-Petersburg, Russia, 2011 RATER AS A DECISION MAKER SELECTION OF PERSONNEL AB INITIO TRAINING RATERS CERTIFICATION INTER-RATER RELIABILITY INTRA-RATER RELIABILITY RASCH ANALYSIS RECURRENT TRAINING EVALUATION OF RATER’S ADEQUACY Raters Training as a Safety Concern
4
Aviation English test results are felt around the globe
Associate Member ICAO LPRI Workshop, St-Petersburg, Russia, 2011 Aviation English test results are felt around the globe Raters Training as a Safety Concern
5
Inadequacy in Level Evaluation
Associate Member ICAO LPRI Workshop, St-Petersburg, Russia, 2011 Inadequacy in Level Evaluation Level 3 instead of Level 4: Loss of job and/or occupation Financial losses for AO or ANSP Market losses Redistribution of air transportation services Lack of confidence for test-takers Raters Training as a Safety Concern
6
Inadequacy in Level Evaluation
Associate Member ICAO LPRI Workshop, St-Petersburg, Russia, 2011 Inadequacy in Level Evaluation Level 4 instead of Level 3: Aviation safety is compromised Degradation to Level 2 Overconfidence for test-takers Raters Training as a Safety Concern
7
SELECTION OF PERSONNEL
Associate Member ICAO LPRI Workshop, St-Petersburg, Russia, 2011 RATER AS A DECISION MAKER SELECTION OF PERSONNEL AB INITIO TRAINING RATERS CERTIFICATION INTER-RATER RELIABILITY INTRA-RATER RELIABILITY RASCH ANALYSIS RECURRENT TRAINING EVALUATION OF RATER’S ADEQUACY Raters Training as a Safety Concern
8
ICAO recommendations on interlocutors and raters qualifications
Associate Member ICAO LPRI Workshop, St-Petersburg, Russia, 2011 ICAO recommendations on interlocutors and raters qualifications Raters Training as a Safety Concern
9
Familiarity with ICAO documentation Annex 1; Doc 9835;
Associate Member ICAO LPRI Workshop, St-Petersburg, Russia, 2011 ICAO recommendations on interlocutors and raters qualifications (Doc 9835) Familiarity with ICAO documentation Annex 1; Doc 9835; ICAO Rated Speech Samples CD; ICAO Circ 318 Raters Training as a Safety Concern
10
Familiarity with ICAO documentation
Associate Member ICAO LPRI Workshop, St-Petersburg, Russia, 2011 ICAO recommendations on interlocutors and raters qualifications (Doc 9835) Familiarity with ICAO documentation Operational and/or language specialist expertise Level 5 or Level 6 proficiency Familiarity with Aviation English RATERS CODE OF ETHICS Raters Training as a Safety Concern
11
AB INITIO TRAINING Raters Training as a Safety Concern
Associate Member ICAO LPRI Workshop, St-Petersburg, Russia, 2011 RATER AS A DECISION MAKER SELECTION OF PERSONNEL AB INITIO TRAINING RATERS CERTIFICATION INTER-RATER RELIABILITY INTRA-RATER RELIABILITY RASCH ANALYSIS RECURRENT TRAINING EVALUATION OF RATER’S ADEQUACY Raters Training as a Safety Concern
12
Theory of language testing
Associate Member ICAO LPRI Workshop, St-Petersburg, Russia, 2011 Theory of language testing ICAO documents: Annex 1, Annex 10, Doc 9432, Doс 9835, Circ 318 The ICAO Proficiency Rating Scale and holistic descriptors Correlating speech samples with the ICAO LPR Scale Raters Training as a Safety Concern
13
Raters Training as a Safety Concern
Associate Member ICAO LPRI Workshop, St-Petersburg, Russia, 2011 Raters training course should be aimed at experienced non-native and native English language teaching professionals and subject-matter experts who are currently teaching Aviation English and who want to be able to rate according to the ICAO Language Proficiency Scale Raters Training as a Safety Concern
14
RATERS CERTIFICATION Raters Training as a Safety Concern
Associate Member ICAO LPRI Workshop, St-Petersburg, Russia, 2011 RATER AS A DECISION MAKER SELECTION OF PERSONNEL AB INITIO TRAINING RATERS CERTIFICATION INTER-RATER RELIABILITY INTRA-RATER RELIABILITY RASCH ANALYSIS RECURRENT TRAINING EVALUATION OF RATER’S ADEQUACY Raters Training as a Safety Concern
15
Associate Member ICAO LPRI Workshop, St-Petersburg, Russia, 2011 Certification by national Civil Aviation Authority or by international aviation organization is mandatory Raters Training as a Safety Concern
16
Is geographical principle necessary?
Associate Member ICAO LPRI Workshop, St-Petersburg, Russia, 2011 Is geographical principle necessary? Raters Training as a Safety Concern
17
Are quotas necessary? Raters Training as a Safety Concern
Associate Member ICAO LPRI Workshop, St-Petersburg, Russia, 2011 Are quotas necessary? Raters Training as a Safety Concern
18
INTER-RATER RELIABILITY
Associate Member ICAO LPRI Workshop, St-Petersburg, Russia, 2011 RATER AS A DECISION MAKER SELECTION OF PERSONNEL AB INITIO TRAINING RATERS CERTIFICATION INTER-RATER RELIABILITY INTRA-RATER RELIABILITY RASCH ANALYSIS RECURRENT TRAINING EVALUATION OF RATER’S ADEQUACY Raters Training as a Safety Concern
19
Different raters but same results
Associate Member ICAO LPRI Workshop, St-Petersburg, Russia, 2011 Different raters but same results Raters Training as a Safety Concern
20
INTRA-RATER RELIABILITY
Associate Member ICAO LPRI Workshop, St-Petersburg, Russia, 2011 RATER AS A DECISION MAKER SELECTION OF PERSONNEL AB INITIO TRAINING RATERS CERTIFICATION INTER-RATER RELIABILITY INTRA-RATER RELIABILITY RASCH ANALYSIS RECURRENT TRAINING EVALUATION OF RATER’S ADEQUACY Raters Training as a Safety Concern
21
Different times but same results
Associate Member ICAO LPRI Workshop, St-Petersburg, Russia, 2011 Different times but same results Raters Training as a Safety Concern
22
RASCH ANALYSIS Raters Training as a Safety Concern
Associate Member ICAO LPRI Workshop, St-Petersburg, Russia, 2011 RATER AS A DECISION MAKER SELECTION OF PERSONNEL AB INITIO TRAINING RATERS CERTIFICATION INTER-RATER RELIABILITY INTRA-RATER RELIABILITY RASCH ANALYSIS RECURRENT TRAINING EVALUATION OF RATER’S ADEQUACY Raters Training as a Safety Concern
23
Evidence of the validity of the responses
Associate Member ICAO LPRI Workshop, St-Petersburg, Russia, 2011 Evidence of the validity of the responses Raters Training as a Safety Concern
24
Evidence of the validity of the responses
Associate Member ICAO LPRI Workshop, St-Petersburg, Russia, 2011 Evidence of the validity of the responses Correctional refresher course Raters Training as a Safety Concern
25
Evidence of the validity of the responses
Associate Member ICAO LPRI Workshop, St-Petersburg, Russia, 2011 Evidence of the validity of the responses Correctional refresher course Administrative measures to marginal raters Raters Training as a Safety Concern
26
RECURRENT TRAINING Raters Training as a Safety Concern
Associate Member ICAO LPRI Workshop, St-Petersburg, Russia, 2011 RATER AS A DECISION MAKER SELECTION OF PERSONNEL AB INITIO TRAINING RATERS CERTIFICATION INTER-RATER RELIABILITY INTRA-RATER RELIABILITY RASCH ANALYSIS RECURRENT TRAINING EVALUATION OF RATER’S ADEQUACY Raters Training as a Safety Concern
27
Annual refresher course for raters
Associate Member ICAO LPRI Workshop, St-Petersburg, Russia, 2011 Annual refresher course for raters Raters Training as a Safety Concern
28
Annual refresher course for raters New materials
Associate Member ICAO LPRI Workshop, St-Petersburg, Russia, 2011 Annual refresher course for raters New materials Raters Training as a Safety Concern
29
Annual refresher course for raters New materials Practical training
Associate Member ICAO LPRI Workshop, St-Petersburg, Russia, 2011 Annual refresher course for raters New materials Practical training Raters Training as a Safety Concern
30
EVALUATION OF RATER’S ADEQUACY
Associate Member ICAO LPRI Workshop, St-Petersburg, Russia, 2011 RATER AS A DECISION MAKER SELECTION OF PERSONNEL AB INITIO TRAINING RATERS CERTIFICATION INTER-RATER RELIABILITY INTRA-RATER RELIABILITY RASCH ANALYSIS RECURRENT TRAINING EVALUATION OF RATER’S ADEQUACY Raters Training as a Safety Concern
31
Rater is not a lifetime profession RATER TRAINING IS A SAFETY CONCERN
Associate Member ICAO LPRI Workshop, St-Petersburg, Russia, 2011 Rater is not a lifetime profession RATER TRAINING IS A SAFETY CONCERN Raters Training as a Safety Concern
32
Thank you Raters Training as a Safety Concern
Associate Member ICAO LPRI Workshop, St-Petersburg, Russia, 2011 Thank you Raters Training as a Safety Concern
Similar presentations
