Inter-rater reliability in the KPG exams The Writing Production and Mediation Module

Inter-rater reliability in KPG AIM: To check the effectiveness of the instruments employed throughout the rating process Rating Grid – Assessment Criteria Training Material & Training Seminars On-the-spot consultancy to raters

Script Raters Profile Experienced teachers Underwent initial training in rating KPG scripts Undergo specialized training for every test administration

Script rater training Specialized training on rating scripts based on expectations for every activity  Analysis of expected output  Presentation of rated scripts  Actual rating of selected samples Rating scripts under supervision

The rating procedure Each script is rated by two script raters randomly selected from a pool of trained raters Second ratings are independent of the first (no identifying information, no marks or symbols) Constant monitoring/consultancy during the process

METHODOLOGY OF STUDY Computing Inter-rater reliability

Sampling Random sample of at least 40% of the total number of scripts Periods: May 2005 to November 2007 Levels: B1, B2 & C1

Intraclass Correlation Coefficient ICC vs. Pearson’s r The ICC is an improvement over Pearson's as it takes into account the differences in ratings, along with the correlation between raters. ICC in SPSS Average measure reliability analysis for one-way random effects

Interpretation of ICC r <0.40  poor agreement 0.40≤ r ≤0.75  good agreement r >0.75  excellent agreement (Fleiss, 1981) r <0.00  poor agreement 0.00 ≤r ≤0.20  slight 0.21 ≤r ≤0.40  fair 0.41 ≤r ≤0.60  moderate 0.61 ≤r ≤0.80  substantial 0.81 ≤r ≤1.00  almost perfect (Landis & Koch, 1977)

KPG module 2 Free writing production Mediation

Findings MAY 2005 NOVEMBER 2005 MAY 2006 NOVEMBER 2006 MAY 2007 NOVEMBER 2007 B2 - FREE WRITING PRODUCTION 0,740,700,760,680,760,72 C1 - FREE WRITING PRODUCTION 0,570,560,630,520,590,66 B1 - FREE WRITING PRODUCTION 0,760,73

Findings

MAY 2005 NOVEMBER 2005 MAY 2006 NOVEMBER 2006 MAY 2007 NOVEMBER 2007 B2 - MEDIATION0,770,750,740,720,800,69 C1 - MEDIATION0,620,600,680,530,690,71 B1 - MEDIATION 0,830,88

Findings

Totals Descriptive Statistics N Min. Max.MeanMAY054,57,77,67NOV054,56,75,65MAY064,63,76,70NOV064,52,72,61MAY076,59,83,73NOV076,66,88,73

Totals

Conclusion Correlations are high – Positive impact of instruments Trendlines are sloping upwards – Experience in rating and training are directly related to rater agreement indices

Further research Task Analysis to investigate correlation between item difficulty and ICC In process: Detailed task analysis project carried out by linguists and psychologists  AIM: To determine the variables affecting the difficulty of a task