1. Introduction to the Validation Phase
Relating language examinations to the Common European Framework of Reference for Languages
Gábor Szabó
ECML ClassRelEx Workshop
Graz, November 2010

2. Suggested Linking Procedures in the Manual
Familiarization with the CEFR
Linking on the basis of specification of examination content
Standardization and Benchmarking
Standard setting
Validation: checking that exam results relate to CEFR levels as intended

3. What is validity? Does the test measure what it intends to measure?
The degree to which evidence and theory support the interpretations of test scores entailed by proposed uses of tests.
Traditional classification of validity:
Content validity
Construct validity
Criterion related validity
Face validity
More modern approach:
Validity seen as single unitary construct

4. Aspects of validity Content Validity
Operational validity: pilots and pretests
Psychometric aspects
Procedural validity of standardization
Internal validity of standard setting
External validation

5. Content validity
Does the test accurately reflect the syllabus on which it is based AND reflect the descriptors in the CEFR?
Does the content specification reflect all areas to be assessed in suitable proportions?

6. Operational validity
Do pilot populations accurately represent the target population of the test?
Is the pilot test takers’ performance representative of their true ability? (response validity)
Begrippen worden hierna uitgelegd

7. Psychometric aspects
Do the test’s psychometric qualities support validity claims?
CTT-based results
Test-level data
Reliability figures
Mean, mode, median
Standard deviation
Measurement error
Score distribution
Item-level data
Facility values
Discrimination indices

8. Psychometric aspects
Do the test’s psychometric qualities support validity claims?
IRT-based results
Item difficulty figures
Person ability figures
Fit statistics
items
persons
DIF (Differential Item Functioning; item bias)

9. Procedural validity of standardization
Has the procedure of standard setting had the effects as intended?
Was the training effective?
Did the judges feel free to follow their own insights?

10. Internal validity of standard setting
Are the judgments of the judges to be trusted?
Are judges consistent within themselves?
Are judges consistent with each other?
Is the aggregated standard to be considered as the definite standard?

11. External validation
Establishing the validity of a test in relation to an external point of reference (CEFR)
Correlation analysis
Validation of standardization
Teacher judgments
Application of anchor tests