Introduction to the Validation Phase Relating language examinations to the Common European Framework of Reference for Languages José Noijons APEOICVA/ECML Valencia, 27-28 March 2009

Suggested Linking Procedures in the Manual Familiarisation with the CEFR Linking on the basis of specification of examination content Standardisation and Benchmarking Standard setting Validation: checking that exam results relate to CEFR levels as intended

What is validity? Validity refers to the degree to which evidence and theory support the interpretations of test scores entailed by proposed uses of tests. Although classical models divided the concept into various "validities," such as content validity, criterion validity, and construct validity, the modern view is that validity is a single unitary construct. (Wikipedia) In simpler terms: does the test measure what it intends to measure?

Aspects of validity Content Validity Operational validity: pilots and pretests Psychometric aspects Procedural validity of standardization Internal validity of standard setting External validation

Content validity Does the test accurately reflect the syllabus on which it is based AND reflect the descriptors in the CEFR? Does the content specification reflect all areas to be assessed in suitable proportions?

Quality criteria for items An item must be relevant at intended level specific objective acceptable transparent efficient in correct language in a clear lay-out These criteria will contribute to the validity and reliability of a test. Begrippen worden hierna uitgelegd

Quality criteria for items Relevance Is the item addressing the intended knowledge or skill at the intended CEFR level? Does the item not test other knowledge and abilities than the intended ones (e.g. reading skill, general intelligence, knowledge of grammar)? How to realize: Refer to specific CEFR descriptors for each skill Use test matrix (or test grid) Relate questions with purpose of the test Make items that are recognizable for the student

Quality criteria for items At intended CEFR level Is the question appropriate for the CEFR level ? Does the question make a correct selection between those who know and those who do not know? How to realize: Make experts work in a team Assign screeners Avoid manipulation of wording to influence level of difficulty Avoid unnecessary information Use data analysis (through pretesting or afterwards)

Quality criteria for items Transparent Does the student know how many answers, details or arguments are expected? Does the item relate to what students expect (syllabus, preparatory work)? Do students know the maximum score for the item? Is the item format known to the students? How to realize: Use clear instructions for student Use clear terminology in line with syllabus and related tests Indicate maximum score for an item Use item formats or types students have been acquainted with

Validation of Standard Setting Has the procedure of standard setting had the effects as intended: was the training effective, did the judges feel free to follow their own insights? These are questions of procedural validity. Are the judgments of the judges to be trusted: Are judges consistent within themselves? Are judges consistent with each other? Is the aggregated standard to be considered as the definite standard? These questions and their answer constitute the internal validity of the standard setting. Are the results of the standard setting – allocating students to a CEFR level on the basis of their test score – trustworthy? The basic answer to this question comes from independent evidence which corroborates the results of a particular standard setting: empirical validity.

Equivalence & Equating How to make sure that exams are equal across sittings years languages regarding content CEFR level

Maintaining equivalence (1) How to make sure that exams are equal across sittings years languages? Starting point: Syllabus approved by the Generalitiat global description per subject of the content of the exam

Maintaining equivalence (2) Regarding content CEFR level Through content specification in an exam model Number of items and examination time Item format: MC, open items knowledge, skills etc. Domains / topics Test grid

Maintaining equivalence (3) How to realise equivalence in difficulty? Assuming that the overall achievements of the group of candidates in subsequent years are comparable: adaptation of standards equalizing the mean score equalizing the percentage of pass/fail