1. Introduction to the Validation Phase
Relating language examinations to the Common European Framework of Reference for Languages
José Noijons
APEOICVA/ECML
Valencia, March 2009

2. Suggested Linking Procedures in the Manual
Familiarisation with the CEFR
Linking on the basis of specification of examination content
Standardisation and Benchmarking
Standard setting
Validation: checking that exam results relate to CEFR levels as intended

3. What is validity?
Validity refers to the degree to which evidence and theory support the interpretations of test scores entailed by proposed uses of tests.
Although classical models divided the concept into various "validities," such as content validity, criterion validity, and construct validity, the modern view is that validity is a single unitary construct.
(Wikipedia)
In simpler terms: does the test measure what it intends to measure?

4. Aspects of validity Content Validity
Operational validity: pilots and pretests
Psychometric aspects
Procedural validity of standardization
Internal validity of standard setting
External validation

5. Content validity
Does the test accurately reflect the syllabus on which it is based AND reflect the descriptors in the CEFR?
Does the content specification reflect all areas to be assessed in suitable proportions?

6. Quality criteria for items
An item must be
relevant
at intended level
specific
objective
acceptable
transparent
efficient
in correct language
in a clear lay-out
These criteria will contribute to the validity and reliability
of a test.
Begrippen worden hierna uitgelegd

7. Quality criteria for items
Relevance
Is the item addressing the intended knowledge or skill at the intended CEFR level?
Does the item not test other knowledge and abilities than the intended ones (e.g. reading skill, general intelligence, knowledge of grammar)?
How to realize:
Refer to specific CEFR descriptors for each skill
Use test matrix (or test grid)
Relate questions with purpose of the test
Make items that are recognizable for the student

8. Quality criteria for items
At intended CEFR level
Is the question appropriate for the CEFR level ?
Does the question make a correct selection between those who know and those who do not know?
How to realize:
Make experts work in a team
Assign screeners
Avoid manipulation of wording to influence level of difficulty
Avoid unnecessary information
Use data analysis (through pretesting or afterwards)

9. Quality criteria for items
Transparent
Does the student know how many answers, details or arguments are expected?
Does the item relate to what students expect (syllabus, preparatory work)?
Do students know the maximum score for the item?
Is the item format known to the students?
How to realize:
Use clear instructions for student
Use clear terminology in line with syllabus and related tests
Indicate maximum score for an item
Use item formats or types students have been acquainted with

10. Validation of Standard Setting
Has the procedure of standard setting had the effects as intended: was the training effective, did the judges feel free to follow their own insights? These are questions of procedural validity.
Are the judgments of the judges to be trusted:
Are judges consistent within themselves?
Are judges consistent with each other?
Is the aggregated standard to be considered as the definite standard?
These questions and their answer constitute the internal validity of the standard setting.
Are the results of the standard setting – allocating students to a CEFR level on the basis of their test score – trustworthy? The basic answer to this question comes from independent evidence which corroborates the results of a particular standard setting: empirical validity.

11. Equivalence & Equating
How to make sure that exams are equal across
sittings
years
languages
regarding
content
CEFR level

12. Maintaining equivalence (1)
How to make sure that exams are equal across
sittings
years
languages?
Starting point:
Syllabus approved by the Generalitiat
global description per subject of the content of the exam

13. Maintaining equivalence (2)
Regarding
content
CEFR level
Through content specification in an exam model
Number of items and examination time
Item format: MC, open items
knowledge, skills etc.
Domains / topics
Test grid

14. Maintaining equivalence (3)
How to realise equivalence in difficulty?
Assuming that the overall achievements of the group of candidates in subsequent years are comparable:
adaptation of standards
equalizing the mean score
equalizing the percentage of pass/fail