1. Introduction to the Specification Phase
Relating language examinations to the Common European Framework of Reference for Languages
José Noijons & Gabor Szabo
Graz, May 2009

2. Aims of today’s session on Specification
Raising awareness of good practice in linking to the CEFR
Defining minimum standards in linking
Providing practical support in linking

3. Raising Awareness
of the importance of good content analysis of language examinations;
of the CEFR, especially its descriptor scales;
of the rationale for relating language examinations to an international framework like the CEFR;
of ways in which the CEFR can be exploited in planning and describing language examinations

4. Defining minimum standards in terms of
the quality of content specification in language examinations;
the process of linking examinations to the CEFR.

5. Providing practical support to
complete the proposed content analysis and linking process;
provide evidence of internal consistency and construct validity;
report a claim that makes the results of the examination in question more transparent to both the users of examination results and to test takers themselves.

6. Four steps in the specification phase
assuring adequate familiarisation with the CEFR;
analysing the content of the examination or test in question in relation to the relevant categories of the CEFR; should an area tested not be covered by the CEFR, the user is asked to describe it;
profiling the examination or test in relation to the relevant descriptor scales of the CEFR on the basis of this content analysis;
making a first claim on the basis of this content analysis that an examination or test in question is related to a particular level of the CEFR.

7. Specification phase: three types of activity
familiarisation activities
filling in a number of checklists with details about the content of the language examination;
using relevant CEFR descriptors to relate the language examination to the levels and categories of the CEFR.

8. (1) Specification process gives the opportunity to
increase the awareness of the importance of a good content analysis of examinations;
become familiar with and use the CEFR in planning and describing language examinations;
describe and analyse in a detailed way the content of an examination or test;
provide evidence of the quality of the examination or test;

9. (2) Specification process gives the opportunity to
provide evidence of the relation between examinations/tests and the CEFR;
provide guidance for item writers;
increase the transparency for teachers, testers, examination users and test takers about the content and quality of the examination or test and its relationship to the CEFR.

10. Validity & Reliability
A test should test what it is supposed to test
Use test grids
Find experts for developing the test
A test should consistently measure what it is supposed to measure.
Organize experts’ work in a team
Assign screeners
Use data analysis

11. Objectivity Need for unambiguous answers to questions
Use written tests
Organize discussion among experts
Use marking schemes

12. Acceptability
The test should be seen to reflect desired and relevant behaviour
Involve all stakeholders
Take educational reform into account
Develop methods for standardisation
Organize research on psychometrics

13. Efficiency
There should be a positive correlation between information to be gained from the test and test length
Use written tests
Use short questions
Limit necessary stimulus material

14. General Descriptions: Forms A1-A8
Test Purpose
Development Process
Marking Procedures
Standards
Reporting
Analysis and Review Procedures
Rationale for decisions
Initial Estimations of CEFR levels

15. Communicative Language Activities: Forms A9 - A18
Form Communicative Language Activity
A9 Listening Comprehension
A10 Reading Comprehension
A11 Spoken Interaction
A12 Written Interaction
A13 Spoken Production
A14 Written Production
A15 Integrated Skill Combinations
A16 Integrated Skills
A17 Spoken Mediation
A18 Written Mediation

16. Communicative Language Competences: Forms A19 – A22
A19 Reception
A20 Interaction
A21 Production
A22 Mediation