Glyn Jones Product Development Manager Dr John H.A.L. De Jong Director of Test development Pearson Language Assessments, London Linking Exams to the Common European Framework

Nature and purpose of CEF A framework of reference (not a prescriptive scheme) A set of interrelated scales of proficiency based on can do statements Front end Global Scale and Self Assessment Grid provide overview of each level, accessible to non- expert users; back end scales and commentary provide analytical tools for teachers, materials writers and assessors

Linking exams to the CEF Strong motivation for exam providers so link to CEF driven by demand for –Transparency (“What level is your exam?”) –Comparability (“is my certificate the same level as your certificate?”) No central accreditation or auditing agency Linking is voluntary Recommendations published in Council of Europe Manual

Problems Gaps and inconsistencies in the CEF –Variable terminology (“understand”, “follow”, “identify”) –Gaps (eg skimming) –Inconsistencies (terms mentioned at some levels but not at others) –Undefined terms (“simple”, “familiar”, “complex”) (See Alderson et al 2004) The nature of language proficiency and the notion of level

Communicative Strategies Communicative Language Competencies Hierarchy of Scales To illustrate hierarchical the principle one branch of the hierarchy is worked out in detail to the right  ReceptionProduction Understanding Note: this branch of the hierarchy is part of the quantity dimension: how much a language learner can do a native speaker Conversation Informal Discussion Formal Discussion Obtaining Goods and Services Interviewing & being interviewed SpokenWritten InteractionMediation Overall language Proficiency Global Scale And at each further node in the hierarchy the CEF offers descriptive scales. Communicative Activities The Global scale is for Overall Language Proficiency.

The Quality Hierarchy Communicative Strategies General Linguistic Vocabulary Range Grammatical Accuracy Phonological Control Vocabulary Control Orthographic Control LinguisticSociolinguisticPragmatic Communicative Language Competencies Communicative Activities Overall language Proficiency Global Scale The same Global Scale analyzed here into the constituent elements of the quality dimension: how well a learner can do. And here too there is a scale at each node.

Language proficiency C2 C2 Conveys finer shades of meaning precisely and naturally. C1 C1 Shows fluent, spontaneous expression in clear, well-structured speech. B2 B2 Relates information and points of view clearly and without noticeable strain. B1 B1 Relates comprehensibly main points he/she wants to make on familiar matters A2 A2 Relates basic information on, e.g. work, background, family, free time etc.. A1 A1 Makes simple statements on personal details and very familiar topics. 1%5% 15% 15%.50%80%95% 80% A person at the border between level A2 and B1 will probably be able to do 80% of the tasks at level A2 but can also do 50% of the tasks at level B2, etc.

Chance to perform task 50% 100% 0% 80% A1A2B1B2C1C2<A1 A2  This person will have 80% probability of being able to perform these A2 tasks CEF scaled on IRT model  A person at the lower border of A2 (i.e. between A1 and A2) will have less chance on the universe of A2 tasks: about 50 %

Linking procedures Familiarisation Specification Standardisation Empirical validation

Familiarisation Discussions of the levels as a whole using the global scale, with special attention to the broad levels A, B and C: what does it mean to be a basic user, an independent user, a proficient user? Self-assessment of own language level in a foreign language Sorting into piles by level of individual descriptors from a CEF scale and follow-up discussion to identify the key features which distinguish one level from the previous one

Specification example

Standardisation Item writers (receptive skills) Judging level of CEF standardised items Applying judgment to own test items Assessors (productive skills) Judging level of CEF standardised samples (video, written samples) Applying judgment to own samples Familiarisation Discussion towards unanimity

Empirical validation Pre-testing Selection of items for internal consistency Calibration of items against anchor items Determining cut scores for in relation to CEF levels

Pre-testing items: linked design

Determining cut scores B x x x x x x x x x x x x x x x x x x x x x x x x x x xx x x x x x x 67 Anchor test Main test

References Council of Europe Relating (2003) Relating Language Examinations to the Common European Framework of Reference for Languages: Learning, Teaching, Assessment (CEFR) Manual – Preliminary Pilot Version. Strasbourg: Author Rasch, G. (1960) Probabilistic models for some intelligence and attainment tests. Copenhagen: Danish Institute for Educational Research J Charles Alderson, Neus Figueras, Henk Kuiper, Guenter Nold, Saulii Takala, Claire Tardieu (2004) Final Report on the Dutch CEF Construct Project The Dutch Grid DIALANG