How does DIALANG use the CEF? overall Framework and test content self-assessment instruments parts of feedback, e.g. the CEF scale standard setting

DIALANG Assessment Framework early decision to adopt CEF as DIALANG Assessment Framework guide on the content of test items context of language use; communicative tasks, purposes, themes, activities; texts; competences; communicative functions scale of proficiency + detailed specifications of activities, tasks, texts, etc from Waystage, Threshold and Vantage checklists based on CEF categories  coverage test specifications only partially based on the CEF

DIALANG self-assessment instruments DEVELOPMENT (PART 1): DIALANG self-assessment instruments are based on CEF descriptors best statements chosen (BN’s study) in English; for reading, listening, writing; 40-50 per skill slight modifications for lay audience, e.g. splitting and simplifying further a few new statements created

EXAMPLE: development of statements Council of E’s statements: (1) Can understand short simple personal letters. (2) Can understand enough to manage simple, routine exchanges without undue effort. DIALANG statements: (1) I can understand short simple personal letters. (2) I can understand enough to manage simple, routine exchanges without too much effort.

DIALANG self-assessment instruments DEVELOPMENT (PART 2): review of modified statements by B. North translation into 13 languages pilot testing of statements (with test items)  statistical analyses & calibration DIALANG Version 1: three 18-statement self-assessment instruments, one each for reading / listening / writing system calculates test taker’s self-assessed proficiency on the CEF scale  feedback

DIALANG feedback and CEF 1 Test result / Level of proficiency name of the level (A1- C2) + graph description (based on the CEF)

DIALANG feedback and CEF 2 Advisory feedback expanded descriptions of the levels advice on how to improve the skill tested

DIALANG feedback and CEF 3 Self-assessment feedback self-assessed level (A1 - C2) comparison of SA and test result (levels)

Standard setting in DIALANG purpose: to enable DIALANG to report test results on the CEF scale (by establishing cut-off scores)

WHY DO WE NEED STANDARD SETTING? ILLUSTRATION WHY DO WE NEED STANDARD SETTING? a DIALANG test TEST SCORE SCALE 110 (MAXIMUM SCORE) 109 … 70 69 68 67 66 65 64 2 1 (MINIMUM SCORE)

C2 C1 B2 ? B1 A2 A1 a DIALANG test CEF levels TEST SCORE CEF LEVEL 110 (MAX SCORE) 109 … 70 69 68 67 66 65 64 2 1 (MINIMUM SCORE) C2 C1 B2 ? B1 A2 A1

C2 ? C1 ? B2 ? B1 ? A2 ? A1? a DIALANG test CEF levels 71 70 69 68 67 TEST SCORE ABILITY SCALE CEF LEVEL … 71 70 69 68 67 66 65 64 63 .455 .421 .343 .300 .276 .240 .233 .199 .138 C2 ? C1 ? B2 ? B1 ? A2 ? A1?

Cut-off points? C2 ? C1 ? B2 ? B1 ? A2 ? A1 ? a DIALANG test Cut-offs / st. setting CEF levels TEST SCORE ABILITY SCALE CEF LEVEL … 71 70 69 68 67 66 65 64 63 .455 .421 .343 .300 .276 .240 .233 .199 .138 C2 ? C1 ? B2 ? B1 ? A2 ? A1 ? Cut-off points?

B2 .275 B1 a DIALANG test Cut-offs / st. setting CEF levels 70 69 68 TEST SCORE ABILITY SCALE CEF LEVEL … 70 69 68 67 .421 .343 .300 .276 .275 B2 66 65 64 .240 .233 .199 B1

B2 .275 B1 a DIALANG test Cut-offs / st. setting CEF levels persons = TEST SCORE ABILITY SCALE CEF LEVEL … 70 69 68 67 .421 .343 .300 .276 .275 B2 66 65 .240 .233 B1 persons = persons items = items = items

a DIALANG test Cut-offs / st. setting CEF levels .944 .275 -.243 TEST SCORE ABILITY SCALE CEF LEVEL … 90 .955 .944 .275 -.243 -1.266 C1 89 67 .924 .276 B2 66 30 .240 - .224 B1 29 15 - .250 - 1.222 A2 14 1.303 A1

Standard setting in DIALANG 1 steps in the DIALANG standard setting procedure: establish a group of experienced language teachers for each language familiarisation with the CEF scale judgement of each item, one skill at a time, against the CEF scale ”Is it reasonable to require that a person at level B1 gets this item right?” 2 versions used teachers / judges mark their ratings on a form

Standard setting in DIALANG 2 analysis of the data with a standard setting programme developed by Norman Verhelst at CITO results of the IRT analyses of the pilot tested items and the standard setting data (on the same items) are both taken into account series of cut-off points on an ability/difficulty (logit) scale for the skill in question comparison of cut-off points with a logit (ability) scale from the test to which the cut-offs are to be applied placing of cut-off points