Developing Descriptors Brian North

1. Conceptualisation >Clarifying the construct. What are we describing? >Collecting relevant example, systems >Deciding categories in the descriptive scheme >Clarifying key questions 2. Construction >Creating the descriptor pool >Editing, drafting – filling gaps in the description 3. Validation >Qualitative: through iterative workshops with teachers >Quantitative: through IRT scaling of use in assessment 4. Interpretation >Set thresholds between levels >Summarise developing proficiency Stages in Developing Descriptors

1. Conceptualisation >Clarifying the construct. What are we describing? >Collecting relevant example, systems >Deciding categories in the descriptive scheme >Clarifying key questions 2. Construction >Creating the descriptor pool >Editing, drafting – filling gaps in the description 3. Validation >Qualitative: through iterative workshops with teachers >Quantitative: through IRT scaling of use in assessment 4. Interpretation >Set thresholds between levels >Summarise developing proficiency Stages in Developing Descriptors

Conceptualisation CEFR > CEFR descriptors: observable, functional outcomes “competence” descriptors also mainly observable proficiency > Interaction (BICS) / Production (CALP) > Illustrative videos of yr olds: difficulty with BICS “C2” LoS more complex than modern languages > Language aspects / non-language aspects > Discourse emphasis: genres; cognitive skills > Developmental – linked to cognitive growth Far less known about LoS than modern languages > 20 years experience with descriptors > 20 years developing descriptive scheme

Characteristics of LoS or (C) ALP We/you know it involves more: > specific, formal, abstract > explicit, detailed, conventionalised (= expectations) > cohesive and structured (e.g. sequencing) > coherent (goal-oriented) > planning, self-monitoring, internal feedback, editing > rhetorical skills and structures, strategies BUT > How much is really known about academic discourse? > Reception of exposition by the teacher > Interaction in class > Production by the teacher > To what extent are skills transversal – a common core?

Need for Collaboration & Research Vollmer > Pooling expertise and materials > Corpus of curricula and examination papers > Classroom observation and research > Interviews with teachers US “No Child Left Behind” (Bailey & Butler) > Analyse and align existing content and language standards > Need for observation and research on Teacher Talk > Need for empirical analysis of performance of mother tongue and second language students > Interviews with teachers (expectations: Recep; Production) > Analyse textbooks

Need for Collaboration & Research CEFR – Preparatory Work > Clarify concept: 1975 (Threshold) – 1992 (Proposal) > Experience with descriptors (BN: ) > Classroom discourse analysis (BN: ) > Involvement of stakeholders (Working Party ) CEFR – Project Design > Analyse and align existing systems > Interactive definition of categories with Authoring Group > Swiss National Research Project > Involvement of teachers in qualitative validation - Workshops

1. Conceptualisation >Clarifying the construct. What are we describing? >Collecting relevant example, systems >Deciding categories in the descriptive scheme >Clarifying key questions 2. Construction >Creating the descriptor pool >Editing, drafting – filling gaps in the description 3. Validation >Qualitative: through iterative workshops with teachers >Quantitative: through IRT scaling of use in assessment 4. Interpretation >Set thresholds between levels >Summarise developing proficiency Stages in Developing Descriptors

Key Questions I > Relationship to: > content standards > European Qualifications Framework > CEFR > Categories to be described > Transversal categories as in Table 5 of ERDLE proposal (p52) > Subcategories of Recep, Inter, Prod, Interp, Evaluation, Mediation? > Cognitive skills & strategies from Situation analysis (Beacco et al) > What else? > Style > concrete-salient features (CEFR-style) / abstract > Length – including assumptions “Can make a complaint”: B1 > broad-holistic / atomistic-analytic / both (Fleming)

Key Questions II > Thresholds to be described > expected language proficiency levels > types of discourse > stages of cognitive development > strategies > How to deal with “difficult parts” (non-language) e.g. Bildung > consideration of others > critical thinking, sound judgement and courage to express it? > flexibility in thinking and argumentation

1. Conceptualisation >Clarifying the construct. What are we describing? >Collecting relevant example, systems >Deciding categories in the descriptive scheme >Clarifying key questions 2. Construction >Creating the descriptor pool >Editing, drafting – filling gaps in the description 3. Validation >Qualitative: through iterative workshops with teachers >Quantitative: through IRT scaling of use in assessment 4. Interpretation >Set thresholds between levels >Summarise developing proficiency Stages in Developing Descriptors

Construction > Creating a classified bank of descriptors: > Collate / deconstruct all source systems > Eliminate doubles, redundancy > Identify gaps > Editing and drafting > Confirm style > Harmonise use of verbs (not done in CEFR English!) > Harmonise formulations > Create variations (for missing levels) > Author missing categories > Organisation > Classify with serial numbers > Translation to key languages / check translations with plurilinguals

1. Conceptualisation >Clarifying the construct. What are we describing? >Collecting relevant example, systems >Deciding categories in the descriptive scheme >Clarifying key questions 2. Construction >Creating the descriptor pool >Editing, drafting – filling gaps in the description 3. Validation >Qualitative: through iterative workshops with teachers >Quantitative: through IRT scaling of use in assessment 4. Interpretation >Set thresholds between levels >Set Summarise developing proficiency Stages in Developing Descriptors

Qualitative Validation > Analysis of teachers discussing proficiency: > Video of two learners > Who is better? Why? Justify your choice > “Repertory grid” analysis of categories teachers use to compare quality > Sorting descriptors into categories > Pile of (maximum 60) descriptors > Set of (maximum 4) envelopes labelled with the relevant categories > Discard envelope > Tick ones that are clear, relevant and useful > Sorting descriptors into levels > Pile of (maximum 15) descriptors for same category > Set of (CEFR 6) envelopes labelled with levels > Discard envelope / Tick ones that are clear, relevant and useful

Quantitative Validation - Purpose > To construct a scale from the descriptors for the “core construct” > To bolt onto / link to this scale sets of descriptors for categories that prove to be less core areas > To find out/confirm what level specific descriptors are > To discover which descriptors do not work > To confirm communality of the interpretation of the descriptors across: > Languages > Regions / countries / systems > Educational sectors

Quantitative Validation = Steps 1.Identify good/best descriptors from the pool after the qualitative validation 2.Confirm the supposed “level” of these descriptors 3.Create a set of overlapping checklists of c50 descriptors (like ELP checklists); each checklist targeted at a “level” 4.Define a rating scale: Yes/No; 0-4 for the descriptors 5.Identify classes at approximately the right level for each checklist 6.Arrange teacher assessment and/or self-assesment with the checklists 7.Collect minimum 150 examples of each checklist 8.IRT Rasch Model “Rating Scale Analysis” to build scale 9.Eliminate descriptors with 80%+ or 20%- (Rasch problem)

Anchor Design: CEFR (North 2000)

Recommended Design (after De Jong) Data Collection:

Vertical Scale of Descriptors

Extending the Core Scale I

Quantitative Validation - Prerequisites > Construct is well-defined – common understanding of what is being described/rated/scaled > Descriptors are well-formulated, clear and relevant > Teachers/learners are capable of making judgements about the areas concerned > There is a solid anchor design in the data collection

1. Conceptualisation >Clarifying the construct. What are we describing? >Collecting relevant example, systems >Deciding categories in the descriptive scheme >Clarifying key questions 2. Construction >Creating the descriptor pool >Editing, drafting – filling gaps in the description 3. Validation >Qualitative: through iterative workshops with teachers >Quantitative: through IRT scaling of use in assessment 4. Interpretation >Set thresholds between levels >Summarise developing proficiency Stages in Developing Descriptors

Setting Thresholds between Levels > Marking out equal intervals on the scale > Identifying „jumps“ in content described, gaps between clusters of descriptors > Comparing to original scale author intention > Comparing to Waystage, Threshold, Eurocentres, Cambridge exam levels > Fine-tuning for equal intervals > Checking for consistency, coherence

CEFR 3.6 Salient CharacteristicsA2 The majority of descriptors stating social functions: > greet people, ask how they are and react to news >handle very short social exchanges >discuss what to do, where to go and make arrangements Descriptors on getting out and about: >make simple transactions in shops, banks etc. >get simple information about travel and services

CEFR 3.6 Salient CharacteristicsB1 Maintain interaction and get across what you want to: > give or seek personal views and opinions > express the main point comprehensibly > keep going comprehensibly, even though pausing evident, especially in longer stretches Cope flexibly with problems in everyday life: > deal with most situations likely to arise when travelling > enter unprepared into conversations on familiar topics

CEFR 3.6 Salient CharacteristicsB2 Effective argument: > account for and sustain opinions in discussion by providing relevant explanations and arguments > explain a viewpoint on a topical issue giving the advantages and disadvantages of various options Holding your own in social discourse: >interact with a degree of fluency and spontaneity that makes regular interaction with native speakers possible >adjust to changes of direction, style and emphasis A new degree of language awareness: >make a note of "favourite mistakes" and monitor speech for them

1. Conceptualisation >Clarifying the construct. What are we describing? >Collecting relevant example, systems >Deciding categories in the descriptive scheme >Clarifying key questions 2. Construction >Creating the descriptor pool >Editing, drafting – filling gaps in the description 3. Validation >Qualitative: through iterative workshops with teachers >Quantitative: through IRT scaling of use in assessment 4. Interpretation >Set thresholds between levels >Summarise developing proficiency Stages in Developing Descriptors

Appendix United States “No Child Left Behind”

US “No Child Left Behind” > States have a legal duty to provide the support to ensure that every child is proficient in the academic language they need to be successful at school. > Must test this. > Must be at least a grade above and a grade below proficient. (not just the usual US master / non-master)

US “No Child Left Behind” > No overall framework or common reference points > Testing-led: dozens of consortia > No time for research > No systematic definition of the construct ALP > Confusion with “English Language Arts” (= creative writing for native speakers) or > Elaborated from language used in subject content standards > No definition of “proficient:” 15 significantly different interpretations > Some states 3, some 4, some 5 grades; all different names, numbers, concepts = CHAOS

Need for Collaboration & Research Vollmer > Pooling expertise and materials > Corpus of curricula and examination papers > Classroom observation and research > Interviews with teachers US “No Child Left Behind” (Bailey & Butler) > Analyse and align existing content and language standards > Need for observation and research on Teacher Talk > Need for empirical analysis of performance of mother tongue and second language students > Interviews with teachers (expectations: Recep; Production) > Analyse textbooks

US Experience US “No Child Left Behind” (Bailey & Butler) > Analyse and align existing content and language standards > Need for observation and research on Teacher Talk > Need for empirical analysis of performance of mother tongue and second language students > Interviews with teachers (expectations: Recep; Production) > Analyse textbooks

>Assumptions in Subject Standards: >Elementary School: observe, analyse, compare, describe, record >Middle School: identify, recognise, compose, explain  High School: recognise, describe, explain (Bailey and Butler 2003) >“Extracting the language features embedded in the content standards presented significant challenges …. >Bailey, Butler and Sato (2005) have been successful developing standards-standards linkages that involve both language and content standards BUT “procedures to establish such linkages … remain to this day in their infancy (Chaloub-Deville 2008) Analysing & Aligning Standards

US Experience US “No Child Left Behind” (Bailey & Butler) > Analyse and align existing content and language standards > Need for observation and research on Teacher Talk > Need for empirical analysis of performance of mother tongue and second language students > Interviews with teachers (expectations: Recep; Production) > Analyse textbooks

From 2001: >Analysis of functions in science classrooms >Teachers >Students >Repair strategies (Bailey and Butler 2003) >BUT >All tests produced before any research results were available – even in consortia aware of the problem (Chaloub-Deville 2008 Classroom Research

US Experience US “No Child Left Behind” (Bailey & Butler) > Analyse and align existing content and language standards > Need for observation and research on Teacher Talk > Need for empirical analysis of performance of mother tongue and second language students > Interviews with teachers (expectations: Recep; Production) > Analyse textbooks

>Students must learn acceptable ways of presenting information to the teacher – not usually explicitly taught >Very little study >“Teachers are rarely explicitly aware of their language expectations”  Dropped the idea of teacher interviews because “anecdotal” unreliable information (Bailey and Butler 2003) Teacher Expectations

US Experience US “No Child Left Behind” (Bailey & Butler) > Analyse and align existing content and language standards > Need for observation and research on Teacher Talk > Need for empirical analysis of performance of mother tongue and second language students > Interviews with teachers (expectations: Recep; Production) > Textbooks

US “No Child Left Behind” > No overall framework or common reference points > Testing-led: dozens of consortia > No time for research > No systematic definition of the construct ALP > Confusion with “English Language Arts” (= creative writing for native speakers) or > Elaborated from language used in subject content standards > No definition of “proficient:” 15 significantly different interpretations > Some states 3, some 4, some 5 grades; all different names, numbers, concepts = CHAOS

1. Conceptualisation >Clarifying the construct. What are we describing? >Collecting relevant example, systems >Deciding categories in the descriptive scheme >Clarifying key questions 2. Construction >Creating the descriptor pool >Editing, drafting – filling gaps in the description 3. Validation >Qualitative: through iterative workshops with teachers >Quantitative: through IRT scaling of use in assessment 4. Interpretation >Set thresholds between levels >Summarise developing proficiency Stages in Developing Descriptors