1. In Lord We Trust Language Assessment Topic : Criterion-Referenced Approach to Language Assessment Thom Hudson Prepared by Shaho Hoorijani

2. Presentation Outline Introduction Background CRM Interpretation Assessments Characteristics Which Differentiate Criterion-Referenced from Norm-Referenced Tests CRM Projects that are applied to Different Scale Contexts Challenges in the Development of CRM approach 2

3. Introduction Criterion-Referenced Approach to Language Assessment Definition of keyterms Criterion:1660s, from Latinized form of Greek kriterion "means for judging. Criterion: represents the underlying skill or knowledge base to which the test is being generalized, not necessarily the level of performance. Referenced: Something for which a name or designation stands. Domain: A set of skills or dispositions that examinees display when called to do so in a testing situation 3

4. Introduction Characteristics of Criterion-Referenced Approach to Language Assessment Measures student performance against a fixed set of predetermined criteria or learning standards. Determine which examinee have mastered a particular domain of knowledge or skill. Measures degree of competence attained by a particular student which is independent of reference to the performance of others. The CRM test assesses the content and skill standards for the domain of interest. 4

5. Background Why Criterion-referenced Language assessment developed? It is a reaction against the misuse of psychometric methods (especially reliability theory) that were developed for assessing individual differences in aptitudes and abilities.... Items that are answered correctly have lower discrimination indexes, they were often eliminated solely on that empirical criterion.... Many achievement test developers lost sight of content validity － that the items must first and foremost be representative of the domain (content) and possess objectives to be assessed, and focused on high reliability as an end in itself. (pp.184-185 emphasis in original) Hopkins, Stanley and Hopkins (1990) 5

6. Background Why Criterion-referenced Language assessment developed? The growing influence of outcomes-based and competency-based assessment has raised an awareness of the need for language assessment to reflect the specific learning objectives of a program or the particular abilities needed for language proficiency certification. The need to assess targeted features of a domain is the focus of CRM. 6

7. Background Test Administration Reasons Norm-referenced interpretation Criterion-referenced interpretation Which one is better? Depends on the desired assessment 7

8. Background Classroom Setting CRM assessment is the primary form of assessment carried out by teachers in classroom settings. Most language teachers are less interested in how their students rank in relation to other students than in what their students can do with the language feature that has been taught. The lack of descriptive clarity resulting from NRM tests does not provide teachers with clear objectives at which to aim specific instruction. 8

9. Background Criterion-referenced in 1960s and 1970s Behavioral psychology, audiolingualism, structural linguistics, and learning systems: – Its identification with the atomization of skills into discrete units and narrow instances of learned information. – It was not useful for higher order skills or complex learning Change from focus on narrow skills to broader concepts – Cognitive language processing and sociolinguistic ability – Importance of performance, task-based or authentic assessment – Assessment tasks that reflect the real-world criterion – Attention to qualifications regarding modality, social status, and language complexity 9

10. Characteristics Which Differentiate Criterion-Referenced from Norm-Referenced Tests 1.Test Purpose NRT The score is interpreted in terms of other examinees score These tests are predominantly used for admission and placement purposes CRT CRM is absolute rather than relative Tests are given to determine how much course material a student has learned and/or tasks are assigned and evaluated to see if they have been completed to established standards 10

11. Characteristics Which Differentiate Criterion-Referenced from Norm-Referenced Tests 2.Test Content and Structure NRT Very general, and students should have no idea of what will be on the test. A few relatively long subsets with heterogeneous item content in each test CRT Measure specific well-defined domain The content should be drawn from the course, and students should have a good idea of what they will encounter on the test. The content of the tests are constrained by the results of the needs analysis Short, well-defined subsets with homogeneous item content in each subset. 11

12. Characteristics Which Differentiate Criterion-Referenced from Norm-Referenced Tests 3.Test Development NRT Test development is primarily psychometric driven Importance is placed on statistical considerations ( item discrimination and score distribution) Designed to providing instruments that allow comparisons among examinees 12

13. Characteristics Which Differentiate Criterion-Referenced from Norm-Referenced Tests 3.Test Development 1.Scores on tests that are interpreted in terms of percentiles, such as the Test of English as a Foreign Language (TOEFL), would not be considered CRM. This is perhaps the least satisfactory meaning of the term in that it does not relate the measurement to a specified domain. 2.A focus of this sense of CRM is on the decision to be made and the cut score that is established. Such a test should be based on a clear real-world activity, perhaps identified through a needs analysis. 3.Status on the criterion is assessed by sampling the well-defined domain and setting a cut score. 4.This level refers to proficiency levels that are the basis for the criterion referencing. The levels of proficiency are viewed as cumulatively relating to development. Each stage of development would thus have a real-world proficiency dimension and a relationship to other stages of development. 13

14. Examples of CRM Projects This projects show examples of assessment attempts to provide tests designed to specify what examinees can do in relation to the criteria developed for each project. 1. ACCESS for ELLs test ((Assessing Comprehension and Communication in English State-to-State for English Language Learners )) Assessing proficiency levels in foreign language learning Designed ELPS for ELLs in Kindergarten through grade 12 (K-12) developed by WIDA Consortium. What is WIDA acronym stands for? It stands for World-class Instructional Design and Assessment 14

15. Examples of CRM Projects WIDA and the ELP Standards – The ACCESS for ELLs is the test operationalization of the WIDA ELPS frame- work – The ELPS are the basis for test and item specifications. – Items written from the PIs reflect the form of the academic language requirements in the standards. What is the responsibility of WIDA consortium? Promote the development of assessment and educational materials for ELLs Create English language proficiency standards for ELLs that were tied to academic content standards Assessments of ELLs in K-12 annually on their language proficiency 15

16. Examples of CRM Projects WIDA and the ELP Standards Three-dimensional ELP Constructs in WIDA: 1.Language complexity 2.Cognitive engagement 3.Context ELPS reflects the: language needed for the classroom and school functions, and Language for learning in school content areas Learning the language and learning school contents concurrently in 4 skills domain 16

17. Examples of CRM Projects WIDA and the ELP Standards ELPS Frameworks : a)Large-scale state assessment b)Classroom assessment Common features of these two framework: 1.English language Proficiency standards 2.Language domains 3.Grade level clusters 4.Language proficiency levels 17

18. Five WIDA ELP Standards Standard 1- SIL: English language learners communicate for SOCIAL AND INSTRUCTIONAL purposes within the school setting. Standard 2 – LoLA: English language learners communicate information, ideas, and concepts necessary for academic success in the content area of LANGUAGE ARTS. Standard 3 – LoMA: English language learners communicate information, ideas, and concepts necessary for academic success in the content area of MATHEMATICS. Standard 4 – LoSC: English language learners communicate information, ideas, and concepts necessary for academic success in the content area of SCIENCE. Standard 5 – LoSS: English language learners communicate information, ideas, and concepts necessary for academic success in the content area of SOCIAL STUDIES. 18

19. Examples of CRM Programs 2. The Common European Framework of Reference (CEFR) – It describes in a comprehensive way what language learners have to learn to do in order to use a language for communication and what knowledge and skills they have to develop so as to be able to act effectively. – The description also covers the cultural context in which language is set. The Framework also defines levels of proficiency which allow learners’ progress to be measured at each stage of learning and on a lifelong basis. – The text of the CEFR emphasizes learner’s ‘communicative needs’, including dealing with the business of everyday life, exchanging information and ideas, and achieving wider and deeper intercultural understanding. This is to be achieved by ‘basing language teaching and learning on the needs, motivations, characteristics and resources of learners.’ 19

21. Examples of CRM Programs 3. Assessment of Language Performance (ALP) – A small-scale project designed to use CRM approach in task-based language assessment – Provide examples of an array of possible task contexts in general topics to demonstrate how task-based assessment can be developed – The focus is on the functions of the real-world tasks on examinee’s language ability in use – Task-based test are goal-oriented use outside of language test situation – Predetermined real-world criterion elements 21

22. Examples of CRM Programs 3. Assessment of Language Performance (ALP) – Task difficulty is controlled through: a)Language code complexity b)Cognitive complexity: adding additional material not relevant to the completion of the task c)Communicative stress – Task factors varied due to their difficulty Needs Analysis plays an important role in task criteria 22

23. Challenges in the Development of CRM approach – Domain Specification, and Standards identification which causes: – Domain overspecifications – Too many item specifications – content standards may not be written in a way that facilitates the generation of items. – Hindering effects of item specification curricular change. That is, once the framework for assessment is refined, there can be a reluctance to make changes in the educational approach that will necessitate a reworking and rethinking of the assessment system. 23

24. Challenges in the Development of CRM approach Solution Test and item writers must collaborate with teachers, administrators, and other stakeholders in clarifying what specific types of examinee performances will lead to inferences about the examinee’s language and ability to engage in scientific inquiry. 24

25. Thank You 25