Basic Issues in Language Assessment 袁韻璧輔仁大學英文系

Contents Introduction: relationship between teaching & testing Introduction: relationship between teaching & testingrelationship between teaching & testingrelationship between teaching & testing Forms of test delivery Forms of test delivery Forms of test delivery Forms of test delivery Characteristics of a good test Characteristics of a good test Characteristics of a good test Characteristics of a good test Validity, reliability, practicality, positive washback Validity, reliability, practicality, positive washback Validityreliabilitypracticality Validityreliabilitypracticality Multiple-choice reading tests Multiple-choice reading tests Multiple-choice reading tests Multiple-choice reading tests Computer-based testing Computer-based testing Advantages and disadvantages Advantages and disadvantages Advantagesdisadvantages Advantagesdisadvantages Conclusion Conclusion Conclusion

Relationship between Teaching & Testing Subordinate  partnership Subordinate  partnership (supportive, corrective) (supportive, corrective)

Forms of Test Delivery alternative assessment alternative assessment paper-&-pencil computer-based paper-&-pencil computer-based tests testing tests testing

Characteristics of a Good Test Validity Validity Reliability Reliability Practicality (feasibility) Practicality (feasibility) Positive washback Positive washback The effect of tests on teaching & learning The effect of tests on teaching & learning

Validity Definition: a test should measure what it is intended to measure, and nothing else (i.e., no external knowledge or other skills measured at the same time). Definition: a test should measure what it is intended to measure, and nothing else (i.e., no external knowledge or other skills measured at the same time). Types of validity Types of validity Face validity, content validity, construct validity, criterion-related validity Face validity, content validity, construct validity, criterion-related validity

Face Validity You know if the test is valid or not by ‘ looking ’ at it. You know if the test is valid or not by ‘ looking ’ at it. It “ looks right ” to other testers, teachers, and testees, etc. It “ looks right ” to other testers, teachers, and testees, etc. Essential to all kinds of tests, but it is not enough. Essential to all kinds of tests, but it is not enough.

Content Validity “ A test is said to have content validity if its content constitutes a representative sample of the language skills, structures, etc. with which it is meant to be concerned. ” (Hughes 1989, p. 22) “ A test is said to have content validity if its content constitutes a representative sample of the language skills, structures, etc. with which it is meant to be concerned. ” (Hughes 1989, p. 22) Also called rational or logical validity. Also called rational or logical validity. Check against: Check against: Test specification (test plan) Test specification (test plan) Teaching materials, textbooks Teaching materials, textbooks Course syllabus/objectives Course syllabus/objectives Another teacher or subject-matter experts Another teacher or subject-matter experts

Definition of Reliability “ The consistency of measures across different times, test forms, raters, and other characteristics of the measurement context ” (Bachman, 1990, p. 24). “ The consistency of measures across different times, test forms, raters, and other characteristics of the measurement context ” (Bachman, 1990, p. 24). The accuracy or precision with which a test measures something; consistency, dependability, or stability of test results. The accuracy or precision with which a test measures something; consistency, dependability, or stability of test results.

How to make sure the test is reliable for teachers Take enough samples of behavior Take enough samples of behavior Try to avoid ambiguous items Try to avoid ambiguous items Provide clear and explicit instructions Provide clear and explicit instructions Well layout Well layout Provide uniform and undistracted condition Provide uniform and undistracted condition Try to use objective tests Try to use objective tests Try to use direct tests Try to use direct tests Have independent, trained raters Have independent, trained raters Try to identify the test takers by number, not by names Try to identify the test takers by number, not by names Try to have more multiple independent scoring in subjective tests (Hughes, 1989, pp ). Try to have more multiple independent scoring in subjective tests (Hughes, 1989, pp ).

Practicality Practical consideration when planning tests or ways of measurement, including cost, time/effort required Practical consideration when planning tests or ways of measurement, including cost, time/effort required Economy Economy Ease of Ease of Scoring and score interpretation Scoring and score interpretation Administration Administration Test compilation Test compilation A test should be practical to use, but also valid and reliable. A test should be practical to use, but also valid and reliable.

Multiple-choice Reading Tests Comprehension — being able to find meaning in what is read Comprehension — being able to find meaning in what is read Three levels of comprehension: Three levels of comprehension: Literal, interpretive (or referential) & critical Literal, interpretive (or referential) & critical Problems of multiple-choice reading tests Problems of multiple-choice reading tests Recall the info. Or text recycling Recall the info. Or text recycling Ambiguous, flawed texts/items Ambiguous, flawed texts/items Information gaps in passages Information gaps in passages Unfair, tricky tasks (e.g., full of unfamiliar words) Unfair, tricky tasks (e.g., full of unfamiliar words) Too much background knowledge assumed Too much background knowledge assumed Scored for wrong reason or vice versa Scored for wrong reason or vice versa Test-taking techniques Test-taking techniques

Advantages of CBT Scoring done automatically and immediately Scoring done automatically and immediately Tests tailored to the particular abilities of each test taker Tests tailored to the particular abilities of each test taker Tests provided on demand Tests provided on demand Many item combos are possible  test security Many item combos are possible  test security Multi-media  multiple-intelligent learning Multi-media  multiple-intelligent learning

Disadvantages of CBT Writing tests: Writing tests: Do raters react differentially to printed vs. handwritten texts? Do raters react differentially to printed vs. handwritten texts? To testees: different composing processes To testees: different composing processes Reading tests: Reading tests: Do testees react in the same way to read texts presented on computer screen and texts printed on paper? Speaking tests (semi-direct tests): Speaking tests (semi-direct tests): Nature of communication: a shared human activity, involving interlocutors & interaction Nature of communication: a shared human activity, involving interlocutors & interaction

Conclusion Variables of test performance: Variables of test performance: Types/formats of tasks, nervousness, physical conditions of testees, rater factors, etc. Types/formats of tasks, nervousness, physical conditions of testees, rater factors, etc. Adoption of multiple methods of assessment, alternative assessment Adoption of multiple methods of assessment, alternative assessment Valid, reliable paper-and-pencil tests that have positive washback Valid, reliable paper-and-pencil tests that have positive washback CBT for classroom teachers — depending on the testing purpose & needs CBT for classroom teachers — depending on the testing purpose & needs