Testing 05 Test Methods

Considerations in Test Methods Like traits tested, test methods also affect test performance. Test methods: features of the test context, the controlled versions of the contextual features that determine the nature of the language performance that is expected for a given test. The closer the correspondence between the characteristics of the test method and the essential features of language use contexts, the more authentic the test task will be for test takers.

Some Common Test Methods multiple choice fill-in cloze translation—sentence translation—passage composition dictation short question answering oral interview

Factors Affecting the Test Performance Test methods: Bachman (1981) found that scores from self-ratings loaded consistently more highly on method factors than on specific traits or ability factors.

Factors Affecting the Test Performance Individual attributes: test takers’ cognitive and affective characteristics, real world knowledge, age, sex, native language, educational and socio- economic background. Prior experience with the test, or test-wiseness: pacing, reading questions before the passages, ruling out alternatives. Such strategies are effective in reducing the unfamiliarity of the test. The provision of careful instructions with examples serves the same purpose.

Framework of Test Method Facets Testing environment Test rubric Input Expected response Relationship between input and response

Testing Environment Familiarity of the place and equipment Personnel Time of testing: morning or afternoon Physical conditions

Test Rubric Test organization Salience of parts: test takers may adopt differing response strategies. Sequence of parts: from the easiest to the most difficult or of the same difficult level Relative importance of parts: weight, percentage of the total score Time allocation: the speed of test taking can be a function.

Test Rubric Instructions Language: native or foreign Channel: visual or aural Specification of procedures and tasks: test takers can perform better whent hey clearly understand the task that is required of them. Explicitness of criteria for correctness: test taker’ knowledge of criteria can affect the test performance directly.

Input & Expected Response Format Nature of language

Format Channel & mode of presentation: visual, aural, oral Form: language, nonlanguage Vehicle: live human input (BEC oral exam) or canned human input (TSE) Language: target vs. native Identification of problem: underlined error identification vs. error correction of a line Degree of speededness: Difficulty in speededness may lead to guessing, or compensatory test taking strategies.

Nature of Language Length: the longer the sample, the greater the potential effects of the other characteristics Prepositional content Vocabulary: frequency, specialization Degree of contextualization: language use in a context rich with familiar information is a context- embedded discourse. (Context-reduced) the more context-embedded the input is, the more likely the test taker will be able to respond to its prepositional content.

Nature of Language Distribution of new information: highly compact input or highly diffuse input are both difficult to process. Type of information: concrete/abstract, positive/negative, factual/counterfactual

Nature of Language Topic: interesting, relevant, neutral. Test bias should be avoided. Genre: formal characteristics that are traditionally recognized. Differences in genres affect the interpretation of discourse. Particular types of language tests constitute genres.

Organizational Characteristics Grammar Cohesion Rhetorical The longer the language sample, the greater the need to incorporate organizational characteristics to make it interpretable.

Pragmatic Characteristics Illocutionary force: functionality of a given test task is the authenticity of the test, but some test task does not need the interpretation of illocutionary force for correct completion.

Sociolinguistic Characteristics Dialect or variety: standard language, American English, British English Register: test results may be misleading if the register of the test language is not appropriate to the register of the target language use context.

Restrictions on Expected Response The language used in language tests is sometimes characterized as “non-natural”, “non-normal”, “artificial”. It is restricted in these five aspects. Channel: noise in a real environment Format: in some tests the format is highly restricted, as in selection or identification response, while in others the format may be fairly unrestricted, as in composition test.

Restrictions on Expected Response Organizational characteristics Grammar: in multiple choice test, test takers deal with the meanings of a specific pair of words without the opportunity to express the equivalent meaning in their own words. Organization of discourse: in role play, test takers have to pretend to be a person he or she is actually not.

Relationship Between Input and Response Reciprocal input and response: with feedback as in oral interview Nonreciprocal input and response: without feedback as in most of the test Adaptive input and response: the input is influenced by the response, but without the feedback as in new TOFLE

Applications of Framework to Language Testing Description of language tests: describe or compare two tests or a test and a language program, e.G. TOEFL and CPE in terms of number of words and clauses, complex sentences, illocutionary acts, abstract, negative. Language test design: in test design, we often use tables of specifications about the components of ability to be tested, but the tables often fail to specify the facets of the test tasks.

Applications of Framework to Language Testing Validation of language tests: the validation must be based on a detailed description of both the abilities to be measured and the facets of the test methods. Formulation of hypotheses for language testing research: examination of the effects of specific facets, e.g. research of cloze test—deletion ratio, type of deletion, type of response, scoring criterion, organization of passage, passage content and discourse organization.