Designing and Developing Useful Language Tests Lyle F. Bachman Department of Applied Linguistics & TESL University of California, Los Angeles

Uses of Language Tests F to make inferences about individuals’ language ability F to make predictions about individuals’ ability to use language in contexts outside the test itself F to make decisions about individuals

Correspondences between Language Test Performance and Language Use

Language Test PerformanceLanguageUse Characteristics of the Language Use Task Characteristics of the Language Test Task Characteristics of the Language User Characteristics of the Test Taker

Language Test PerformanceLanguageUse Characteristics of the Language Use Task Characteristics of the Language Test Task Language Ability of the Language User Language Ability of the Test Taker

Characteristics of the Language Use Task Language Test PerformanceLanguageUse Language Ability of the Test Taker Language Ability of the Language User Characteristics of the Language Test Task

A Framework of Task Characteristics Language Use Task an activity that involves individuals in using language for the purpose of achieving a particular goal in a particular situation (Bachman & Palmer, 1996, p. 44)

A Framework of Task Characteristics Target Language Use (TLU) Domain a set of specific language use tasks that the test taker is likely to encounter outside of the test itself, and to which we want our inferences about language ability to extrapolate (Bachman & Palmer, 1996, p. 44)

A Framework of Task Characteristics Task Characteristics  Setting: the physical circumstances under which either language use or language testing takes place. Includes the physical characteristics of the setting, the participants involved, and the time of the task.

A Framework of Task Characteristics Task Characteristics  Rubric: the context for the task; includes those characteristics that provide the structure for the task, and constrain how language users or test takers are expected to respond to these tasks

A Framework of Task Characteristics Task Characteristics  Input: material contained in the task, which test takers need to process in some way, and to which they are expected to respond.

A Framework of Task Characteristics Task Characteristics  Expected response: the language use that is expected, given the way in which the rubric, or context, for the task is configured, and the particular input that is provided

A Framework of Task Characteristics Task Characteristics  Relationship between input and response: Reactivity: the degree of reciprocity, or interaction involved,Reactivity: the degree of reciprocity, or interaction involved, Scope: the amount and range of input that needs to be processed in order to respond, andScope: the amount and range of input that needs to be processed in order to respond, and Directness: the extent to which the response can be made by using information in the input by itself, or whether the language user or test taker must also rely on information in the context or in his or her own real world knowledge.Directness: the extent to which the response can be made by using information in the input by itself, or whether the language user or test taker must also rely on information in the context or in his or her own real world knowledge.

Describing Target Language Use Tasks 1.Identify the TLU tasks in the relevant domain. 2.Select TLU tasks as a basis for designing test tasks. 3.Describe these tasks in terms of their task characteristics.

Describing Target Language Use Tasks 1.Identify the TLU tasks in the relevant domain. 2.Select TLU tasks as a basis for designing test tasks. 3.Describe TLU tasks in terms of their task characteristics.

TLU Domain PR Department of a Phone Company Task 2 phone discussions with clients, co-workers Task 1 writing promotional material Task 3 responding to customer complaints on the phone Task 4 responding to customercomplaints in writing Task 5 giving complaint information to co-workers by phone Task 6 giving complaint information to co-workers in writing Task Characteristics Framework TLU Task Type A (letter) TLU Task Type B (memo)

Developing Test Tasks F Modify TLU task types for use as test tasks F Create original test tasks

TLU Task Type A (letter) Modify TLU Task Type to Develop Test Task Specifications Test Task Specifications A (write formal letter to customer) TLU Task Type B (memo) Test Task Specifications B (write memo to co-worker)

Developing Test Tasks F Modify TLU task types for use as test tasks F Create original test tasks

Distinctive Characteristics of TLU Task Type Create Specifications for Original Test Tasks Test Task Specifications

Example Test Development Project Project setting: telephone company needs to hire new employees

Describing Target Language Use Tasks 1.Identify the TLU tasks in the relevant domain. 2.Select TLU tasks as a basis for designing test tasks. 3.Describe these tasks in terms of their task characteristics.

Example TLU Domain PR Department of a Phone Company Task 1 writing promotional material Task 2 phone discussions with clients, co-workers Task 3 responding to customer complaints on the phone Task 4 responding to customercomplaints in writing Task 5 giving complaint information to co-workers by phone Task 6 giving complaint information to co-workers in writing

Selecting TLU Tasks for Use in Designing Test Tasks F Some TLU tasks may be essential to performing the job

Example TLU Domain PR Department of a Phone Company Task 4 responding to customercomplaints in writing Task 2 phone discussions with clients, co-workers Task 6 giving complaint information to co-workers in writing Task 1 writing promotional material Task 3 responding to customer complaints on the phone Task 5 giving complaint information to co-workers by phone

Selecting TLU Tasks for Use in Designing Test Tasks F Some TLU tasks may not require the areas of language ability we want to measure. F Some TLU tasks may not be appropriate for all test takers.

Describing Target Language Use Tasks 1.Identify the TLU tasks in the relevant domain. 2.Select TLU tasks as a basis for designing test tasks. 3.Describe these tasks in terms of their task characteristics.

TLU Domain PR Department of a Phone Company Task 2 phone discussions with clients, co-workers Task 1 writing promotional material Task 3 responding to customer complaints on the phone Task 4 responding to customercomplaints in writing Task 5 giving complaint information to co-workers by phone Task 6 giving complaint information to co-workers in writing Task Characteristics Framework TLU Task Type A (letter) TLU Task Type B (memo)

Developing Test Tasks F Modify TLU task types for use as test tasks. F Create original test tasks.

Developing Test Tasks F Modify TLU task types for use as test tasks F Create original test tasks

TLU Task Type A (letter) Modify TLU Task Type to Develop Test Task Specifications Test Task Specifications A (write formal letter to customer) TLU Task Type B (memo) Test Task Specifications B (write memo to co-worker)

Developing Test Tasks F Modify TLU task types for use as test tasks F Create original test tasks

Distinctive Characteristics of TLU Task Type Create Specifications for Original Test Tasks Test Task Specifications

Conclusion Bachman-Palmer Approach to Language Test Development FProvides test developers with a theoretically grounded and principled basis for developing and using language tests.

Conclusion Bachman-Palmer Approach to Language Test Development FProvides test developers with an understanding that will enable them to make their own decisions about either selecting or developing a language test that will be useful for their particular situation.

Conclusion Bachman-Palmer Approach to Language Test Development FRequires test developers to consider the fundamental correspondence between language test performance and non-test language use.

TLU Domain Task Task Task Task Task Task

Using Needs Analysis to Identify TLU Tasks 1.Identify stakeholders. 2.Develop procedures for gathering information about tasks. 3.Gather information about the domain and tasks. 4.Analyze tasks in terms of their task characteristics. 5.Make an initial grouping of tasks into categories.

Test Usefulness F Principle 1: Overall test usefulness, rather than individual qualities is to be maximized F Principle 2: Individual qualities cannot be evaluated independently, but most evaluated in terms of their combined effect on overall usefulness

Test Usefulness F Principle 3: Test usefulness and the appropriate balance among the different qualities can only be determined for a given specific testing situation.

Qualities of Usefulness F Reliability F Construct Validity F Authenticity F Interactiveness F Impact F Practicality

Describing Target Language Use Tasks 1.Identify the TLU tasks in the relevant domain. 2.Select TLU tasks as a basis for designing test tasks. 3.Describe these tasks in terms of their task characteristics.