Testing Grammar Dr. Muhammad Shahbaz

What are we trying to MEASURE? “ English grammar is chiefly a system of syntax, that decides syntax, that decides the order and the order and patterns in which patterns in which words are arranged into sentences. ” ( Close, 1982) into sentences. ” ( Close, 1982)

Why should we test grammar? Linguistic or communicative competence? Linguistic or communicative competence? Structures or functions? Structures or functions? Usage or use? Usage or use? Prescriptive or descriptive? Prescriptive or descriptive?

How should we test grammar? Separated from skills? Separated from skills? Discrete point or integrative items? Discrete point or integrative items? Focus on grammar or meaning? Focus on grammar or meaning?

What are the most common test formats for testing grammar? RECOGNITION Multiple choice items (Liu, 121) Multiple choice items (Liu, 121) Error-recognition items (Liu, 129) Error-recognition items (Liu, 129) True/False (Baxter, 35) True/False (Baxter, 35) Pairing and matching items Pairing and matching items Cloze/Modified Cloze (Hughes, 144) Cloze/Modified Cloze (Hughes, 144)

What are the most common test formats for testing grammar? Production Completion items (Hughes, 143) Completion items (Hughes, 143) Transformation items (Liu, 131) Transformation items (Liu, 131) Paraphrase (Hughes, 143) Paraphrase (Hughes, 143) Rearrangement Rearrangement Editing Editing Combination and addition items Combination and addition items Items involving the changing of words Items involving the changing of words

When should we go for grammar recognition? When… More material needs to be covered. More material needs to be covered. You want to test different levels of learning. You want to test different levels of learning. You have little time for scoring. You have little time for scoring. You are not interested in evaluating how well a test taker can formulate a correct answer. You are not interested in evaluating how well a test taker can formulate a correct answer. You have a large number of test takers. You have a large number of test takers.

When should we go for grammar production? When… You want to evaluate a person’s ability to formulate a correct answer. You want to evaluate a person’s ability to formulate a correct answer. You have more time to score the items You have more time to score the items You want to test a persons ability to apply concepts and information to a new situation. You want to test a persons ability to apply concepts and information to a new situation. You have a clear idea of the aspects and concepts that should be tested. You have a clear idea of the aspects and concepts that should be tested.

True/False Is this sentences true or false? “He works from Monday to Friday” Does not demonstrate broader knowledge Does not demonstrate broader knowledge Difficult to construct in higher levels Difficult to construct in higher levels Encourages guessing due to 50/50 chance Encourages guessing due to 50/50 chance Difficult to test attitudes toward learning Difficult to test attitudes toward learning

Multiple Choice John usually ______from Monday to Friday a. works b. worked c. has worked d. is working They can be tricky or too picky They can be tricky or too picky Difficult to test attitudes towards learning Difficult to test attitudes towards learning Knowledge is limited to options provided Knowledge is limited to options provided Difficult to construct at higher levels Difficult to construct at higher levels Encourages guessing (25% chance) Encourages guessing (25% chance) More than one option may be possible More than one option may be possible All options must be grammatically possible All options must be grammatically possible

Gap-Filling John usually ______(work) from Monday to Friday They must be part of a broader context They must be part of a broader context More than one option may be possible More than one option may be possible First gap may be offered as a sample First gap may be offered as a sample Difficult to test attitudes towards learning Difficult to test attitudes towards learning Knowledge is limited to options provided Knowledge is limited to options provided Limitations result in negative backwash Limitations result in negative backwash Focus should be on the aspect assessed. Focus should be on the aspect assessed.

Cloze/C-test John ______ works ______Monday ______Friday Tests student’s understanding of the whole language Tests student’s understanding of the whole language More than one option may be possible More than one option may be possible First letter may be offered as a sample First letter may be offered as a sample Difficult to test attitudes towards learning Difficult to test attitudes towards learning Some words may be impossible to guess Some words may be impossible to guess Can test student’s IQ rather than language proficiency Can test student’s IQ rather than language proficiency Difficult for some students (bias) Difficult for some students (bias)

Sentence Building Friday/usually/to/John/Monday/work Tests recognition and/or production Tests recognition and/or production More than one option may be possible More than one option may be possible Capital letters and punctuation may interfere in the production of the answer Capital letters and punctuation may interfere in the production of the answer Difficult to test attitudes towards learning Difficult to test attitudes towards learning Some context may be added Some context may be added Can test student’s IQ rather than language proficiency Can test student’s IQ rather than language proficiency Difficult for some students (bias) Difficult for some students (bias)

Sentence Transformation John is a doctor. He ______________ (work) More than one option may be possible More than one option may be possible Clues will limit the answers Clues will limit the answers Difficult to test attitudes towards learning Difficult to test attitudes towards learning Some context may be added Some context may be added Instructions are essential Instructions are essential Few aspects might be assessed (passive, reported, comparatives, conditionals…) Few aspects might be assessed (passive, reported, comparatives, conditionals…)

Correction/Editing John usually working from Monday to Friday Tests recognition of mistakes Tests recognition of mistakes Can also test production through corrections Can also test production through corrections More than one correction may be possible More than one correction may be possible Mistakes must be grammatically possible Mistakes must be grammatically possible Good backwash Good backwash Context is essential Context is essential

Five characteristics to measure communicative grammar 1. The test must provide more context than only a single sentence. 2. The test taker should understand what the communicative purpose of the task is. 3. He or she should also know who the intended audience is. 4. He or she must have to focus on meaning and not form to answer correctly. 5. Recognize is not sufficient. The test taker must be able “ to produce grammatical responses. ” (R. Dickins)

Therefore… Have the test taker say or write something of discourse length in order to perform some communicative task for a known audience. Have the test taker say or write something of discourse length in order to perform some communicative task for a known audience. What is said or written must make sense. What is said or written must make sense.

Split Sentences Write out some sentences, and then cut each sentence in half. Place the two halves in two separate columns and ask students to find the matching half. Write out some sentences, and then cut each sentence in half. Place the two halves in two separate columns and ask students to find the matching half. Students ’ abilities in combing sentences Students ’ abilities in combing sentences Grammar knowledge to all forms of the sentences Grammar knowledge to all forms of the sentences

Examples If you eat that If you eat that If you touch the dog If you touch the dog If you steal my boyfriend If you steal my boyfriend If you go out now If you go out now If you don ’ t leave If you don ’ t leave If you don ’ t book a ticket If you don ’ t book a ticket you ’ ll be sick you ’ ll be sick it ’ ll bite you it ’ ll bite you I ’ ll never speak to you again I ’ ll never speak to you again you ’ ll get soaked you ’ ll get soaked I ’ ll call the police I ’ ll call the police you ’ ll be lucky to get a seat you ’ ll be lucky to get a seat