Module 6 Test Construction &Evaluation

Lesson’s focus Stages in Test Construction Tasks in Test Test Evaluation

Questions to ask yourself in designing a Test What objectives will (should) I be testing? What types of items will be included in the test? How long will the test be in terms of time and number of items? How much will each objective be worth in terms of weighting and number of items?  Writing a test specification

Stages in Test Construction Order the following stages in the test construction. -Decide on the marks -Make a key -Decide what kind of test it is -Find appropriate text -Pilot the test -Write a list of what the test covers -Weigh the sections in the order of importance and timing -Write the marking scheme -Analyse the test and decide what can be kept/rejected -Review and revise the test and key -Write the questions -Think about length, layout and format -Write the instructions and examples

Stages in Test Construction Suggested order -Decide what kind of test it is -Write a list of what the test covers -Think about length, layout and format -Find appropriate text -Weigh the sections in the order of importance and timing -Write the questions -Write the instructions and examples -Decide on the marks -Make a key -Pilot the test -Write the marking scheme -Review and revise the test and key -Analyse the test and decide what can be kept/rejected Which stages can be repeated several times? Why?

Question Types vs Levels of Performance Knowledge Comprehension ApplicationAnalysis Synthesis Evaluation Multiple Choice (MC) True/False (TF) Matching Completion Short Answer … MC Short Answer Problems Essay … MC Short Answer Essay … Can you add more to the columns?

Multiple Choice Items Advantages:  Can measure the higher level mental processes (application, analysis, synthesis and evaluation)  Can cover a wide range of content  Efficiently and reliably scored  Different response alternatives can provide diagnostic feedback (item analysis) Disadvantages  Difficult to construct plausible alternative responses

Types of Multiple Choice Items Four Basic Types  Question Type  Incomplete Statement Type  Right Answer Type  Best Answer Type

1. The capital city of Canada is a. Vancouver a. Vancouver b. Montreal b. Montreal c. Toronto c. Toronto *d. Ottawa Terminology: Multiple Choice 1. Stem: presents the problem 2. Keyed Response: correct or best answer 3. Distracters: appear to be reasonable answers to the examinee who does not know the content 4. Options: include the distracters and the keyed response

Guidelines for Writing MC Place most of the subject matter in the Stem  ensures full statement of problem Avoid Negatively phrased Stems  students may miss the qualifier  use only when learning outcome requires this type of differentiation Ensure similarity among alternatives with regard to:  grammatical structure  length  mode of expression Grammatical errors provide unintentional clues to the answer When in doubt, students will select the longest alternative as the correct answer

Example: Length of Alternatives Neurotics are more likely than psychotics to a.be dangerous to society b.have delusional symptoms c.be dangerous to themselves *d.have insight into their own inappropriate behavior but nevertheless feel rather helpless in terms of dealing with their difficulties

Guidelines for Writing MC Randomly distribute answers across the alternative positions  inexperienced test writers emphasize “b” and “c” alternatives  do NOT use an interpretable order of keyed responses Response alternatives should not overlap, logically or numerically ordered Avoid “All of the above” and “None of the above” Avoid Negatively phrased Stems  students may miss the qualifier  use only when learning outcome requires this type of differentiation

What is the cost of an item that normally sells for 9.99 USD that is discounted 25%? Order Options Numerically Wrong A.5.00 B.7.50 C.2.50 D.6.66 Right A B C D. 7.50

What are the three important concerns in achievement testing? V=validity R=reliability E=efficiency O=objectivity Order Options Logically Wrong A.V, R, E B.O, V, R C.R, O, E D.E, O, V Right A. V, R, O B. V, R, E C. V, E, O D. R, E, O

True-False Test Items Best suited for testing 3 kinds of info.: Knowledge level learning Understanding of misconceptions When there are two logical responses Advantages:  Sample a large amount of learning per unit of student testing time Disadvantages:  Tends to be very easy  chance of guessing  Tends to be low in reliability

Tips for constructing True-False Items oProvide a “T” and “F” beside each statement and ask students to circle correct answer (students writing illegible letters). oInclude background, qualifications, and context "According to....,...." oParaphrase, and do not directly quote oEvery part of a true sentence must be "true"

Tips for constructing True-False Items - Avoid Unfamiliar vocabulary and concepts Double negatives* Specific determiners (always, never, only, etc.) = overgeneralization* long or complex sentences Two ideas in each statement*ideas in each statement Exact quantitative (two, three, four) language is better than qualitative (some, few, many) A pattern of answers

True / False negative words Poor:It was not unheard of for Henry VIII to close monasteries in England.T F Better:Henry VIII closed some monasteries in England. (T) “never”, “none”, “always”, and “all”.  Testwise students will recognize that there are few absolutes.

True / False Two ideas in one statement  Poor:Porpoises are able to communicate because they are mammals.T F  Better: Porpoises are mammals. T F Porpoises are able to communicate. T F

Test Evaluation List of criteria for a good test: -Test purpose -Test format -Test Content -Test Validity -Test Reliability -Test Practicality -Test washback

Test Evaluation Below is the comment note of an experienced teacher on a reading test for grade 10 students. Read and tell what criteria have been broken: C1: Questions 5-10 tend to test students’ grammar rather than reading comprehension. C2: Questions try to test the specialized knowledge of geography rather than language perception and use. C3: The checkbox for true/false questions is rather confusing since it is not in parallel with the statements.

Test Evaluation C4: The test seems to be a proficiency test than an achievement test. C5: Some open questions seek for a very long answer or many possible answers, then students find it hard to write them all and teachers find it hard to score. C6:The test gives students a feeling that they can get high marks if they focus on learning grammar and vocabulary rather than reading skills.

Summary Stages in Test Construction: Purpose  Plan  Items  Pilot  Analyse  Revise (Pilot, Analysis and Revision: repeated some time) Tasks in Test: Multiple Choice, Gap-filling, True/False, Open question, etc. Test Evaluation: purpose, format, content, validity, reliability, practicality, and washback