1. How to Avoid Common Pitfalls in Writing STEM Tests Marcia Kastner, Ph.D. Former MCAS Math Assessment Lead Massachusetts Department of Elementary and Secondary Education Copyright © 2011 by Marcia Kastner

2. Outline Why are tests important? Definition of terms Examples of flaws in math questions How to fix the flawed questions 2 Copyright © 2011 by Marcia Kastner

3. Why Are Tests Important? Used for accountability Provide information on what students know – and don’t know! For targeting instruction (from deconstructing test results) 3 Copyright © 2011 by Marcia Kastner

4. Definition of Terms Valid test question: measures what it was designed to measure Pitfall: flaw in a test question that prevents it from being valid Parts of a multiple-choice question: –Options: answer choices –Key: correct option –Distractors: incorrect options 4 Copyright © 2011 by Marcia Kastner

5. Math Testing Book 5 Copyright © 2011 by Marcia Kastner TESTING THE TEST How to Recognize When Math Tests Are Flawed How to Fix Them Why We Should Care (free books available after the presentation) Presentation is based on pitfalls described in my book: MarciaKastner.com

6. PITFALL: Right Answer for Wrong Reason (RAWR) Example 1: What is the median of the numbers below? 6, 2, 3, 1, 3 (Answer: 3) (Not valid because: mean = 3; mode = 3; middle number in unordered list = 3) (median = middle number in ordered list: 1, 2, 3, 3, 6) 6 Copyright © 2011 by Marcia Kastner

7. How to Fix Example 1 Example 1 Revised: What is the median of the numbers below? 7, 2, 6, 2, 3 (Answer: 3) Change some of the numbers. (mean = 4; mode = 2; middle number in unordered list = 6) (ordered list: 2, 2, 3, 6, 7) 7 Copyright © 2011 by Marcia Kastner

8. PITFALLS: Too Many Steps, Too Many Concepts, Option Sticks Out 8 Copyright © 2011 by Marcia Kastner Example 2: (from New Eng. Common Assessment Program [NECAP], 2007) Renata is a sales representative for a printer company. She sells two models of printers – Model P and Model Q. Last month she sold a total of 120 printers. The ratio of Model P printers sold to Model Q printers sold was 3:5. If Renata is paid a $25 commission for every Model P printer sold and a $20 commission for every Model Q printer sold, what was her total commission last month? A. $1480 B. $2475 C. $2625 [key: 25(45) + 20(75)] D. $2760 (Not valid because: requires approx. 14 steps)

9. Pitfalls in Example 2 Too many steps: approx. 14 steps Too many concepts: –Define variables –Model the bullet points as linear equations –Model the word problem as a system of linear equations –Solve the system of linear equations –Compute total commission One option sticks out: option A is different Copyright © 2011 by Marcia Kastner 9

10. How to Fix Example 2 Break up into multiple parts. Give partial credit for correct work shown. Possible parts: a)[write first bullet as equation] b)[write second bullet as equation] c)[determine amount of each model] d)[calculate total commission] 10 Copyright © 2011 by Marcia Kastner

11. PITFALLS: RAWR, Not Testing Common Mistakes Example 3: (from New York State Testing Program, 2010) Simplify the expression below. 7 2 – 9 + 1 3 A.37 B.39 C.41 D.43 (Answer: C) 11 Copyright 2011 by Marcia Kastner

12. Pitfalls in Example 3 7 2 – 9 + 1 3 RAWR: 1 3 = 1 = 1 0 = 1 1 = 1 2 = etc. Distractor logic: A.37[49 – (9 + 3): PEMDAS and cube errors] B.39[49 – (9 + 1): PEMDAS error] C.41*[49 – 9 + 1: key] D.43[49 – 9 + 3: cube error] Common mistakes not tested by distractors: 1. square error: thinking 7 2 = 7 x 2 2. perform operations left-to-right 12 Copyright 2011 by Marcia Kastner

13. How to Fix Example 3 Example 3 Revised: What is the value of the expression below? 7 2 – 9 + 2 3 A.11[14 – 9 + 6: exponent errors] B.21[(14 – 9 + 2) x 3: PEMDAS & exponent errors] C.32[49 – (9 + 8): PEMDAS error] D.48[49 – 9 + 8: key] (Answer: D) Change the expression and distractors. (testing common mistakes) 13 Copyright 2011 by Marcia Kastner

14. PITFALL: Implausible Distractors Example 4 (same as Example 1 Revised but as a multiple- choice question): What is the median of the numbers below? 7, 2, 6, 2, 3 A. 3 B. 6 C. 9 D.12 (Answer: A) (Not valid because: cannot determine how errors made) 14 Copyright © 2011 by Marcia Kastner

15. How to Fix Example 4 Example 4 Revised: What is the median of the numbers below? 7, 2, 6, 2, 3 A.2[mode] B.3[key: median] C.4[mean] D.6[middle number of unordered list] (Answer: B) Change the distractors. (can determine how errors made) 15 Copyright © 2011 by Marcia Kastner

16. Your Chance to TEST THE TEST Find the pitfalls in the following examples. 16 Copyright © 2011 by Marcia Kastner

17. What Is the Pitfall in Example 5? Calculators Allowed: All types Example 5 (same as Example 1 Revised except that a calculator is allowed): What is the median of the numbers below? 7, 2, 6, 2, 3 (Answer: 3) (Pitfall: Calculator, not student, solves the question. Not valid because: graphing calculators determine median. ) 17 Copyright © 2011 by Marcia Kastner

18. How to Fix Example 5 Calculators Allowed: All types except graphing calculators, OR no calculators allowed Do not allow graphing calculators. 18 Copyright © 2011 by Marcia Kastner

19. What Are the Pitfalls in Example 6? Example 6: (from New England Common Assessment Program [NECAP], 2010) Courtney walks three laps around a ¼-mile track. How many feet does she walk? [1 mi. = 5280 ft.] A. 440 ft.[1/3 x ¼ x 5280] B.1320 ft.[¼ x 5280] C.3960 ft.[key: 3/4 x 5280] D.7040 ft.[4/3 x 5280] (Answer: C) 19 Copyright © 2011 by Marcia Kastner

20. Pitfalls in Example 6 Confusing & imprecise language: “laps around a ¼-mile track” (Particularly disadvantages ELL and SPED students) Biased context: track Too many concepts –Fractions –Unit conversion 20 Copyright © 2011 by Marcia Kastner

21. How to Fix Example 6 Break up into multiple parts & allow partial credit. Add diagram showing oval track. Show how distance around is ¼ mile. Eliminate or explain word “lap.” Bold the word “three” or replace it with “3.” 21 Copyright © 2011 by Marcia Kastner

22. Summary Tests must be valid –Scrutinize test questions for pitfalls –Revise questions to remove pitfalls Diagnose students’ incorrect answers to see where they went wrong Focus instruction on areas that need improvement 22 Copyright © 2011 by Marcia Kastner