2. Part #3

3. © 2014 Rollant Concepts, Inc.2

4. Assembling a Test #

5. © 2014 Rollant Concepts, Inc. Assembling a Test First page = easier questions Test should progress from easier to more difficult (Nitko, 2004) #

6. © 2014 Rollant Concepts, Inc. Assembling a Test Unnecessary to group content topics #

7. © 2014 Rollant Concepts, Inc. Assembling a Test Avoid the correct answer to be the same letter for more than 3 or 4 questions in a row Vary the difficulty level within a content area #

8. © 2014 Rollant Concepts, Inc. Assembling a Test Edit w/ having a colleague proofread for punctuation, grammar, reading level, ‘makes sense’, clarity and content Ensure stem and options are on same page #

9. © 2014 Rollant Concepts, Inc. Assembling a Test Print on one side of the page – if two sided use 2 staples like a book Provide margins for making notes – these can be reviewed for areas that students may have ‘fuzzy’ thinking #

10. © 2014 Rollant Concepts, Inc. Are you tense? #

11. 10© 2014 Rollant Concepts, Inc.

12. 11© 2014 Rollant Concepts, Inc.

13. 12© 2014 Rollant Concepts, Inc.

14. Test Construction Curriculum = 100, 200, 300, 400 Within a Course –Formative Evaluation = –Summative Evaluation = –Exam Matrix for curriculum/course –Exam Blueprint Revisions –Verbs for Objectives –Types of questions = structure/level Decision Points © 2014 Rollant Concepts, Inc.13

15. Test Analysis Thought for the day… “There are three kinds of lies: lies, damned lies and statistics.” »Mark Twain Statistical findings in isolation are meaningless. Informed interpretation is the goal without distortion. 14© 2014 Rollant Concepts, Inc.

16. Where to Analyze? #1 Exam – overall [KR-20] #2 Items for Correct Responses #3 Items for discrimination #4 “Option A” as incorrect select #5 OPTIONAL – Option discrimination 15© 2014 Rollant Concepts, Inc.

17. Summary – Steps to Analysis 1.Exam – desired - KR-20 of 0.65 to 0.85 2. & 3. Items** – mark & review ? that have  P-value of 0.80  Point Biserial Index [PBI] If negative check first [may be keyed incorrectly] Then if time review items with <0.09 **Make decisions to reject ? or accept more than one answer 4. Option Choices** – mark & review PBI if  Correct option [-] negative  Incorrect option [+] positive 16© 2014 Rollant Concepts, Inc.

18. What to Analyze – the EXAM? STEP #1 Look the Exam – overall –Reliability coefficient [KR – 20] –desired - KR-20 of 0.65 to 0.85 –OPTIONAL: Score distribution – histogram Review raw scores and percentages Positive or negative sided = red flag 17© 2014 Rollant Concepts, Inc.

19. 18 Step #1

20. What to Analyze - the ITEM? STEP #2 Look at the Items – Are the questions acceptable? –Item difficulty level [P-value = # of correct responses to an item] –P-value of 0.80 STEP #3 –Discrimination Item discrimination ratio [IDR] = % of upper 27% minus the lower 27% who answered item correctly Point Biserial Index [PBI] = considers the total group variance 19© 2014 Rollant Concepts, Inc.

21. 20 Step #2 Step #3

22. © 2014 Rollant Concepts, Inc.21 Step #3

23. What to Analyze – in Options? STEP #4 Options – Which were selected by test-taker? Distractor analysis –Response frequencies & patterns = # number of students who selected each option –Who selects option “A” as a pattern for incorrect items? 22© 2014 Rollant Concepts, Inc.

24. 23 Step #4

25. What to Analyze – in Options? STEP #5 Options – Which were selected by test-taker? Distractor analysis - Point biserial for options = indicated the average performance of those selecting the option  Positive [+] correlates w/ correct option  Negative [-] correlates w/ other options 24© 2014 Rollant Concepts, Inc.

26. 25 Step #5

27. Summary – 3 Steps to Analysis 1.Exam – desired - KR-20 of 0.65 to 0.85 2.Items** – mark & review ? that have  P-value of 0.80  Point Biserial Index [PBI] negative first [check for being correctly keyed] Then <0.09 if time 3.Options** – mark & review PBI if  Correct option [-] negative  Incorrect option [+] positive **Make decisions to reject ? or accept more than one answer 26© 2014 Rollant Concepts, Inc.