1. TEXAS TECH UNIVERSITY HEALTH SCIENCES CENTER SCHOOL OF PHARMACY KRYSTAL K. HAASE, PHARM.D., FCCP, BCPS ASSOCIATE PROFESSOR BEYOND MULTIPLE CHOICE QUESTIONS

2. OBJECTIVES Describe the pro’s and con’s of using different question formats other than multiple choice. Identify and resolve common problems when constructing open-ended questions. Develop standardized grading procedures for open-ended questions. Discuss exam length and other challenges when using open-ended question formats.

3. ASKING THE RIGHT QUESTION

4. REVIEW Bloom’s Evaluation Synthesis Analysis Application Comprehension Knowledge Low High

5. QUESTION TYPES Selection Response Construction Response RecognitionRecall

6. QUESTION TYPES Selection Response True / False Matching Multiple Choice Construction Response Fill in the Blank Short AnswerEssay

7. TRUE / FALSE Pro’s easy to write easy to score Con’s limited ability to assess mastery high probability of guessing Best Use: Dichotomous, factual info Bloom’s Evaluation Synthesis Analysis Application Comprehension Knowledge

8. MATCHING Pro’s Can assess a lot of info in a confined space Fairly low probability of guessing Con’s Assess recognition not recall Best Use: Knowledge recall Add additional “distractor” items to increase rigor Bloom’s Evaluation Synthesis Analysis Application Comprehension Knowledge

9. MULTIPLE CHOICE Pro’s Very versatile Easy to score Con’s More challenging to write Assess recognition over recall Best Use: Factual, conceptual, or procedural information Bloom’s Evaluation Synthesis Analysis Application Comprehension Knowledge

10. SHORT-ANSWER Pro’s Assess unassisted recall Relatively easy to write Con’s Only useful if you can give a short answer. Must be worded carefully to avoid scoring problems Best Use: Assessing information that you expect to be memorized Bloom’s Evaluation Synthesis Analysis Application Comprehension Knowledge

11. ESSAY Pro’s Can test higher complex objectives Can test process / reasoning Realistic tasks Con’s Take longer to answer Hard to grade fairly, consistently Take longer to grade Best Use: Assessing highest level objectives Bloom’s Evaluation Synthesis Analysis Application Comprehension Knowledge www.siop.org/workplace/employment%20testing/testformats.aspx

12. SUMMARY Each question type has pro’s and con’s Question type should be guided by the learning objective to be assessed. Limited options for assessing higher taxonomy

13. CONSTRUCTING OPEN- ENDED QUESTIONS

14. WELL-DEVELOPED ESSAY QUESTIONS Mirror well-defined learning objectives Assess most appropriate content types Require content recall, evaluation, and reasoning Are clearly written Provide boundaries Have well-defined grading criteria http://testing.byu.edu/info/handbooks/WritingEffectiveEssayQuestions.pdf

15. APPROPRIATE CONTENT Content that justifies high-level mastery Construction, higher-order taxonomy Analysis (analyze, compare, contrast, interpret) Evaluation (evaluate, explain, justify) Synthesis (develop, construct, modify) Complex, multi-step thought processes Simulation of real-world processes “given a patient-case scenario” If content / processes can be assessed by methods other than essay questions, they probably should.

16. KEY ELEMENTS An ideal essay question requires students to: Recall facts Make an evaluative judgment or develop a novel solution Explain reasoning behind response The question should include: Task Problem situation

17. WRITING FOR CLARITY Ensure your question requires higher-order thinking Make sure the task is defined and focused Make sure the problem situation includes adequate detail

18. SETTING BOUNDARIES Increasing structure prevents grading problems bluffing Avoid indeterminate questions Students can redefine and answer with info the know well Give time / space limits Establish rules for answers

19. ESTABLISHING GRADING CRITERIA Must have specific criteria for grading identified a priori Create a model answer Assign point values Peer review Identify essentials in response Determine whether partial credit is allowable and how will be awarded. Grade blinded

20. DIFFERENT GRADING APPROACHES Comprehension / Understanding Screen responses for key elements Assign points for each element present Scores for presence of content only (potential bluffing) Reasoning / Complex Processes Must assess complete response (time consuming) Consider a rubric approach to limit subjectivity Problem – Solution Encourage requiring students to show work Can grade multiple steps in process

21. GRADING TIPS Grading scheme should be easily interpreted by other graders Self-explanatory Point values that can easily be tallied When multiple graders, encourage frequent communication and comparison of results Before grading, screen a sample of responses for consistency

22. OTHER ISSUES Exam length Addressing unidentified “correct” answers Partial credit or all-or-none Practice examples Multiple examples Should not be the same questions as on exam