1. FUNDAMENTAL PROBLEMS WITH QUESTIONS Marcus Burnstein ASCRS, Boston, May 30, 2015

2. Q: What is the fundamental problem with questions? A: QUESTION-WRITERS untrained ^

3. THE PROBLEM: IT SUPERFICIALLY LOOKS EASY, BUT…

4. PROBLEMS WITH THE FUNDAMENTALS Question-writers haven’t got a prayer of writing decent MCQs without i. Taking this course ii. Studying MCQ preparation guidelines iii. Peer-review (with feedback) of their MCQs iv. Repeating steps ii. & iii. on a regular basis

5. IF YOU DON’T BELIEVE ME… FROM THE EDUCATION LITERATURE: “Question writing is a creative art which is mastered through extensive and critically supervised practice.” (Rodrigues, 1997) “Effective question writers are trained, not born… without specific training writers tend to produce poor quality, flawed, low cognitive level questions that test unimportant content.” (Downing & Haladyna, 1997) “The quality of questions can be significantly improved by formal training.” (Josefowicz, 2002)

6. So, congratulations (or condolences) for being here… this is an important step on the arduous but satisfying road to writing a potentially decent question

7. National Board of Medical Examiners Other excellent resources http://download.usmle.org/IWTutorial/intro.htm

8. FUNDAMENTAL PROBLEMS WITH QUESTIONS (that Dr.s Mahmoud, McNevin and Ault may not have already reviewed) COGNITIVE LEVEL TRIVIA BEST ANSWER FILLER DISTRACTORS QUESTION-WRITERS CHECKLIST FAILURE

9. PROBLEM #1: COGNITIVE LEVEL TOO MUCH NOT ENOUGH

10. HIERARCHY IN THE COGNITIVE DOMAIN: KNOWLEDGE IN ACTION Knowledge Understanding & Judgment Problem- solving

11. PROBLEM #2: TRIVIA

12. SOME EXAMPLES OF MCQs AT DIFFERENT COGNITIVE LEVELS

13.  None of the MCQ examples have been subjected to peer-review

14. PURE RECALL AND A BIT TRIVIAL Key concept: LGV is caused by C. trachomatis Ref: Burnstein MJ, et al. Burnie’s J Exper Surg 2015; 1: 2-3 Lymphogranuloma venereum is caused by which of the following microorganisms? A. Chlamydia trachomatis B. Haemophilus ducreyi C. Mycobacterium avium intracellulare D. Donovania granulomatis

15. A BIT OF ‘KNOWLEDGE APPLICATION’ AND A BIT LESS TRIVIAL Key concept: LGV has a characteristic clinical presentation Ref: Burnstein MJ, et al. Burnie’s J Exper Surg 2015; 1: 2-3 A 32 year old man with a history of anoreceptive intercourse complains of 12 weeks of anorectal pain and discharge. There is inguinal adenopathy and on proctoscopy there is mucosal inflammation with ulceration and stenosis. Which of the following is the most likely diagnosis? A. Gonococcal proctitis B. LGV proctitis C. Syphillitic proctitis D. Herpes proctitis

16. A BIT OF KNOWLEDGE APPLICATION, AND A BIT LESS TRIVIAL Key concept: LGV infection is treated with doxycycline Ref: Burnstein MJ, et al. Burnie’s J Exper Surg 2015; 1: 2-3 A 32 year old man with a history of anoreceptive intercourse complains of 12 weeks of anorectal pain and discharge. There is inguinal adenopathy and on proctoscopy there is mucosal inflammation with ulceration and stenosis. Which of the following is the most likely diagnosis? A. Penicillin B. Tetracycline C. Acyclovir D. Doxycycline

17. SOME PROBLEM-SOLVING Key concept: Transverse colon/ascending colon can be anastomosed to the rectum by clockwise rotation of the mobilized colon Ref: Burnstein MJ, et al. Burnie’s J Exper Surg 2015; 1: 20-24 A 65 year old man is undergoing resection of a cancer at 12 cm form the anal verge. During left colon mobilization, the inferior mesenteric artery is divided at its origin and the colon becomes ischemic to the level of the mid-transverse colon. Which of the following is the best treatment option? A. End ileostomy B. Ileorectal anastomosis C. Colorectal anastomosis D. Ileal pouch anal anastomosis

18. PROBLEM #3: THE ‘BEST’ ANSWER MUST BE THE BEST ANSWER… WITHOUT CONTROVERSY AMONGST EXPERTS Controversy and options… consider for Oral Exam question

19. SOME EXAMPLES OF TOO MUCH CONTROVERSY …

20. PROBLEM #3: THE ‘BEST’ ANSWER MUST BE THE BEST ANSWER… WIHTOUT CONTROVERSY AMONGST EXPERTS Key concept: Recurrent rectal prolapse after rectopexy is best treated by ??? Ref: Burnstein MJ, et al. Burnie’s J Exper Surg 2015; 1: 4-5 A 72 year old woman had laparoscopic suture rectopexy for prolapse 2 years ago. There is now a 4 cm prolapse. Which of the following is the treatment of choice? A. Delorme procedure B. Altemeier procedure C. Ventral mesh rectopexy D. Suture rectopexy

21. FUNDAMENTAL PROBLEM #3: THE ‘BEST’ ANSWER MUST BE THE BEST ANSWER… WIHTOUT CONTROVERSY AMONGST EXPERTS Key concept: Stage 3 mid and distal rectal cancer should have nCRT Ref: Burnstein MJ, et al. Burnie’s J Exper Surg 2015; 1: 6-7 A 53 year old man has a 3 cm adenocarcinoma at 7 cm on the posterior rectal wall. MRI and CT scan indicate stage 3 (T3N1M0) with tumor invading 3 mm into the mesorectum and 3 mm from the mesorectal fascia. Which of the following is the best next step? A. Induction chemotherapy B. Short course radiotherapy C. Neo-adjuvant chemo-radiotherapy D. Anterior resection

22. The distractors have to be plausible to the non-expert… they don’t have to be totally wrong PROBLEM #4: FILLER DISTRACTORS RIDICULOUS FILLERS LEAST CORRECT MOST CORRECT A C DB

23. PROBLEM #4: FILLERS Key concept: Acute, severely symptomatic thrombosed external hemorrhoid is best treated by excision Ref: Burnstein MJ, et al. Burnie’s J Exper Surg 2015; 1: 9-10 A 24 year old woman presents with a 24 hr history of severe anal pain and a very tender lump at the right lateral anal verge (Fig. 1). Which of the following is the best treatment? A. Sitz baths B. Incision C. Excision D. Botox Fig. 1 (prone position)

24. PROBLEM #5: NO CHECKLIST Focus on a key concept… of importance? Pass ‘the cover test’ Clearly a best answer Avoid ‘filler’ distractors Clinical vignette? Beyond recall?

25. IN ADDITION TO THE CHECKLIST… DOES IT LOOK GOOD? GOOD BAD A B C D A C D B

26. Best answer Clinical vignette TO OVERCOME THE FUNDAMENTAL PROBLEMS… Take a course Study guidelines, e.g. ABCRS and NBME Get feedback Rinse and repeat Apply the fundamentals Important concept Application of knowledge Pass the cover test Checklist to avoid technical flaws Stem must be a clear question Homogenous, plausible distractors

27. Thank you