Written Exam Assessing knows and knows how Departemen Obstetri dan Ginekologi FKUI-RSCM

Three levels of test 1. Knowledge (recall or recognition of specific information) 2. Combined comprehension and application 3. Problem solving (transferring existing knowledge and skills to new situations)

Guidelines for Basic Item Content Clinical vignettes Focus on key concepts and principles that are essential information Test material relevant to learning in clinic Avoid items that only require recall of isolated facts Avoid esoteric or interesting topics that are not essential

Tools Case vignettes should include: age, gender site of care presenting complaint Duration patient history physical findings +/- diagnostic studies +/- initial treatment Stems should: not be completely based on real patients include reference material when it would be realistic in practice not use the patient’s or doctor’s own words not include patients who lie

Multiple Choice Questions (MCQs)

Introduction MCQs can assess taxonomically higher- order cognitive processing such as interpretation, synthesis and application of knowledge rather than the test of recall of isolated facts. Should not be used as a sole assessment method in summative examinations, but alongside other test forms

Advantage Can assess a broad range of learner knowledge in a short period of time (time effective) Appropriately constructed MCQs result in objective testing that can measure knowledge,comprehension application and analysis reliable, discriminatory, reproducible and cost- effective Disadvantage they test recognition (choosing an answer) rather than recall (constructing an answer) allow for guessing difficult and time- consuming to construct

Writing MCQs Begins with identifying the most important information or skill that is to be learned Focus on important and relevant content, avoiding testing the knowledge of medical trivia Controversial items should be avoided

Components of MCQs The item : entire unit and consists of a stem and several options. The stem : the question, statement or lead-in to the question. options or choices : the possible answers keyed response: The correct option foils or distractors : The incorrect options

Parts of MCQs “item” A 62-year-old woman comes to the physician because of bleeding from the vagina. She states that her last menstrual period came 11 years ago and that she has had no bleeding since that time. She has hypertension and type 2 diabetes mellitus. Examination shows a mildly obese woman in no apparent distress. Pelvic examination is unremarkable. An endometrial biopsy is performed that shows grade I endometrial adenocarcinoma Which of the following is the most appropriate next step in management? a. Chemotherapy b. Cone biopsy c. Dilation and curettage d. Hysteroscopy e. Hysterectomy * vignette Lead in distractors Option Stem answer

Technical Item Flaws Issues Related toTestwiseness Grammatical cues Logical cues Absolute terms - such as always or never Long correct answer Word repeats Convergence strategy

Options are long, complicated, or double Numeric data are not stated consistently Terms in the options are vague (eg, “rarely,” “usually”) Language in the options is not parallel “None of the above” is used as an option Stems are tricky or unnecessarily complicated The answer to an item is “hinged” to the answer of a related item Issues Related to Irrelevant Difficulty

Make sure the item can be answered without looking at the options, (the options are 100% true / false). Avoid technical Item Flaws Write distractors that are plausible and the same relative length as the answer. General Guidelines for Item Construction

General…. Avoid using absolutes such as always, never, and all in the options; also avoid using vague terms such as usually and frequently. Avoid negatively phrased items. If you must use a negative stem, use only short (preferably single word) options. And most important of all: Focus on important concepts; don’t waste time testing trivial facts

MCQ writing step DoDon’t Before writing  assess learning outcomes or important concepts  Identify the cognitive level at which the MCQ should be pitched  Decide on the topic and content area  assess trivial, insignificant facts Writing the stem  a common clinical case  Include as much information as required to arrive at the correct answer, i.e. a long stem (with short options)  synthesise for the candidate,  technical item flaws, such as  A word in the stem repeated in the option(s)  Tricky/complicated stems  Clues to the answer in the stem Writing the lead-in  clearly indicate how to answer the MCQ  preferably be a question  Refer back to the topic & content area, when constructing the lead-in  Use questions and avoid phrases e.g. Regarding epilepsy:  Avoid technical item flaws, such as:  Absolute terms, e.g. always, never  Frequency terms, e.g. rarely  Which of the following statements is correct?’ This type of lead-in may lead to heterogeneous options  Negative questions

MCQ writing step DoDon’t Checking the stem and lead-in  Lead-in and stem must give enough information to answer the MCQ, without/before reading the options  Both should be clear, precise and simple  create a ‘test within a test’. Writing the options  only one clear answer  Distractors clearly incorrect, but plausible  short and uncomplicated  All options homogeneous  List in a logical order  The positions of the correct option should vary with other MCQs  All options are of similar length  Use coherent, consistent terminology  technical item flaws, such as:  Related to testwiseness  Related to irrelevant difficulty After writing1.Does the MCQ assess an important concept? 2.Does the MCQ test factual recall of knowledge, application or evaluation? 3.Can the MCQ be answered by only reading the stem & lead-in? 4.Are all the options homogeneous? 5.Is the MCQ (stem, lead-in and options) devoid of technical item flaws?

EXTENDED MATCHING QUESTIONS

Extended Matching items are multiple-choice items organized into sets that use one list of options for all items in the set. A well-constructed Extended-Matching set includes four components (R-type): 1. a theme 2. an option list 3. a lead-in statement 4. at least two item stems

1. Identify the theme for the set. 2. Write the lead-in for the set. 3. Prepare the list of options. 4. Write the items. 5. Review the items. Check to make sure that there is only a single “best” answer for each question. Overview of the Steps for Writing Extended Matching Items

Example of an EMQs Theme: Bleeding per vagina Options A. CervicitisG. Endometrial cancer B. Atrophic vaginitisH. Endometrial polyp C. Cervical cancerI. Incomplete miscarriage D. Cervical polypJ. Threatened miscarriage E. Complete miscarriageK. Toxic-shock syndrome F. Ectopic pregnancy Instructions/Lead-in For each of the patients described below choose the SINGLE most appropriate diagnosis from the list of options above. Each optionmay be used once, more than once or not at all..

Item: 1.A 62-year-old woman comes to the physician because of bleeding from the vagina. She states that her last menstrual period came 11 years ago and that she has had no bleeding since that time. She has hypertension and type 2 diabetes mellitus. Examination shows a mildly obese woman in no apparent distress. Pelvic examination is unremarkable. Ultrasound revealed thickened endometrium 15 mm, with increase vascularization. 2.A 25 years old woman comes to emergency because of bleeding from vagina. She states that her last menstrual period came 2 months ago and she already did pack test and the result was positive. On vaginal examination the doctor found that the ostium was open and there was conception tissue in it.

Short Answer Questions

Introduction Short-Answer questions require a student to respond to a question by writing a word or short phrase. Questions can vary in expected student response- from one word to short sentence, and include forms : complete the sentence supply the missing line short descriptive or qualitative answers

Advantage Assess higher-order or critical thinking skills Provide authentic experience. easier to mark. Evaluate student thinking and reasoning Disadvantage Assess a limited sample of the range of content. Are difficult and time consuming to grade.

Procedures for setting and marking Make the questions precise Direct questions are better than incomplete statements. If a numerical answer is required, indicate the units and degree of precision required. Prepare a structured marking sheet Allocate marks or part-marks for acceptable answer(s). Be prepared to accept other equally acceptable answers, some of which you may not have predicted.

Example of short answer questions A 62-year-old woman comes to the physician because of bleeding from the vagina. She states that her last menstrual period came 11 years ago and that she has had no bleeding since that time. She has hypertension and type 2 diabetes mellitus. Examination shows a mildly obese woman in no apparent distress. Pelvic examination is unremarkable. Ultrasound revealed thickened endometrium 15 mm, with increase vascularization. What is the most likely diagnosis of this patient