Evaluating Residents’ Medical Knowledge R. J. Bartlett, MLS | AHIP Medical Librarian, Texas Medical Center, Houston 77030 Abstract Overall Test Performance by Residency Year and Academic Year Summary The difference between the pre-test and post-test scores was not statistically significant using a two-tailed paired t-test. Because by their very nature these analyses have no control group, they tend to have lower validities, and so any inferences with regard to the cause of any changes in the class’ performance require caution. ¶ Efficacy of teaching by growth year and subject matter needs to be quantified. The teaching faculty of this three-year ACGME program wanted granular detail of the successes and failures of their pedagogy. ¶ Pre- and Post-Academic year tests were taken by the residents over two academic years (N=45), the differences were evaluated using a two-tailed paired t-test. ¶ The null hypothesis was not disproved; in all classes and years p > 0.05 ¶ The tests and their administration need analysis and modification for any meaningful use. Conclusions and Suggestions Failing to show statistical significance may have resulted from many causes or combinations of causes. No feedback was given to the residents regarding their performance either pre or post. Nor was there any consequence to level of performance, either carrot or stick. Because this was an in-house developed test, it might help to judge it by incorporating results from an already-validated test. One that was suggested was the In-Training Exam (ITE) given to residents annually each October. This was not practical to do ex post facto because of the different division of subject matter. A revision of the three tests done in accord with the ITE might prove of value in evaluating the teaching faculty. Introduction & Methods The teaching faculty of the program created three residency-year-specific tests covering the subjects included in each growth year. R-1 consisted of internal medicine (IM), surgery, geriatrics, emergency, behavioral, obstetrics, out-patient pediatrics. R-2 consisted of IM, neurology, gynecology, cardiology, orthopædics, in-patient pediatrics, ICU, dermatology. R-3 consisted of IM, surgery, geriatrics, urology, community medicine, sports medicine, obstetrics. There were 10-questions for each subject area, hence 70-80-70 questions for R-1s. -2s, 3s, respectively, a timed test at one-minute per question. Acknowledgements Many thanks to Dr Raul G. López Valle (author of Lo esencial de Bioestadística para médicos) for guidance on choosing an appropriate test and the running of the Excel data analysis tool.