1. Medical Record Reviews – The Rules of the Road David H. Rubin, MD Department of Pediatrics

2. INTRODUCTION Any study that uses pre-recorded patient focused data as the primary source of information in a research studyAny study that uses pre-recorded patient focused data as the primary source of information in a research study Physician, nurses notes Physician, nurses notes Ambulance call reports Ambulance call reports Diagnostic tests Diagnostic tests Clinic, administrative, government records Clinic, administrative, government records Computerized databases Computerized databases

3. WHY SELECT THIS DESIGN? Allows research to address issues that cannot be addressed with prospective studiesAllows research to address issues that cannot be addressed with prospective studies Effect of harmful exposures (no randomization possible) Effect of harmful exposures (no randomization possible) Effect of potentially beneficial exposures Effect of potentially beneficial exposures Occurrence of rare events Occurrence of rare events Studies of patterns of disease or behavior Studies of patterns of disease or behavior Quality assurance studies Quality assurance studies Studies where cases may be shared (trauma database) Studies where cases may be shared (trauma database) Pilot studies for prospective studies Pilot studies for prospective studies

4. DATA QUALITY “Free form” quality of medical records may increase missing and/or erroneous data“Free form” quality of medical records may increase missing and/or erroneous data Handwriting may be illegible or uninterruptibleHandwriting may be illegible or uninterruptible May miss examining potential casesMay miss examining potential cases Computer vs paper recordsComputer vs paper records Data abstraction techniques require standardizationData abstraction techniques require standardization

5. MISSING OR CONFLICTING DATA Missing information may lead to nonresponse biasMissing information may lead to nonresponse bias Subjects with missing information may be very different from subjects without any missing information Subjects with missing information may be very different from subjects without any missing information Missing values managementMissing values management Case deletion – bias and reduced sample size are problems Case deletion – bias and reduced sample size are problems Case insertion – use what you have and compare missing/non missing cases for differences Case insertion – use what you have and compare missing/non missing cases for differences

6. SAMPLE SIZE Usually determined based on the summary measure and the size/width of the confidence interval desiredUsually determined based on the summary measure and the size/width of the confidence interval desired An interval with a greater CI (eg 99% CI v 95% CI) is wider and more likely includes the true population value An interval with a greater CI (eg 99% CI v 95% CI) is wider and more likely includes the true population value The width of the CI depends on sample size The width of the CI depends on sample size

7. SAMPLING Select all cases within a given time frameSelect all cases within a given time frame For nonconsecutive sampling it is best to choose probability samplingFor nonconsecutive sampling it is best to choose probability sampling Provides equal opportunity for each eligible case to be selected Provides equal opportunity for each eligible case to be selected Use random number generator Use random number generator Triage levelTriage level Incidental sampling – choosing most easily accessible casesIncidental sampling – choosing most easily accessible cases Systematic sampling – choosing every x th caseSystematic sampling – choosing every x th case

8. RELIABILITY Very importantVery important Any differences in data extraction by 2 different people?Any differences in data extraction by 2 different people? KappaKappa Value ranges from -1 (perfect disagreement) to 1 (perfect agreement) Value ranges from -1 (perfect disagreement) to 1 (perfect agreement) K = [observed agreement (%) – expected agreement (%) / [100% - expected agreement (%)] K = [observed agreement (%) – expected agreement (%) / [100% - expected agreement (%)] Try to achieve kappa of 0.6 or better (60% agreement) Try to achieve kappa of 0.6 or better (60% agreement)

9. MINIMUM REQUIREMENTS FOR MEDICAL RECORD REVIEWS (Lowenstein, 2005) 1)Explicit protocols for case selection/exclusion 2)Abstractor training 3)Precise definitions of key variables 4)Use of standardized abstraction and coding forms 5)Monitoring of abstractor performance 6)Blinding of abstractors to study hypothesis and patient groups 7)Testing of interrater reliability

10. QUALITY OF MEDICAL RECORD REVIEWS (Badcock, 2005) Observational study of medical record reviews published in several emergency medicine journalsObservational study of medical record reviews published in several emergency medicine journals 107 articles analyzed107 articles analyzed Clear aim reported in 93%Clear aim reported in 93% Standard abstraction forms: 51%Standard abstraction forms: 51% Interrater reliability: 25%Interrater reliability: 25% Ethics approval: 68%Ethics approval: 68% Sample size/power: 10%Sample size/power: 10%

11. MEDICAL RECORD STUDIES IN EM RESEARCH (Worster, 2005) From 1993 -1998 assessed medical record review studies in 6 EM journalsFrom 1993 -1998 assessed medical record review studies in 6 EM journals 79 (14%) medical review studies in 563 original research articles79 (14%) medical review studies in 563 original research articles Adherence to methodological criteriaAdherence to methodological criteria Sampling method: 99% Sampling method: 99% Abstractor blinding to hypothesis: 4% Abstractor blinding to hypothesis: 4% Interobserver agreement for the 12 criteria ranged from 57% - 95% Interobserver agreement for the 12 criteria ranged from 57% - 95%

12. ACCURACY IN HOSPITAL DISCHARGE SUMMARIES (Callen 2009) Retrospective analysis of 966 handwritten and 842 electronically generated discharge summaries in AustraliaRetrospective analysis of 966 handwritten and 842 electronically generated discharge summaries in Australia 12.1% of handwritten and 13.3% of electronic summaries contained errors12.1% of handwritten and 13.3% of electronic summaries contained errors Medication omission was biggest problemMedication omission was biggest problem NO difference in training levelNO difference in training level

13. MISSING INFORMATION IN ED CHARTS (Richason 2009) Examined case reports in 4 ED journals from 2000-2005 and used 11 reporting standardsExamined case reports in 4 ED journals from 2000-2005 and used 11 reporting standards 1,316 case reports identified1,316 case reports identified Poor reporting ofPoor reporting of Co-morbidities Co-morbidities Outcomes Outcomes Concurrent medications Concurrent medications

14. METHODOLOGY FOR RETROSPECTIVE REVIEWS IN CHILD PSYCHIATRY Conceive questionConceive question Literature reviewLiterature review Proposal methodsProposal methods Create data abstraction instrument and manualCreate data abstraction instrument and manual Sample sizeSample size Obtain IRB approvalObtain IRB approval Pilot studyPilot study

15. SAMPLE SIZE (Gearing 2006) Estimate 10 charts per variable (Sackett, 1991)Estimate 10 charts per variable (Sackett, 1991) Others estimate 5-7 charts/variableOthers estimate 5-7 charts/variable Convenience sampling – select cases over specific time periodConvenience sampling – select cases over specific time period Quota sampling – predetermined number sampledQuota sampling – predetermined number sampled Systematic sampling – every “nth” case chosenSystematic sampling – every “nth” case chosen

16. PRACTICAL ISSUES Check all possible CPT codes for diagnosis or procedure codeCheck all possible CPT codes for diagnosis or procedure code Febrile seizure may have been coded as seizure Febrile seizure may have been coded as seizure Gastroenteritis may have been coded as viral syndrome Gastroenteritis may have been coded as viral syndrome Pilot your Data Abstraction FormPilot your Data Abstraction Form Create detailed “Codebook” for your studyCreate detailed “Codebook” for your study Especially critical if > 1 researcher on study Especially critical if > 1 researcher on study

17. REFERENCES Worster A, Haines T. Advanced statistics: understanding medical record review (MRR) studies. Acad Emerg Med 2004;11:187-192.Worster A, Haines T. Advanced statistics: understanding medical record review (MRR) studies. Acad Emerg Med 2004;11:187-192. Lowenstein SR. Medical record reviews in emergency medicine: the blessing and the cure. Annals Emerg Med April 2005;45(4):452-455.Lowenstein SR. Medical record reviews in emergency medicine: the blessing and the cure. Annals Emerg Med April 2005;45(4):452-455. Babcock D et al. The quality of medical record review studies in the international emergency medicine literature. 2005;45(4):444-447.Babcock D et al. The quality of medical record review studies in the international emergency medicine literature. 2005;45(4):444-447. Worster A. et al. Reassessing the methods of medical record review studies 2005;45:448-451.Worster A. et al. Reassessing the methods of medical record review studies 2005;45:448-451. Gearing et al. Methodology for Retrospective chart review in child adolescent psychiatry. J Can Acad Child Adoles Psychiatry 15:3:2006Gearing et al. Methodology for Retrospective chart review in child adolescent psychiatry. J Can Acad Child Adoles Psychiatry 15:3:2006

18. REFERENCES Callen JJ, McIntosh. Accuracy of medication in hospital discharge summaries. Int J. Med Inform. 2009;Oct 1Callen JJ, McIntosh. Accuracy of medication in hospital discharge summaries. Int J. Med Inform. 2009;Oct 1 Richardson TP et al. Case reports describing treatrments in the ED. BMC Emerg Med.2009. June 15:9:10.Richardson TP et al. Case reports describing treatrments in the ED. BMC Emerg Med.2009. June 15:9:10.