Retrospective Chart Reviews: How to Review a Review Adam J. Singer, MD Professor and Vice Chairman for Research Department of Emergency Medicine Stony Brook University

Retrospective chart reviews  Generate original research 25% of all published studies  Hypothesis generating  Early observations  Disease description and natural course  May demonstrate associations  Quality audits  Derivation and validation of clinical prediction rules  Cost analysis

Chart reviews  Advantages Simple Quick Inexpensive For rare diseases  Disadvantages No cause-effect Incomplete/inaccurate/ conflicting data No study protocol

Process of transforming clinical record information into “hard” data  Occurrence of event Basic clinical data may be erroneous Clinicians vary in obtaining, interpreting, and recording data from histories, physicals, and tests  Select patients to be studied Patients may be highly selected limiting generalizability  Systematic errors more likely than random errors

 Assemble charts Charts may be missing Bias may be introduced if missing cases are death or more (or less) severe, or have better (or worse) outcomes  Locate desired information Documentation is often incomplete Important data may be missing

 Read note Note may be illegible or ambiguous Two or more notes may give conflicting information for some variables  Code information Coding information into categories and Likert scales requires interpretation, judgment and guess work Problems arise if information must be coded as “missing”, “negative”, or “unsure”

 Transfer data to computer database Mistakes may be made in the transcription process  Perform statistical analysis Statistical test are performed Quality of data is not evaluated Errors in extraction of data are forgotten

Strategies to improve accuracy of chart reviews  Training  Case selection  Definition and variables  Abstraction forms  Meetings  Monitoring  Blinding  Testing of inter-rater agreement

Important questions when reviewing a (chart) review  Is chart review appropriate for research question? Available charts representative of population of interest Charts must include pertinent data Sample size must be adequate  Enough with outcome of interest  Only 10% of ED chart reviews include sample size calculation

Is investigator bias transparent?  Identify any financial or philosophical COI  Data collection forms as definitions should be submitted as an appendix

Study and target population  Representative sample  Random sample  External validity  Study setting should be typical of population of interest  All charts included and have equal chance of selection Multiple methods of chart identification helpful

Collected variables  Possible conflicting entries  Inaccuracies in documentation  Errors in reading, interpreting, coding and transcribing data  One study found 30% of care not documented and 19% of care recorded not given  Variables should be well defined a priori  Coding rules should be provided

Systematic data collection  Use standardized data collection forms Organized similar to data in chart  Enter directly into computer to minimize transcription errors  Describe coding options  How was missing data handled?  Pilot testing of data collection form

Missing and conflicting data  Missing data creates selection bias  Conflicting data common leading to incorrect conclusions  Determine whether amount of missing data threatens validity  Sensitivity analysis allows readers to estimate effects of missing data Assume all missing data positive or negative  Consider imputations

Abstractor bias  Abstractors should be blinded to objectives and hypothesis  If abstractors know desirable value they may overlook contradictory information or search diligently for it  Abstractors often the investigators  Only 4% of studies adhere to this principle  Investigators should select and train abstractors who are blinded to objectives and hypotheses

Abstractor training  Some chart reviewers non-medical  May fail to recognize medical jargon or may misinterpret data  May not know where in chart to find data  Difficulty resolving internal discrepancies  Less than 20% of studies describe training  Investigators should use abstractors with sufficient training Describe background and training

Abstractor monitoring  Decrement in consistency over time Especially in long term studies  Periodic comparison of forms with actual charts  Multiple meetings with abstractors required to retrain and resolve disputes  Only 9% of chart reviews have documented monitoring

Abstractor inter-rater reliability  Ideally multiple abstractors would review all charts  Seldom done  Alternative to multiple abstractors is to establish inter-rater reliability  May need to be periodically reassessed

How much is enough?  Both raw agreement and kappa Cohen’s kappa for binary data Weighted kappa with multiple categories Intra-class correlation for continuous data  Level of agreement may differ by variable (for death should be 1.0)  Most important for the most important confounders and outcomes  10% sample may not be adequate if few or no yes answers

What should investigators do  Consider other data sources Large and representative sources NHAMCS  Do not overlook methods section  Be cautious about making conclusions  Adherence to reporting standards doesn’t mean review should be published  Era of electronic records

Gilbert et al. Ann Emerg Med 1996;27:305  986 original articles from 3 peer- reviewed EM journals reviewed, 244 were chart reviews

Adherence to methodological standards Percent95% CI Abstractor training Selection criteria Variables defined Standard abstraction forms Performance monitored Abstractor blinding Interrater reliability mentioned Interrater agreement tested0.40-2

Reassessing methods of medical review studies in EM research (2003) Abstractor training18 Case selection96 Variable definition77 Abstraction forms27 Performance monitored9 Blind to hypothesis4 Agreement Adherence Worster et al. Ann Emerg Med 2005;45:448

The quality of medical record review studies in the international EM literature Badcock Ann Emerg Med 2005;45:444 All Clear hypothesis Training Selection criteria Standard forms Defined variables Monitoring Blinding73814 Inter-rater agreement Sample size calculation109 14

RECENT CHART REVIEW: Ann EM 2016

METHODS