© 2003 By Default! A Free sample background from Slide 1 Strengths & Weaknesses of a Pre-Post Controlled Randomized Trial Michael E. Matheny, MD MS Decision Systems Group Brigham & Women’s Hospital, Boston, MA

© 2003 By Default! A Free sample background from Slide 2 Trial Design

© 2003 By Default! A Free sample background from Slide 3 Trial Design

© 2003 By Default! A Free sample background from Slide 4 Pre-Post Randomized Design Strengths Allows the study design to account for changes in measured outcomes by unmeasured factors over time Allows the study design to account for changes in measured outcomes by unmeasured factors over time Provides a single p value for the effect of the intervention that incorporates all of the data Provides a single p value for the effect of the intervention that incorporates all of the data Used primarily for: Used primarily for: –Studies that occur over a longer time period –Studies in an environment where other changes taking place could affect the measured outcomes

© 2003 By Default! A Free sample background from Slide 5 Pre-Post Randomized Design Weaknesses Requires allocation of significant time and resources prior to introduction of the intervention Requires allocation of significant time and resources prior to introduction of the intervention –Doubles the cost of data collection Very little power gained from “doubling” the sample size Very little power gained from “doubling” the sample size –calculations are still based on two arm design Unable to provide a single odds ratio result for the intervention Unable to provide a single odds ratio result for the intervention

© 2003 By Default! A Free sample background from Slide 6 Data Analysis SAS Code

© 2003 By Default! A Free sample background from Slide 7 Data Analysis Pre-Post Controlled Design Use of interaction term in this study design Use of interaction term in this study design –There are 4 study groups –Multiplicative interaction term of (Pre/Post) * (Control/Intervention) can be interpreted as the relative change in outcome between comparison groups from the baseline to the follow-up evaluation Pre/PostPre/Post Control/InterventionControl/Intervention Post*InterventionPost*Intervention –Reported as a p value only Odds Ratios are reported for both the Control arm and for the Intervention arm Odds Ratios are reported for both the Control arm and for the Intervention arm

© 2003 By Default! A Free sample background from Slide 8 Example Results 150 Per Arm GroupPre % Post % OR95% CIInteraction p Outcome of InterestControl Intervention

© 2003 By Default! A Free sample background from Slide 9 Example Results 150 Per Arm GroupPre % Post % OR95% CIInteraction p Outcome of InterestControl Intervention

© 2003 By Default! A Free sample background from Slide 10 Example Results 150 Per Arm GroupPre % Post % OR95% CIInteraction p Outcome of InterestControl Intervention

© 2003 By Default! A Free sample background from Slide 11 Impact of an Automated Test Results Management System on Patients’ Satisfaction of Test Result Communication Michael E. Matheny, Tejal K. Gandhi, E. John Orav, Zahra Ladak-Merchant, David W. Bates, Gilad J. Kuperman, Eric G. Poon Decision Systems Group / Division of General Medicine Brigham & Women’s Hospital, Boston, MA

© 2003 By Default! A Free sample background from Slide 12 Background Test Result Communication Test result communication between patients and physicians is a critical part of the diagnostic and therapeutic process Test result communication between patients and physicians is a critical part of the diagnostic and therapeutic process However, follow-up of test results in the primary care setting is often challenging: However, follow-up of test results in the primary care setting is often challenging: –High volume of test results –Test results arrive when physician not focused on the patient –Lack of systems to ensure reliability and efficiency

© 2003 By Default! A Free sample background from Slide 13 Background Test Result Communication Problems 31% of women with abnormal mammograms did not receive care consistent with established guidelines 31% of women with abnormal mammograms did not receive care consistent with established guidelines 39% of abnormal TSH at BWH were not followed up within 60 days 39% of abnormal TSH at BWH were not followed up within 60 days 36% of abnormal pap smear were lost to follow-up 36% of abnormal pap smear were lost to follow-up

© 2003 By Default! A Free sample background from Slide 14 Background Physician Workflow 33% of physicians reported they did not always notify patients of abnormal test results 33% of physicians reported they did not always notify patients of abnormal test results ~30% of physicians reported they did not have a reliable method of test result communication ~30% of physicians reported they did not have a reliable method of test result communication 59% of physicians were dissatisfied with how well they managed test results despite spending over an hour a day in this activity 59% of physicians were dissatisfied with how well they managed test results despite spending over an hour a day in this activity

© 2003 By Default! A Free sample background from Slide 15 Background Patient Expectations Patients do not normally discuss their preferences for test result notification with their providers Patients do not normally discuss their preferences for test result notification with their providers Patients preferred telephone notification to regular mail, and found electronic notification to be uncomfortable due to security issues Patients preferred telephone notification to regular mail, and found electronic notification to be uncomfortable due to security issues Patients wanted to be notified of all test results, regardless of whether the results were abnormal Patients wanted to be notified of all test results, regardless of whether the results were abnormal

© 2003 By Default! A Free sample background from Slide 16 Background Patient Satisfaction These problems reduce patient satisfaction with their medical care, and impair future patient-physician interactions These problems reduce patient satisfaction with their medical care, and impair future patient-physician interactions Improving patient satisfaction has been identified as one of the most important issues currently facing healthcare Improving patient satisfaction has been identified as one of the most important issues currently facing healthcare

© 2003 By Default! A Free sample background from Slide 17 Objective To evaluate the impact of an automated test results notification system imbedded into an electronic health record on patient satisfaction regarding test results communication To evaluate the impact of an automated test results notification system imbedded into an electronic health record on patient satisfaction regarding test results communication

© 2003 By Default! A Free sample background from Slide 18 Methods Study Setting Partners HealthCare System Partners HealthCare System –Brigham & Women’s Hospital –Massachusetts General Hospital –Faulkner Hospital –McLean Hospital –Newton-Wellesley Hospital –Free Standing Outpatient Clinics Longitudinal Medical Record (LMR) Longitudinal Medical Record (LMR) –Released July 2000 –Scheduling –Medication lists –Problem lists –Health maintenance record –Clinic notes (free form & templates)

© 2003 By Default! A Free sample background from Slide 19 Methods LMR Summary Screen

© 2003 By Default! A Free sample background from Slide 20 Methods Study Setting Usual care regarding test results management Usual care regarding test results management –Test results were embedded directly into the patients’ electronic health record –No automated test results tracking –All test results were mailed to the physician’s clinic office –Physicians were paged directly for critical results

© 2003 By Default! A Free sample background from Slide 21 Methods Patient Test Results Screen

© 2003 By Default! A Free sample background from Slide 22 Methods Intervention Results Manager - an electronic test results management system embedded into the LMR Results Manager - an electronic test results management system embedded into the LMR –Tracks and displays all test results associated with an ordering physician –Prioritizes by degree of test result abnormality –Facilitates review of test results in context of patient’s history –Generates test result letters –Allows clinicians to set reminders for future testing

© 2003 By Default! A Free sample background from Slide 23 Methods Results Manager Summary Screen

© 2003 By Default! A Free sample background from Slide 24 Methods Results Manager Letter Generation Screen

© 2003 By Default! A Free sample background from Slide 25 Methods Study Design

© 2003 By Default! A Free sample background from Slide 26 Methods Randomization Stratified randomization of 26 primary care clinics based on: Stratified randomization of 26 primary care clinics based on: –Primary Hospital Affiliation (BWH / MGH) –Academic or Community Based –Average Patient Socioeconomic Status Rolling implementation of Results Manager for intervention clinics was conducted from July, 2003 to March, Rolling implementation of Results Manager for intervention clinics was conducted from July, 2003 to March, 2004.

© 2003 By Default! A Free sample background from Slide 27 Methods Study Criteria Patients were randomly sampled in both control and intervention clinics during: Patients were randomly sampled in both control and intervention clinics during: –Pre-Intervention: 12/2002 – 06/2003 –Post-Intervention: 09/2003 – 04/2005 Inclusion Criteria: All patients who had any laboratory, pathology, microbiology, or radiology tests from participating clinics. Inclusion Criteria: All patients who had any laboratory, pathology, microbiology, or radiology tests from participating clinics. Exclusion Criteria: Primary care physician determined that patient should not be contacted or patient did not speak English. Exclusion Criteria: Primary care physician determined that patient should not be contacted or patient did not speak English.

© 2003 By Default! A Free sample background from Slide 28 Methods Survey Instrument Internally developed Internally developed Outcomes were measured on dichotomized Likert scale Outcomes were measured on dichotomized Likert scale Administered by trained research assistants 5 to 7 weeks after test results were posted Administered by trained research assistants 5 to 7 weeks after test results were posted Up to three attempts were made to contact each patient Up to three attempts were made to contact each patient

© 2003 By Default! A Free sample background from Slide 29 Methods Primary Outcome Measure Overall satisfaction with test result communication Overall satisfaction with test result communication –“I am satisfied with the way test results are communicated to me”

© 2003 By Default! A Free sample background from Slide 30 Methods Secondary Outcome Measures Satisfaction with PCP listening skills Satisfaction with PCP listening skills –“My primary care doctor always listens to my concerns” Satisfaction with information given about treatment and condition Satisfaction with information given about treatment and condition –“My primary care doctor gives me as much information about my condition and treatment as I wanted” Satisfaction with general PCP communication Satisfaction with general PCP communication –“My primary care doctor and I communicate very well”

© 2003 By Default! A Free sample background from Slide 31 Methods Outcomes Whether a patient’s expectations were met by the method of test result communication was determined by: Whether a patient’s expectations were met by the method of test result communication was determined by: –Test result type: normal / abnormal Defined as requiring follow-up or a management plan changeDefined as requiring follow-up or a management plan change –Method of test result receipt –Patient’s expected delivery method for test –Hierarchy of test result communication Same Visit > Telephone > Letter > > Next Visit > NeverSame Visit > Telephone > Letter > > Next Visit > Never If receipt was by a more desired method, it was countedIf receipt was by a more desired method, it was counted

© 2003 By Default! A Free sample background from Slide 32 Methods Data Analysis Intention-to-treat analysis Intention-to-treat analysis Multivariate logistic regression (GEE) Multivariate logistic regression (GEE) –Clustered by primary care physician –Adjusted for Patient Age, Gender, Race, Insurance Status, and Self-Reported Health Status –SAS v9.1

© 2003 By Default! A Free sample background from Slide 33 Results Response Rates Contacted Eligible (Surveyed) Contacted Eligible (Refused) Unable to Determine Eligibility p Patient Age (Years) <0.001 Patient Sex (% Female)399 (70.0%)124 (62.6%)495 (65.9%)0.109 Patient Race White374 (65.6%)119 (60.1%)426 (52.7%)0.005 Black109 (19.1%)44 (22.2%)161 (21.4%)0.498 Other87 (15.3%)35 (17.7%)164 (21.9%)0.009 Patient Insurance Commercial257 (45.1%)88 (44.9%)444 (59.1%)<0.001 Medicare246 (43.2%)90 (45.5%)189 (25.2%)<0.001 Medicaid56 (9.8%)16 (8.1%)95 (12.7%)0.099 Self Pay4 (0.7%)3 (1.5%)23 (3.1%)0.008 Totals

© 2003 By Default! A Free sample background from Slide 34 Results Demographics Sample Patient Age (Mean Years)57.3 Patient Sex (% Female)70.0 Patient Race (%) White65.6 Black19.1 Other15.3 Patient Insurance (%) Commercial45.1 Medicare43.1 Medicaid9.6 Self Pay1.1 Totals570

© 2003 By Default! A Free sample background from Slide 35 Results Demographics ControlInterventionp Patient Age (Mean Years) Patient Sex (% Female) Patient Race (%) White African American Other Patient Insurance (%) Commercial Medicare Medicaid Self Pay Totals313257

© 2003 By Default! A Free sample background from Slide 36 Results Primary Outcome GroupPrePostOR95% CIInteraction p Satisfaction with TestControl Result CommunicationIntervention

© 2003 By Default! A Free sample background from Slide 37 Results Secondary Outcome GroupPrePostOR95% CIInteraction p Expectation Met RegardingControl Method of CommunicationIntervention

© 2003 By Default! A Free sample background from Slide 38 Results Secondary Outcome GroupPrePostOR95% CIInteraction p Satisfaction with InformationControl Given for TreatmentIntervention And Condition

© 2003 By Default! A Free sample background from Slide 39 Results Secondary Outcome GroupPrePostOR95% CIInteraction p Satisfaction with PCPControl Listening SkillsIntervention

© 2003 By Default! A Free sample background from Slide 40 Results Secondary Outcome GroupPrePostOR95% CIInteraction p Satisfaction with GeneralControl PCP CommunicationIntervention

© 2003 By Default! A Free sample background from Slide 41 Discussion These findings could be related to a number of potential workflow improvements in RM These findings could be related to a number of potential workflow improvements in RM –Provided a concise summary page for the management of test results ordered by a provider –Template-based results letter generator Can imbed actual test results into letterCan imbed actual test results into letter Improve patient-friendly interpretations of resultsImprove patient-friendly interpretations of results –One-click patient reference information

© 2003 By Default! A Free sample background from Slide 42 Discussion Limitations Generalizability Generalizability –Imbedded in non-commercial EHR –English speaking only Telephone Survey Bias Telephone Survey Bias –Responders vs. Non-Responders –Patient Recall

© 2003 By Default! A Free sample background from Slide 43 Conclusions An automated management system that provides centralized test result tracking and facilitates contact with patients: An automated management system that provides centralized test result tracking and facilitates contact with patients: –improved overall patient satisfaction with communication of test results –Increased patient satisfaction with the discussion of treatments/conditions –Improved receipt of results by an expected method of communication

© 2003 By Default! A Free sample background from Slide 44 Acknowledgements Co-Authors Co-Authors –Tejal K. Gandhi, MD MPH –John Orav, PhD –Zahra Ladak-Merchant, BDS MPH –David W. Bates, MD MS –Gilad J. Kuperman, MD PhD –Eric G. Poon, MD MPH Funding Funding –AHRQ U18-HS –NLM T15-LM-07092

© 2003 By Default! A Free sample background from Slide 45 Michael Matheny, MD MS Brigham & Women’s Hospital Thorn Francis Street Boston, MA Michael Matheny, MD MS Brigham & Women’s Hospital Thorn Francis Street Boston, MA The End

© 2003 By Default! A Free sample background from Slide 46 Methods Survey Response Data