1. © 2003 By Default! A Free sample background from www.powerpointbackgrounds.com Slide 1 Strengths & Weaknesses of a Pre-Post Controlled Randomized Trial Michael E. Matheny, MD MS Decision Systems Group Brigham & Women’s Hospital, Boston, MA

2. © 2003 By Default! A Free sample background from www.powerpointbackgrounds.com Slide 2 Trial Design

3. © 2003 By Default! A Free sample background from www.powerpointbackgrounds.com Slide 3 Trial Design

4. © 2003 By Default! A Free sample background from www.powerpointbackgrounds.com Slide 4 Pre-Post Randomized Design Strengths Allows the study design to account for changes in measured outcomes by unmeasured factors over time Allows the study design to account for changes in measured outcomes by unmeasured factors over time Provides a single p value for the effect of the intervention that incorporates all of the data Provides a single p value for the effect of the intervention that incorporates all of the data Used primarily for: Used primarily for: –Studies that occur over a longer time period –Studies in an environment where other changes taking place could affect the measured outcomes

5. © 2003 By Default! A Free sample background from www.powerpointbackgrounds.com Slide 5 Pre-Post Randomized Design Weaknesses Requires allocation of significant time and resources prior to introduction of the intervention Requires allocation of significant time and resources prior to introduction of the intervention –Doubles the cost of data collection Very little power gained from “doubling” the sample size Very little power gained from “doubling” the sample size –calculations are still based on two arm design Unable to provide a single odds ratio result for the intervention Unable to provide a single odds ratio result for the intervention

6. © 2003 By Default! A Free sample background from www.powerpointbackgrounds.com Slide 6 Data Analysis SAS Code

7. © 2003 By Default! A Free sample background from www.powerpointbackgrounds.com Slide 7 Data Analysis Pre-Post Controlled Design Use of interaction term in this study design Use of interaction term in this study design –There are 4 study groups –Multiplicative interaction term of (Pre/Post) * (Control/Intervention) can be interpreted as the relative change in outcome between comparison groups from the baseline to the follow-up evaluation Pre/PostPre/Post Control/InterventionControl/Intervention Post*InterventionPost*Intervention –Reported as a p value only Odds Ratios are reported for both the Control arm and for the Intervention arm Odds Ratios are reported for both the Control arm and for the Intervention arm

8. © 2003 By Default! A Free sample background from www.powerpointbackgrounds.com Slide 8 Example Results 150 Per Arm GroupPre % Post % OR95% CIInteraction p Outcome of InterestControl33 1.000.62-1.610.042 Intervention33502.001.25-3.19

9. © 2003 By Default! A Free sample background from www.powerpointbackgrounds.com Slide 9 Example Results 150 Per Arm GroupPre % Post % OR95% CIInteraction p Outcome of InterestControl33401.330.83-2.140.047 Intervention33572.611.64-4.18

10. © 2003 By Default! A Free sample background from www.powerpointbackgrounds.com Slide 10 Example Results 150 Per Arm GroupPre % Post % OR95% CIInteraction p Outcome of InterestControl33170.400.23-0.700.014 Intervention33 1.000.62-1.61

11. © 2003 By Default! A Free sample background from www.powerpointbackgrounds.com Slide 11 Impact of an Automated Test Results Management System on Patients’ Satisfaction of Test Result Communication Michael E. Matheny, Tejal K. Gandhi, E. John Orav, Zahra Ladak-Merchant, David W. Bates, Gilad J. Kuperman, Eric G. Poon Decision Systems Group / Division of General Medicine Brigham & Women’s Hospital, Boston, MA

12. © 2003 By Default! A Free sample background from www.powerpointbackgrounds.com Slide 12 Background Test Result Communication Test result communication between patients and physicians is a critical part of the diagnostic and therapeutic process Test result communication between patients and physicians is a critical part of the diagnostic and therapeutic process However, follow-up of test results in the primary care setting is often challenging: However, follow-up of test results in the primary care setting is often challenging: –High volume of test results –Test results arrive when physician not focused on the patient –Lack of systems to ensure reliability and efficiency

13. © 2003 By Default! A Free sample background from www.powerpointbackgrounds.com Slide 13 Background Test Result Communication Problems 31% of women with abnormal mammograms did not receive care consistent with established guidelines 31% of women with abnormal mammograms did not receive care consistent with established guidelines 39% of abnormal TSH at BWH were not followed up within 60 days 39% of abnormal TSH at BWH were not followed up within 60 days 36% of abnormal pap smear were lost to follow-up 36% of abnormal pap smear were lost to follow-up

14. © 2003 By Default! A Free sample background from www.powerpointbackgrounds.com Slide 14 Background Physician Workflow 33% of physicians reported they did not always notify patients of abnormal test results 33% of physicians reported they did not always notify patients of abnormal test results ~30% of physicians reported they did not have a reliable method of test result communication ~30% of physicians reported they did not have a reliable method of test result communication 59% of physicians were dissatisfied with how well they managed test results despite spending over an hour a day in this activity 59% of physicians were dissatisfied with how well they managed test results despite spending over an hour a day in this activity

15. © 2003 By Default! A Free sample background from www.powerpointbackgrounds.com Slide 15 Background Patient Expectations Patients do not normally discuss their preferences for test result notification with their providers Patients do not normally discuss their preferences for test result notification with their providers Patients preferred telephone notification to regular mail, and found electronic notification to be uncomfortable due to security issues Patients preferred telephone notification to regular mail, and found electronic notification to be uncomfortable due to security issues Patients wanted to be notified of all test results, regardless of whether the results were abnormal Patients wanted to be notified of all test results, regardless of whether the results were abnormal

16. © 2003 By Default! A Free sample background from www.powerpointbackgrounds.com Slide 16 Background Patient Satisfaction These problems reduce patient satisfaction with their medical care, and impair future patient-physician interactions These problems reduce patient satisfaction with their medical care, and impair future patient-physician interactions Improving patient satisfaction has been identified as one of the most important issues currently facing healthcare Improving patient satisfaction has been identified as one of the most important issues currently facing healthcare

17. © 2003 By Default! A Free sample background from www.powerpointbackgrounds.com Slide 17 Objective To evaluate the impact of an automated test results notification system imbedded into an electronic health record on patient satisfaction regarding test results communication To evaluate the impact of an automated test results notification system imbedded into an electronic health record on patient satisfaction regarding test results communication

18. © 2003 By Default! A Free sample background from www.powerpointbackgrounds.com Slide 18 Methods Study Setting Partners HealthCare System Partners HealthCare System –Brigham & Women’s Hospital –Massachusetts General Hospital –Faulkner Hospital –McLean Hospital –Newton-Wellesley Hospital –Free Standing Outpatient Clinics Longitudinal Medical Record (LMR) Longitudinal Medical Record (LMR) –Released July 2000 –Scheduling –Medication lists –Problem lists –Health maintenance record –Clinic notes (free form & templates)

19. © 2003 By Default! A Free sample background from www.powerpointbackgrounds.com Slide 19 Methods LMR Summary Screen

20. © 2003 By Default! A Free sample background from www.powerpointbackgrounds.com Slide 20 Methods Study Setting Usual care regarding test results management Usual care regarding test results management –Test results were embedded directly into the patients’ electronic health record –No automated test results tracking –All test results were mailed to the physician’s clinic office –Physicians were paged directly for critical results

21. © 2003 By Default! A Free sample background from www.powerpointbackgrounds.com Slide 21 Methods Patient Test Results Screen

22. © 2003 By Default! A Free sample background from www.powerpointbackgrounds.com Slide 22 Methods Intervention Results Manager - an electronic test results management system embedded into the LMR Results Manager - an electronic test results management system embedded into the LMR –Tracks and displays all test results associated with an ordering physician –Prioritizes by degree of test result abnormality –Facilitates review of test results in context of patient’s history –Generates test result letters –Allows clinicians to set reminders for future testing

23. © 2003 By Default! A Free sample background from www.powerpointbackgrounds.com Slide 23 Methods Results Manager Summary Screen

24. © 2003 By Default! A Free sample background from www.powerpointbackgrounds.com Slide 24 Methods Results Manager Letter Generation Screen

25. © 2003 By Default! A Free sample background from www.powerpointbackgrounds.com Slide 25 Methods Study Design

26. © 2003 By Default! A Free sample background from www.powerpointbackgrounds.com Slide 26 Methods Randomization Stratified randomization of 26 primary care clinics based on: Stratified randomization of 26 primary care clinics based on: –Primary Hospital Affiliation (BWH / MGH) –Academic or Community Based –Average Patient Socioeconomic Status Rolling implementation of Results Manager for intervention clinics was conducted from July, 2003 to March, 2004. Rolling implementation of Results Manager for intervention clinics was conducted from July, 2003 to March, 2004.

27. © 2003 By Default! A Free sample background from www.powerpointbackgrounds.com Slide 27 Methods Study Criteria Patients were randomly sampled in both control and intervention clinics during: Patients were randomly sampled in both control and intervention clinics during: –Pre-Intervention: 12/2002 – 06/2003 –Post-Intervention: 09/2003 – 04/2005 Inclusion Criteria: All patients who had any laboratory, pathology, microbiology, or radiology tests from participating clinics. Inclusion Criteria: All patients who had any laboratory, pathology, microbiology, or radiology tests from participating clinics. Exclusion Criteria: Primary care physician determined that patient should not be contacted or patient did not speak English. Exclusion Criteria: Primary care physician determined that patient should not be contacted or patient did not speak English.

28. © 2003 By Default! A Free sample background from www.powerpointbackgrounds.com Slide 28 Methods Survey Instrument Internally developed Internally developed Outcomes were measured on dichotomized Likert scale Outcomes were measured on dichotomized Likert scale Administered by trained research assistants 5 to 7 weeks after test results were posted Administered by trained research assistants 5 to 7 weeks after test results were posted Up to three attempts were made to contact each patient Up to three attempts were made to contact each patient

29. © 2003 By Default! A Free sample background from www.powerpointbackgrounds.com Slide 29 Methods Primary Outcome Measure Overall satisfaction with test result communication Overall satisfaction with test result communication –“I am satisfied with the way test results are communicated to me”

30. © 2003 By Default! A Free sample background from www.powerpointbackgrounds.com Slide 30 Methods Secondary Outcome Measures Satisfaction with PCP listening skills Satisfaction with PCP listening skills –“My primary care doctor always listens to my concerns” Satisfaction with information given about treatment and condition Satisfaction with information given about treatment and condition –“My primary care doctor gives me as much information about my condition and treatment as I wanted” Satisfaction with general PCP communication Satisfaction with general PCP communication –“My primary care doctor and I communicate very well”

31. © 2003 By Default! A Free sample background from www.powerpointbackgrounds.com Slide 31 Methods Outcomes Whether a patient’s expectations were met by the method of test result communication was determined by: Whether a patient’s expectations were met by the method of test result communication was determined by: –Test result type: normal / abnormal Defined as requiring follow-up or a management plan changeDefined as requiring follow-up or a management plan change –Method of test result receipt –Patient’s expected delivery method for test –Hierarchy of test result communication Same Visit > Telephone > Letter > Email > Next Visit > NeverSame Visit > Telephone > Letter > Email > Next Visit > Never If receipt was by a more desired method, it was countedIf receipt was by a more desired method, it was counted

32. © 2003 By Default! A Free sample background from www.powerpointbackgrounds.com Slide 32 Methods Data Analysis Intention-to-treat analysis Intention-to-treat analysis Multivariate logistic regression (GEE) Multivariate logistic regression (GEE) –Clustered by primary care physician –Adjusted for Patient Age, Gender, Race, Insurance Status, and Self-Reported Health Status –SAS v9.1

33. © 2003 By Default! A Free sample background from www.powerpointbackgrounds.com Slide 33 Results Response Rates Contacted Eligible (Surveyed) Contacted Eligible (Refused) Unable to Determine Eligibility p Patient Age (Years)57.458.650.0<0.001 Patient Sex (% Female)399 (70.0%)124 (62.6%)495 (65.9%)0.109 Patient Race White374 (65.6%)119 (60.1%)426 (52.7%)0.005 Black109 (19.1%)44 (22.2%)161 (21.4%)0.498 Other87 (15.3%)35 (17.7%)164 (21.9%)0.009 Patient Insurance Commercial257 (45.1%)88 (44.9%)444 (59.1%)<0.001 Medicare246 (43.2%)90 (45.5%)189 (25.2%)<0.001 Medicaid56 (9.8%)16 (8.1%)95 (12.7%)0.099 Self Pay4 (0.7%)3 (1.5%)23 (3.1%)0.008 Totals570198751

34. © 2003 By Default! A Free sample background from www.powerpointbackgrounds.com Slide 34 Results Demographics Sample Patient Age (Mean Years)57.3 Patient Sex (% Female)70.0 Patient Race (%) White65.6 Black19.1 Other15.3 Patient Insurance (%) Commercial45.1 Medicare43.1 Medicaid9.6 Self Pay1.1 Totals570

35. © 2003 By Default! A Free sample background from www.powerpointbackgrounds.com Slide 35 Results Demographics ControlInterventionp Patient Age (Mean Years)57.157.70.648 Patient Sex (% Female)64.676.30.003 Patient Race (%) White65.965.30.929 African American19.1 1.000 Other14.915.70.816 Patient Insurance (%) Commercial50.042.00.064 Medicare40.945.80.270 Medicaid8.411.40.260 Self Pay0.70.81.000 Totals313257

36. © 2003 By Default! A Free sample background from www.powerpointbackgrounds.com Slide 36 Results Primary Outcome GroupPrePostOR95% CIInteraction p Satisfaction with TestControl90820.6820.345-1.3470.012 Result CommunicationIntervention85932.3801.068-5.306

37. © 2003 By Default! A Free sample background from www.powerpointbackgrounds.com Slide 37 Results Secondary Outcome GroupPrePostOR95% CIInteraction p Expectation Met RegardingControl61470.4690.326-0.6740.001 Method of CommunicationIntervention44581.3940.827-2.350

38. © 2003 By Default! A Free sample background from www.powerpointbackgrounds.com Slide 38 Results Secondary Outcome GroupPrePostOR95% CIInteraction p Satisfaction with InformationControl95870.7110.261-1.9370.034 Given for TreatmentIntervention94963.6711.300-10.37 And Condition

39. © 2003 By Default! A Free sample background from www.powerpointbackgrounds.com Slide 39 Results Secondary Outcome GroupPrePostOR95% CIInteraction p Satisfaction with PCPControl99931.0240.062-16.790.501 Listening SkillsIntervention99982.9620.760-11.54

40. © 2003 By Default! A Free sample background from www.powerpointbackgrounds.com Slide 40 Results Secondary Outcome GroupPrePostOR95% CIInteraction p Satisfaction with GeneralControl96911.3730.404-4.6680.367 PCP CommunicationIntervention97962.8600.957-8.547

41. © 2003 By Default! A Free sample background from www.powerpointbackgrounds.com Slide 41 Discussion These findings could be related to a number of potential workflow improvements in RM These findings could be related to a number of potential workflow improvements in RM –Provided a concise summary page for the management of test results ordered by a provider –Template-based results letter generator Can imbed actual test results into letterCan imbed actual test results into letter Improve patient-friendly interpretations of resultsImprove patient-friendly interpretations of results –One-click patient reference information

42. © 2003 By Default! A Free sample background from www.powerpointbackgrounds.com Slide 42 Discussion Limitations Generalizability Generalizability –Imbedded in non-commercial EHR –English speaking only Telephone Survey Bias Telephone Survey Bias –Responders vs. Non-Responders –Patient Recall

43. © 2003 By Default! A Free sample background from www.powerpointbackgrounds.com Slide 43 Conclusions An automated management system that provides centralized test result tracking and facilitates contact with patients: An automated management system that provides centralized test result tracking and facilitates contact with patients: –improved overall patient satisfaction with communication of test results –Increased patient satisfaction with the discussion of treatments/conditions –Improved receipt of results by an expected method of communication

44. © 2003 By Default! A Free sample background from www.powerpointbackgrounds.com Slide 44 Acknowledgements Co-Authors Co-Authors –Tejal K. Gandhi, MD MPH –John Orav, PhD –Zahra Ladak-Merchant, BDS MPH –David W. Bates, MD MS –Gilad J. Kuperman, MD PhD –Eric G. Poon, MD MPH Funding Funding –AHRQ U18-HS-11046 –NLM T15-LM-07092

45. © 2003 By Default! A Free sample background from www.powerpointbackgrounds.com Slide 45 Michael Matheny, MD MS mmatheny@partners.org Brigham & Women’s Hospital Thorn 309 75 Francis Street Boston, MA 02115 Michael Matheny, MD MS mmatheny@partners.org Brigham & Women’s Hospital Thorn 309 75 Francis Street Boston, MA 02115 The End

46. © 2003 By Default! A Free sample background from www.powerpointbackgrounds.com Slide 46 Methods Survey Response Data