Validation of a Toolkit to Assess the Communication Climate in Health Care Organizations Matthew Wynia, MD, MPH, 1 Megan Johnson, MA, 1 Thomas P McCoy, MS, 2 Leah Griffin, MS 2 1 Institute for Ethics, American Medical Association, Chicago; 2 Wake Forest University School of Medicine, Winston-Salem NC. Statement of Problem Effective communication is the foundation for quality health care Most measures of communication focus on the patient-clinician dyad. Communication might be affected by organizational factors, including a hospital or clinic’s internal climate and infrastructure. If organizations could assess the extent to which they promote (or hinder) effective communication, such assessments would be useful for quality monitoring, to focus QI interventions and, perhaps, for public reporting and benchmarking. Objectives of Program To validate a 360° organizational assessment toolkit, designed to measure organizational supports for effective, patient-centered communication. Validation of the toolkit through the development process (content validity), and by examining correlations between each of the 9 domains assessed (construct validity), and comparing results in each domain with several related patient-reported measures of quality (convergent validity). Description of Program Multi-stakeholder consensus on what organizations should do to ensure effective, patient-centered communication All performance expectations voted on by 21- member Oversight body 1-10 scales, assessing feasibility, importance and measurability Any mean score <7 or individual score ≤3 eliminated the performance expectation Agreement on 9 domains for assessment Toolkit developed to assess performance on these 9 domains through: Patient surveys Clinical staff surveys Non-clinical staff surveys Executive surveys Organizational policy workbook Focus group protocols (optional) 5 hospitals and 4 clinics nationwide were assessed using the toolkit 1,763 patient surveys, 651 staff surveys, 29 executive surveys Data Analyses Correlations and Coefficient alphas of performance scores within and between sub- domains Associations of performance scores in each sub-domain with patient-perceived quality (including org. ethics) using Logistic Regression Comparisons of performance within and between the 9 organizations RESULTS This research was supported by the Ethical Force Program. The Institute for Ethics at the AMA initiated the Ethical Force Program ® in 1997 as a collaborative to develop health care system–wide performance measures for ethics. The protocol, surveys, and focus group questions were approved by Western Institutional Review Board. Funding was provided by a grant from The California Endowment. Collect information Evaluate performance Understand organizational commitment Engage communities Develop workforce Engage individuals a) culture b) language c) health literacy Key lessons learned Organizations can undertake a valid 360° assessment of performance with regard to supporting effective communication, with discrete results in 9 important domains. Assessment results may be useful for tracking organizational performance, benchmarking, and to inform tailored quality improvement interventions. Summary of findings to date Survey domains are reliable (average Chronbach’s alpha = 0.83) and correlate to other quality measures. Considerable within and between site variability: all sites show strong results (score >80) in some domains and weak (score <60) in others – no domain received consistently high scores at all sites. Figure 1 Correlations of performance in the 9 domains Table 2 Correlations* between performance on each domain and several patient-perceived measures of quality Figure 2 Performance variability within and between organizations Communication Quality Improvement: The Domains Assessed by the Toolkit Table 1 Coefficient Alpha of measures within each domain Content areaCoefficient Alpha Patient surveys Coefficient Alpha Staff surveys Org. Commitment Data Collection Develop Workforcen/a0.93 Engage Community Engage Individuals Health Literacy Language Culture Perf. Monitoringn/a0.84 Communication Domain I receive high quality medical care from the health care system My medical records are kept private If a mistake were made in my health care, the system would try to hide it from me OR (95% CI) Org. Commitment1.34 ( )1.22 ( )0.73 ( ) Data Collection0.95 ( )1.00 ( )1.0 ( ) Develop Workforce1.47 ( )1.28 ( )0.73 ( ) Engage Community1.54 ( )1.28 ( )0.73 ( ) Engage Individuals1.40 ( )1.22 ( )0.73 ( ) Health Literacy1.40( )1.28 ( )0.73 ( ) Language0.90 ( )1.05 ( )1.0 ( ) Culture1.28 ( )1.16 ( )0.82 ( ) Perf. monitoring1.40 ( )1.22 ( )0.73 ( ) *Correlations between a 5-point change in scale score and patient responses to trust and quality items.