Necessary but Not Sufficient: Why Strategic Implementation Climate and Molar Organizational Climate Both Matter for EBP Implementation Nate Williams, PhD, LCSW Assistant Professor, Boise Statue University School of Social Work

Q: How can mental health service organizations support EBP implementation? Q: How do molar climate and implementation climate relate to EBP implementation? Q--What are the causal determinants, or levers, that they can pull to support EBP implementation?

Molar Organizational Climate & EBP Implementation Climate Molar Climate EBP Implementation Climate Employees’ shared perceptions of the impact of the work environment on their personal well-being Long research history in the organizational sciences Common to all organizational environments Mixed evidence linking it to implementation Clinicians’ shared perceptions of their organization’s policies, procedures, and practices and the extent to which the organization expects, supports, and rewards the implementation of EBP Considered most proximal to EBP implementation Linked to implementation outcomes in manufacturing organizations, but… Much less research and equivocal support in health services

Theoretical Models Linking Climate to EBP Implementation Implementation Climate Molar Climate EBP implementation Implementation Climate Molar Climate EBP implementation Implementation Climate Molar Climate EBP implementation

An interactive Model of Climate Effects We propose that EBP implementation climate and molar climate interact in their cross-level effects on clinicians’ EBP use Sub-optimal EBP Implementation Optimal EBP implementation Poor EBP implementation Sub-optimal EBP implementation Low High Implementation Climate Positive Molar Climate Negative The equivocal results of prior research and theory on organizational climate and organizational support suggest an interaction may be present

Short- and Long-Term Effects of Climate We propose that implementation climate and molar climate will have long-term (interactive) effects on clinicians’ EBP use Climate Climate Climate

Study Hypotheses 1 & 2: Concurrent Effects of Climate Time 1 Time 2 Implementation Climate (T1) Clinician EBP implementation H1 Molar Climate (T1) H2

Study Hypotheses 3 & 4: Long-Term Effects of Climate Time 1 Time 2 Molar Climate (T1) H4 Implementation Climate (T1) Clinician EBP implementation (T2) H3

Setting – City of Philadelphia

Timing of Data Collection Pre-EPIC 19 agencies 23 sites 130 therapists 22 agencies 28 sites 247 therapists 21 agencies 26 sites 249 therapists

Study Sample (k = 20 sites) Time 1 Time 2 Climate n = 112 clinicians response rate = 46% n = 164 clinicians response rate = 65% M within-organization response rate = 59% (min = 23%, max = 92%) M clinicians per organization = 11.75 (min = 3, max = 39) Total N = 235 n = 41 clinicians present at both times (17.4% of total sample)

Measures: Molar Climate OSC Functionality Scale (Glisson et al., 2008) 15 items (α = .92) Cooperation There is a feeling of cooperation among my coworkers Growth & Advancement This agency provides numerous opportunities to advance if you work for it Role Clarity My job responsibilities are clearly defined

Measures: Implementation Climate Implementation Climate Scale (Ehrhart et al., 2014) 18 items (α = .91) Focus Using EBP is a top priority in this agency Recognition Clinician who use EBP are held in high esteem in this agency Educational Support This agency provides EBP trainings or in-services Rewards The better you are at using EBP, the more likely you are to get a bonus or raise Selection (EBP) This agency selects staff who have previously used EBP Selection (Openness) This agency selects staff who are adaptable

Construct Validity of Organizational Climate Scores Within-organization agreement rwg(j) all > .70 range = .79 to .98 Between-organization variance ICS - η2 = .41 OSC - η2 = .35

Outcome Measure: Clinician EBP Use CBT subscale of the Therapy Procedures Checklist-Family Revised (Weersing, Weisz, & Donenberg, 2002) Consider a specific, representative client you’re currently treating and endorse specific techniques from a list 33 items α = .94 (T1), α = .92 (T2) 1-5 point range 33 items Example items: Teach child that cognitions affect mood and can cause emotional problems Give direct instruction and information to change thoughts Encourage appropriate self-talk Teach problem-solving and decision-making strategies Use time-out Develop secondary reinforcers Parent training in child management Make up a contract

Preliminary Analyses – Variation in DV Time 1 M = 3.25, SD = .70 ICC(1) = .29*** Time 2 M = 3.33, SD = .65 ICC(1) = .12**

Results: Concurrent Effects of Climate Implementation Climate (T1) Clinician EBP implementation H1 Molar Climate (T1) H2 * * β= .38 Controlling for the other variables, every one standard deviation increase in implementation climate predicted a .38 standard deviation increase in clinicians’ expected use of EBP H1 - β= .38, p = .043, R2Δ = .29 H2 - p = .016, R2Δ = .30 Control variables - org size (N therapists) - Clinician tenure - Clinician hours per week Climate Time 1 Time 2

Concurrent Interaction between Molar and Implementation Climates The standardized beta for positive molar climates is B=.35

Results: Long-Term Effects of Climate Molar Climate (T1) H4 Implementation Climate (T1) Clinician EBP implementation (T2) H3 * * β = .27 H3 was supported β = .27, p = .028 R2Δ = 55% H4 was supported B = .01, p = .042 R2Δ = 1% Higher levels of implementation climate at Time 1 predicted increased use of EBP among clinicians who were present in the organizations two years later (B = .34, SE = .14, p = .028), accounting for 55% of the organizational intercept variance beyond that accounted for by the other variables. For every one standard deviation increase in implementation climate at Time 1, clinicians’ expected use of EBP in these organizations at Time 2 increased by .27 standard deviations Climate Time 1 Time 2

Long-Term Interaction Between Molar and Implementation Climates Standardized beta in positive molar climates = .23

What Does it Mean for Theory & Practice? Confirmation of the role of implementation climate & identification of a boundary condition Support for long-term stability of climate effects Implications Pre-implementation assessments of climate may be valuable for optimizing implementation efforts Leaders must attend to the effects of the work environment on clinicians in addition to EBP issues

Study Strengths and Limitations Correlational not experimental Common method variance Clinician reports of EBP use via standardized measure CBT is not the only EBP for youth psychiatric disorders

Bottom Line for Executives Organizations can optimize the delivery of effective treatments by developing organizational climates that engender high expectations and support for EBP implementation as well as strong support for clinician well-being

THANK YOU! Questions? Nate Williams, PhD, LCSW natewilliams@boisestate.edu 208-426-3145