Implementation and Sustainability in the US National EBP Project Gary R. Bond Dartmouth Psychiatric Research Center Lebanon, NH, USA May 27, 2014 CORE Symposium London

Presentation Outline Describe the three initial implementation phases of the National EBP Project Summarize methods and findings: 1. Fidelity 2. Implementation Factors 3. Sustainability

(1998) Phase 1: Identifying Evidence-Based Practices (EBPs) for People with Severe Mental Illness (1998)

1998 RWJ Conference on Evidence-Based Practices Assertive community treatment (ACT) Supported employment (SE) Integrated dual disorders treatment (IDDT) Illness management and recovery (IMR) Family psychoeducation (FPE) Medication management approaches in psychiatry (MedMAP)

Phase 2: Developing EBP Toolkits and EBP Implementation Model ( )

National EBP Project: Rationale Agency leaders, practitioners, and state leaders need user-friendly resource materials Agency leaders, practitioners, and state leaders need user-friendly resource materials BUT information is not enough… BUT information is not enough… Training and consultation also needed Training and consultation also needed

EBP Toolkits Contents Scientific review articles and PowerPoint lectures Practitioner manuals Educational videos (DVDs) Handouts for policymakers, consumers, & families Fidelity scales Outcome monitoring software

Phase 3: Evaluate Effectiveness of Implementation Model

Design of National EBP Project Examined 5 psychosocial EBPs Examined 5 psychosocial EBPs 8 participating states 8 participating states Each state implemented 2 different EBPs in multiple sites Each state implemented 2 different EBPs in multiple sites 53 sites started (49 final sites) 53 sites started (49 final sites) 2 years of qualitative observation of implementation factors 2 years of qualitative observation of implementation factors Fidelity reviews every 6 months Fidelity reviews every 6 months

Primary Measure of Implementation Outcome Fidelity: Degree to which a particular program follows the standards for an EBP

Correlational Design

1. Fidelity: Methods and Findings

Structure of EBP Fidelity Scales Each scale consists of items Items rated on 5-point continuum 1 = Not Implemented 5 = Fully Implemented ≥ 4.0 considered good implementation

Content of Fidelity Items “Structural” items “Clinical” items

“Structural” Fidelity Items Things that can be done by administrative fiat, such as: – –Daily team meetings – –Multidisciplinary staffing – –Low caseload ratio – –Following a curriculum – –Distributing educational handouts No clinical skill needed to implement

Clinical Fidelity Items Assessed the provisions of interventions requiring clinical skill Examples: – –Stagewise interventions for dual disorders – –Motivational techniques Clinical items involved qualitative judgments

Data Collection for EBP Fidelity Scales Ratings made by two independent assessors Day-long site visit Multiple data sources (interviews, chart review, observation) Fidelity report given afterwards to agencies for quality improvement

Finding #1: Usual practice definitely not evidence-based

Finding #2: Over Half the Sites Successfully Implemented the EBP

National EBP Project: 2-Year Rates of Successful Program Implementation

Finding #3: Some EBPs were easier to implement than others

Finding #4: Most improvements in EBP fidelity occurred within the first year

2. Implementation Factors: Methods and Findings

Implementation Factors: Data Collection Procedures Implementation monitors recorded notes at study sites ~ monthly for 2 years Periodic interviews with key staff Notes and interviews entered as “documents” in qualitative data base

Coding System Documents contained codable “events” Each event coded according to – –Type: Barrier, Facilitator, or Strategy – –Content: 26 Dimensions grouped into 5 Domains

Conceptual Framework for Implementation Factors

Total Number of Units Coded

Overall, Which Barrier, Facilitator, and Strategy Domains Were Most Common?

Most Frequently Noted Domains of Barriers, Facilitators, and Strategies (Total Sample) Barriers: – –Leadership, Prioritization, Work Force Facilitators: – –Prioritization, Leadership, Work Force Strategies: – –Work Force, Leadership

How Do Barriers, Facilitators, and Strategies Affect Fidelity? Correlational Findings

Correlations: Implementation Factors over Both Years with 24-Month EBP Fidelity

Summary: Barriers Content domains relatively similar across EBPs Did not appear to affect fidelity!

Summary: Facilitators EBPs differed substantially Leadership generally had a huge positive effect on fidelity Work Force factors unrelated or negatively correlated with fidelity Feedback structures in place had a positive impact

Summary: Strategies EBPs differed on Leadership and Work Flow Work Force (mainly Training!) was unrelated or had a negative effect on fidelity Work Flow and Reinforcement had a positive effect on fidelity

Negative Findings for Training: Speculations Didactic training is a poor method for teaching clinical skills Clinical skills needed to implement an EBP are not identical to components of program fidelity Staff turnover also helps explain negative correlation between training and fidelity outcomes

Implementation Factors: Conclusions Active, observable Leadership has dramatic impact on implementation A focus on Work Flow (policies, documentation) and Reinforcement (fidelity and outcome monitoring) important strategy Work Force facilitators and strategies had a puzzling negative relationship with fidelity

3. Sustainability: Methods and Findings

Methods Interviews with site leaders 2 years after end of 2-year implementation phase (2007) Second round of interviews 4 years later (2011) Close-ended and open-ended questions

Barriers to Sustainability

Strategies Used to Sustain EBP

Factors Influencing Sustainability Stable and adequate funding Fidelity and outcome monitoring Weekly supervision State leadership supportive of EBP

Overall Conclusions Systematic efforts at implementation have a powerful effect on successful implementation Some elements in implementation model more effective than others Several factors in common for successful implementation and sustainability