Hannah E. Frank, Lisa Saldana, Philip C. Kendall, Holle Schaper Bringing evidence-based interventions into the schools: The impact of organizational factors on implementation success Hannah E. Frank, Lisa Saldana, Philip C. Kendall, Holle Schaper

Background Children with mental health problems often do not receive mental health services (Farrell & Barrett, 2007) When they do, it is most often through their schools (Canino et al. 2004) Several barriers to implementing interventions (Paulus et al. 2016) Planning required Staff time Financial Constraints Confidentiality concerns Organizational support for the program

Computerized and Internet-Based Intervention Programs Alternative to traditional manualized treatments Cost effective, standardized, improve feasibility of widespread dissemination Camp Cope-A-Lot (Kendall & Khanna, 2008) Computer-assisted CBT based on Coping Cat (Kendall & Hedtke, 2006) Requires minimal staff training, staff time, and financial burden Dissemination and implementation of Camp Cope-A-Lot in schools is an appropriate model to examine factors affecting the likelihood of school-based intervention adoption

Implementation Framework to Guide Success Consolidated Framework for Implementation Research (CFIR; Damschroder et al., 2009) Five major domains Inner setting: characteristics of organization implementing the intervention Minimal research has examined how organizational factors affect the implementation of evidence-based interventions in schools

Current Study Aim: Examine organizational factors that relate to Camp Cope-a-Lot implementation outcomes as measured by the Stages of Implementation Completion (SIC) Part of larger study examining sustainability of computer-assisted Camp Cope- A-Lot intervention to treat anxious youth in schools in the United States (U.S.) and Canada (NIMH R01MH086438) Hypotheses: Higher scores on organizational measures will be associated with more successful implementation outcomes (i.e., program start-up, competency) Schools with better organizational scores will be more likely to successfully implement Camp Cope-A-Lot with at least one child

Methods: Participants Schools: New Jersey/Pennsylvania (n= 7) Canada (n = 13) Providers: Participants (N = 38) were school staff members Did not need to have prior mental health training (24% did not) Received Camp Cope-A-Lot training

Methods: Measures Organizational Social Context (OSC: Glisson et al. 2008) 105 items administered directly to school staff Yields T-scores Culture domain: Proficiency, rigidity, resistance Climate domain: Engagement, functionality, stress Stages of Implementation Completion (SIC; Saldana, 2014) Examines implementation process and milestones across 3 phases and 8 stages specific to this intervention Scoring: Duration and proportion scores

Methods: Measures Knowledge Test (Beidas et al. 2009): Role Plays: 20 items that assess providers’ knowledge about CBT for youth anxiety Three versions Role Plays: Providers read a vignette about an anxious child and were asked to conduct an exposure with that child Had up to 8-minutes to demonstrate how they would prepare the “child” (a trained graduate student using a script) for the task Role plays were transcribed and coded for adherence and skill

Methods: Treatment Program Camp Cope-A-Lot Computer-assisted treatment Target Audience Schools Community Mental Health Centers Private Practices Medical Centers Training Clinics Social Service Agencies 12 “Levels” (6 child only; 6 with coach/therapist)

Methods: Procedures Provider Training One-day workshop Completed measures before and after workshop Received weekly consultation calls by phone Youth Referral Youth with elevated BASC anxiety scales or teacher nomination were eligible to receive Camp Cope-A-Lot Scheduled to meet with providers weekly

Methods: Procedures Organizational Data Collection Teachers and school staff members at each school were asked to complete organizational measures prior to use of the Camp Cope-A-Lot program Implementation Behavior Measured by the SIC Competency measured by: Knowledge Tests: before training, after training, end of year 1 Role Plays: after training, end of year 1

Results: Correlations   Duration of Scored Activities (Phase 1) Duration of Scored Activities (Phase 2) Proportion of Scored Activities (Phase 1)1 Proportion of Scored Activities (Phase 2) OSC Proficiency OSC Rigidity OSC Resistance OSC Engagement OSC Functionality OSC Stress — -.55* -- .23 .18 .65** .37 .03 OSC Rigidity -.03 .02 .07 .43┼ -.33 .04 -.15 .13 .61** .39┼ -.24 .17 .25 -.05 -.04 .41┼ -.30 -.01 .34 -.22 -.28 .68** -.16 -.003 -.13 .24 .36 .20 -.57** -.43┼ ┼p <.10 *p <.05 **p <.01; 1Cells are blank for proportion in phase 1 because there was no variability due to all participating schools completing all activities in phase 1

Results: Comparisons between U.S. and Canada Provider training dates: Varied for US schools Only two dates for Canadian schools T-tests compared organizational factors and implementation behaviors: Organizational Factors: Proficiency was significantly higher at domestic schools Pre-Implementation (Phase 1): Significantly longer duration for domestic sites; No variability in proportion scores (all schools completed all activities) Implementation (Phase 2): No significant differences in duration or proportion Program Start-Up: No significant differences between domestic and international sites regarding whether a child was treated, c2 (2, N = 20) = 2.69, p = .10

Results: Predictors of Pre-Implementation SIC duration Variable Model 1   Model 2 B SE p SE B P OSC Proficiency 3.76 1.04 .002* 1.90 0.96 .07+ International (0 = domestic; 1 = international) -55.51 15.58 R2 .42 .67 ∆R2 .25 Fchange (df1, df2) 12.69 (1, 17)* * p < .05; + p < .10

Results: Predictors of Implementation Duration and Proportion Longer pre-implementation durations predicted shorter implementation durations, b = -0.38, t = -2.78, p = 0.01, R2 = .30 Site location and OSC proficiency were not significant predictors of implementation duration. Predictors of implementation proportion scores (i.e., OSC proficiency, pre-implementation duration, site location) were not significant

Results: Predictors of Camp Cope-A-Lot Start Up Camp Cope-A-Lot start-up: defined as a site treating at least one child Logistic regression with OSC proficiency, site location, and pre- implementation duration as predictors was not significant Number of children treated with Camp Cope-A-Lot Same predictors (OSC proficiency, site location, and pre- implementation duration) Site location was the only significant predictor, such that domestic sites treated significantly more children, b = -6.03, t = - 10.49, p < 0.001, R2 = .93.

Results: Predictors of competency Measures of competency: Knowledge test – 3 total Role plays – 2 total Predictors of competency: Pre-implementation duration Implementation duration Site location OSC proficiency Proficiency emerged as the only significant predictor of: Number of knowledge tests passed, b = 0.06, t = 2.23, p = 0.04, R2 = .40 Number of role plays passed: b = 0.06, t = 2.33, p = 0.03, R2 = .29

Discussion: Overview of Findings Organizational Factors Proficiency was the only organizational factor associated with implementation outcomes Trend for higher proficiency scores being associated with longer pre-implementation duration Longer pre-implementation duration predicted shorter implementation duration Higher proficiency scores predicted passing of more knowledge tests and role plays (i.e., competency)

Discussion: Overview of Findings Comparisons between Domestic and International Sites Proficiency and pre-implementation duration were higher at domestic sites No differences in duration of implementation activities Differences in organizational structure (i.e., Canadian sites governed by district administrator) Predictors of Successful Start Up 16 out of 20 programs achieved program start-up Domestic sites treated a significantly higher number of children than international sites, likely because of number of providers per school

Implications for School Settings Importance of proficiency Proficient schools may spend longer on pre-implementation activities in order to ensure that they have the necessary resources to implement the intervention Implementation may proceed more quickly when more time is spent on pre-implementation Differences in organizational structure affects outcomes Size of the workforce may affect the degree to which use of the program is widespread throughout the school

Implications Beyond School Settings Importance of creating proficient organizational cultures to improve implementation outcomes Has been demonstrated in previous studies as well (e.g., Williams & Glisson, 2013) Careful planning for implementation activities during pre- implementation may reduce the amount of time needed during implementation

Limitations and Future Directions Small sample size Only incorporated one element of larger contextual framework (CFIR) Future directions With larger sample size, could examine a mediational model: Pre-implementation phase duration  Organizational culture  Provider behavior Examine in middle school and high school contexts

Acknowledgments The Oregon Social Learning Center, especially: This research was supported by NIMH under award number R01MH086438 The Oregon Social Learning Center, especially: Lisa Saldana Holle Schaper Jason Chapman The Child and Adolescent Anxiety Disorders Clinic, especially: Philip Kendall Matthew Carper The children and school staff who participated Thank you!!!