Measurement of Implementation Components Ten Years After a Nationwide Introduction of Empirically Supported Programs Society for Prevention Research, 21st annual meeting, The science of prevention: Building a comprehensive National Strategy for Well-Being May 28-31, 2013; Hyatt Regency San Francisco  Terje Ogden The Norwegian Center for Child Behavioral Development Unirand, University of Oslo, Norway 9/7/2018 © The Norwegian Center for Child Behavioral Development

Co-authors of the study Gunnar Bjørnebekk, John Kjøbli, Joshua Patras, Terje Christiansen, Knut Taraldsen, Nina Tollefsen   Norwegian Center for Child Behavioral Development, University of Oslo, P.O. Box 7053, Majorstuen 0306, Oslo, Norway. 9/7/2018 © The Norwegian Center for Child Behavioral Development

Empirically supported programs 1998 a national initiative was launched by the Norwegian government in order to increase and improve services, competence and research in relation to children and youth with conduct problems, 1998: A task force appointed by the Norwegian Research Council recommended that empirically supported family and community treatment programs should be implemented and evaluated in Norway, The Behavior project (1998) was transformed into a permanent center (2003): “The Norwegian Center for Child Behavioral Development”. 9/7/2018 Side 3 © The Norwegian Center for Child Behavioral Development

Center for implementation and research The Norwegian Center for Child Behavioral Development Program selection Long term funding Evidence based policy support RecruitimentTraining Supervision Technical support Center for implementation and research Program development and implementation Technical support Monitoring of program- and treatment integrity Multi-allegiance research on: Outcomes, implementation, mediators and change processes Sustainability of programs and clincial outcomes

Multiple evidence-based model programs PMTO Parent Management Training MST Multisystemic Therapy PALS School-wide PBS model TIBIR Early intervention For Children at Risk Functional Family Therapy FFT MST and Contingency Management PMTO for Minority Families and Children Multidimensional Treatment Foster Care (MTFC) Multiple evidence-based model programs 9/7/2018 Side 5 © The Norwegian Center for Child Behavioral Development

An RCT effectiveness study was conducted with encouraging outcomes. PMTO was implemented in Norway from 1999, and by 2011, six generations with 383 therapists were trained and certified, In 2010 a total of 4661 families received PMTO, according to the Norwegian statistics, An RCT effectiveness study was conducted with encouraging outcomes. 9/7/2018 © The Norwegian Center for Child Behavioral Development

Multisystemic therapy (MST) MST was implemented in Norway from 2000, in all regions of the country, The number of teams has varied from 22 to 25, A randomized controlled trial was conducted with positive outcomes at post assessment and at follow-up two years after intake. By 2011; 3823 families had gone through therapy, An extensive system of monitoring and quality assurance has been established with regular evaluations and reports to stakeholders. 9/7/2018 © The Norwegian Center for Child Behavioral Development

Background of the study Ten years after the nationwide dissemination of two evidence-based treatment programs (MST & PMTO) in Norway, the status of the implementation components was evaluated in a cross-sectional study. The aim of the study was to pilot a standardized measure of implementation components by examining the factor structure, the reliabilities of the scores, and their association with implementation outcome variables. The aim was also to compare implementation profiles of the two empirically supported programs based on multi informant assessments. 9/7/2018 © The Norwegian Center for Child Behavioral Development

«Implementation drivers» Structured interviews were based on: The Measures of Implementation Components of the National Implementation Research Network Frameworks (Fixsen, Panzano, Naoom, & Blase, 2008), The interviews had eight subscales based on the core implementation components or implementation drivers identified by Fixsen and colleagues (2005) in their extensive review of implementation research. 9/7/2018 © The Norwegian Center for Child Behavioral Development

Core implementation components Performance assessment Decision support data systems Supervision/ coaching Integrated and compensatory Facilitative administration Training Systems intervention Practitioner selection Leadership Fixsen, Naoom, Blase, Friedman, & Wallace, 2005

Implementation drivers Recruitment - practitioner selection - how staff is recruited to work with the program during its initial phases of implementation, and how staff is recruited to help sustain the program over time. Training - activities related to providing information, instruction, or initial skill development to practitioners and other key staff in the implementing organization. Supervision/coaching - how supervisors provide guidance to therapists and practitioners and also how often, where, and how feedback information is collected. Performance assessment - the type, frequency, and method of performance evaluation of the practitioners in relation to their use of the program. Specifically, questions deal with measures of integrity to the method, as well as how often and by whom their performance is evaluated. 9/7/2018 © The Norwegian Center for Child Behavioral Development

Implementation drivers II Decision support data systems - the availability of information through systematic acquisition of data in order to provide feedback to stakeholders, therapists, coaches, and policy makers inside and outside of the organization. Facilitative administration - whether those in charge of implementation in the host organization had restructured and adapted the organization to make implementation and sustainability successful. Systems interventions – the participants’ organizations work to influence the systems and policies in their region to develop better support for the innovation. Leadership - the ways different people within the organization engaged in leadership behavior and provided systematic support, clear communication with practitioners, provided decision-making, garnered feedback, and engaged actively in a manner that was conducive to successful program implementation. 9/7/2018 © The Norwegian Center for Child Behavioral Development

Method and participants Structured interviews were conducted, adapted from the Measures of Implementation Components of the National Implementation Research Network Frameworks (Fixsen, Panzano, Naoom, & Blasé, 2008), 56 MST therapists, their teamleaders (clinical supervisors) (N=21) and managers (N=11) were interviewed, A random selection of 93 PMTO therapists, their supervisors (17) and organizational managers (20) were interviewed. 9/7/2018 © The Norwegian Center for Child Behavioral Development

Outcomes The factor analysis of the eight one-dimensional subscales resulted in an individual clinical-competency factor and an organizational system-level factor. MST and PMTO had different implementation profiles and therapists, supervisors, and managers evaluated some of the implementation drivers significantly differently. The oldest and most experienced therapists who had the highest number of colleagues gave the most negative ratings. 9/7/2018 © The Norwegian Center for Child Behavioral Development

Factor loadings for implementation component scales INDIVIDUAL CLINICAL LEVEL ORGANIZATION AND SYSTEM LEVEL Supervision/coaching .86 Recruitment/selection .78 Performance assessment .74 Facilitative administration .61 Leadership .58 Systems intervention .50 Decision support data system .41 .42 Factor loadings < .40 not included 9/7/2018 © The Norwegian Center for Child Behavioral Development

MST and PMTO therapists compared (mean scores) 9/7/2018 © The Norwegian Center for Child Behavioral Development

Positive ratings Among the therapists, the most positive ratings came from young therapists with shorter professional careers who worked more than 80% on the program and in the company of program colleagues. This included tightly organized MST teams, but also team-based PMTO therapists. Ratings were not influenced by the number of years of experience with the program, a finding that attests to the ability of the programs to sustain therapist engagement. Also, the median number of three years of program experience indicated a considerable stability among program staff. 9/7/2018 © The Norwegian Center for Child Behavioral Development

Mean scores for the informant groups 9/7/2018 © The Norwegian Center for Child Behavioral Development

Competency and organizational components Performance Assessment (Fidelity) Individual COMPETENCY ORGANIZATION Systems Intervention Coaching Training Facilitative administration Integrated & Compensatory Selection Decision support data system Leadership

Implementation quality 10 years after Overall, the respondents reported mean scores at the upper end of the competency scales, Both programs seemed to have established and sustained highly structured and consistent procedures for recruitment, training and supervision. Lower mean ratings were given for the organization scales which were under the control of the local agencies and represented new demands imposed by empirically supported programs. 9/7/2018 © The Norwegian Center for Child Behavioral Development

Implications for sustainability Implementation components at the policy and organizational level is vital for the sustainability of large scale implementation of empirically supported programs; Long term funding and political support Local commitment and accountability Preventing turnover among staff and ongoing training of new therapists are vital for the sustainability of evidence-based programs, ‘ Establish and maintaining quality assurance procedures by monitoring: program and treatment fidelity, “reach”, recruitment and productivity, and prevalence rates, Research on short- and long-term outcome effectiveness, moderators, mediators and implementation.

Open access: http://www.implementationscience.com/content/7/1/49