1. Measurement of Implementation Components Ten Years After a Nationwide Introduction of Empirically Supported Programs Society for Prevention Research, 21st annual meeting, The science of prevention: Building a comprehensive National Strategy for Well-Being May 28-31, 2013; Hyatt Regency San Francisco 
Terje Ogden
The Norwegian Center for Child Behavioral Development
Unirand, University of Oslo, Norway
9/7/2018
© The Norwegian Center for Child Behavioral Development

2. Co-authors of the study
Gunnar Bjørnebekk,
John Kjøbli,
Joshua Patras,
Terje Christiansen,
Knut Taraldsen,
Nina Tollefsen
Norwegian Center for Child Behavioral Development,
University of Oslo, P.O. Box 7053, Majorstuen 0306, Oslo, Norway.
9/7/2018
© The Norwegian Center for Child Behavioral Development

3. Empirically supported programs
1998 a national initiative was launched by the Norwegian government in order to increase and improve services, competence and research in relation to children and youth with conduct problems,
1998: A task force appointed by the Norwegian Research Council recommended that empirically supported family and community treatment programs should be implemented and evaluated in Norway,
The Behavior project (1998) was transformed into a permanent center (2003): “The Norwegian Center for Child Behavioral Development”.
9/7/2018
Side 3
© The Norwegian Center for Child Behavioral Development

4. Center for implementation and research
The Norwegian Center for Child Behavioral Development
Program
selection
Long term funding
Evidence based policy
support
RecruitimentTraining
Supervision
Technical
support
Center for implementation and research
Program development and implementation
Technical support
Monitoring of program- and treatment integrity
Multi-allegiance research on:
Outcomes, implementation, mediators and change processes
Sustainability of programs and clincial outcomes

5. Multiple evidence-based model programs
PMTO
Parent Management
Training
MST
Multisystemic
Therapy
PALS
School-wide PBS
model
TIBIR
Early intervention
For Children at
Risk
Functional
Family Therapy
FFT
MST and
Contingency
Management
PMTO for Minority
Families and Children
Multidimensional
Treatment Foster
Care (MTFC)
Multiple evidence-based model programs
9/7/2018
Side 5
© The Norwegian Center for Child Behavioral Development

6. An RCT effectiveness study was conducted with encouraging outcomes.
PMTO was implemented in Norway from 1999, and by 2011, six generations with 383 therapists were trained and certified,
In 2010 a total of 4661 families received PMTO, according to the Norwegian statistics,
An RCT effectiveness study was conducted with encouraging outcomes.
9/7/2018
© The Norwegian Center for Child Behavioral Development

7. Multisystemic therapy (MST)
MST was implemented in Norway from 2000, in
all regions of the country,
The number of teams has varied from 22 to 25,
A randomized controlled trial was conducted with
positive outcomes at post assessment and at
follow-up two years after intake.
By 2011; 3823 families had gone through therapy,
An extensive system of monitoring and quality assurance has been established with regular evaluations and reports to stakeholders.
9/7/2018
© The Norwegian Center for Child Behavioral Development

8. Background of the study
Ten years after the nationwide dissemination of two evidence-based treatment programs (MST & PMTO) in Norway, the status of the implementation components was evaluated in a cross-sectional study.
The aim of the study was to pilot a standardized measure of implementation components by examining the factor structure, the reliabilities of the scores, and their association with implementation outcome variables.
The aim was also to compare implementation profiles of the two empirically supported programs based on multi informant assessments.
9/7/2018
© The Norwegian Center for Child Behavioral Development

9. «Implementation drivers»
Structured interviews were based on:
The Measures of Implementation Components of the National Implementation Research Network Frameworks (Fixsen, Panzano, Naoom, & Blase, 2008),
The interviews had eight subscales based on the core implementation components or implementation drivers identified by Fixsen and colleagues (2005) in their extensive review of implementation research.
9/7/2018
© The Norwegian Center for Child Behavioral Development

10. Core implementation components
Performance assessment
Decision support data systems
Supervision/ coaching
Integrated and compensatory
Facilitative
administration
Training
Systems
intervention
Practitioner selection
Leadership
Fixsen, Naoom, Blase, Friedman, & Wallace, 2005

11. Implementation drivers
Recruitment - practitioner selection - how staff is recruited to work with the program during its initial phases of implementation, and how staff is recruited to help sustain the program over time.
Training - activities related to providing information, instruction, or initial skill development to practitioners and other key staff in the implementing organization.
Supervision/coaching - how supervisors provide guidance to therapists and practitioners and also how often, where, and how feedback information is collected.
Performance assessment - the type, frequency, and method of performance evaluation of the practitioners in relation to their use of the program. Specifically, questions deal with measures of integrity to the method, as well as how often and by whom their performance is evaluated.
9/7/2018
© The Norwegian Center for Child Behavioral Development

12. Implementation drivers II
Decision support data systems - the availability of information through systematic acquisition of data in order to provide feedback to stakeholders, therapists, coaches, and policy makers inside and outside of the organization.
Facilitative administration - whether those in charge of implementation in the host organization had restructured and adapted the organization to make implementation and sustainability successful.
Systems interventions – the participants’ organizations work to influence the systems and policies in their region to develop better support for the innovation.
Leadership - the ways different people within the organization engaged in leadership behavior and provided systematic support, clear communication with practitioners, provided decision-making, garnered feedback, and engaged actively in a manner that was conducive to successful program implementation.
9/7/2018
© The Norwegian Center for Child Behavioral Development

13. Method and participants
Structured interviews were conducted, adapted from the Measures of Implementation Components of the National Implementation Research Network Frameworks (Fixsen, Panzano, Naoom, & Blasé, 2008),
56 MST therapists, their teamleaders (clinical supervisors) (N=21) and managers (N=11) were interviewed,
A random selection of 93 PMTO therapists, their supervisors (17) and organizational managers (20) were interviewed.
9/7/2018
© The Norwegian Center for Child Behavioral Development

14. Outcomes
The factor analysis of the eight one-dimensional subscales resulted in an individual clinical-competency factor and an organizational system-level factor.
MST and PMTO had different implementation profiles and therapists, supervisors, and managers evaluated some of the implementation drivers significantly differently.
The oldest and most experienced therapists who had the highest number of colleagues gave the most negative ratings.
9/7/2018
© The Norwegian Center for Child Behavioral Development

15. Factor loadings for implementation component scales
INDIVIDUAL CLINICAL LEVEL
ORGANIZATION AND SYSTEM LEVEL
Supervision/coaching
.86
Recruitment/selection
.78
Performance assessment
.74
Facilitative administration
.61
Leadership
.58
Systems intervention
.50
Decision support data system
.41
.42
Factor loadings < .40 not included
9/7/2018
© The Norwegian Center for Child Behavioral Development

16. MST and PMTO therapists compared (mean scores)
9/7/2018
© The Norwegian Center for Child Behavioral Development

17. Positive ratings
Among the therapists, the most positive ratings came from young therapists with shorter professional careers who worked more than 80% on the program and in the company of program colleagues.
This included tightly organized MST teams, but also team-based PMTO therapists.
Ratings were not influenced by the number of years of experience with the program, a finding that attests to the ability of the programs to sustain therapist engagement.
Also, the median number of three years of program experience indicated a considerable stability among program staff.
9/7/2018
© The Norwegian Center for Child Behavioral Development

18. Mean scores for the informant groups
9/7/2018
© The Norwegian Center for Child Behavioral Development

19. Competency and organizational components
Performance Assessment (Fidelity)
Individual COMPETENCY
ORGANIZATION
Systems Intervention
Coaching
Training
Facilitative administration
Integrated & Compensatory
Selection
Decision support data system
Leadership

20. Implementation quality 10 years after
Overall, the respondents reported mean scores at the upper end of the competency scales,
Both programs seemed to have established and sustained highly structured and consistent procedures for recruitment, training and supervision.
Lower mean ratings were given for the organization scales which were under the control of the local agencies and represented new demands imposed by empirically supported programs.
9/7/2018
© The Norwegian Center for Child Behavioral Development

21. Implications for sustainability
Implementation components at the policy and organizational level is vital for the sustainability of large scale implementation of empirically supported programs;
Long term funding and political support
Local commitment and accountability
Preventing turnover among staff and ongoing training of new therapists are vital for the sustainability of evidence-based programs, ‘
Establish and maintaining quality assurance procedures by monitoring: program and treatment fidelity, “reach”, recruitment and productivity, and prevalence rates,
Research on short- and long-term outcome effectiveness, moderators, mediators and implementation.

22. Open access: