1. IMPLEMENTATION QUALITY RESEARCH OF PREVENTION PROGRAMS IN CROATIA MIRANDA NOVAK University of Zagreb, Faculty of Education and Rehabilitation Sciences Josipa Mihic, Clemens Hosman, Celene Domitrovich Health Promotion Research Centre 18th Annual Summer Conference National University of Ireland Galway

2. MOTIVATION FOR RESEARCH o Although the efficacy of various evidence-based interventions has been established through carefully designed trials in control conditions, there is a lack of evidence for its utilization in natural community conditions (Kam, Greenberg and Weiss, 2003; Greenberg et al., 2005; Fixsen et al., 2005, Proctor and Rosen, 2008) o A more systematic process is warranted to translate efficacy results into positive participants’ outcomes SCIENCE TO PRACTICE CONTEXT FACTORS AND SUPPORT SYSTEM CHARACTERISTICS OF PROGRAM IMPLEMENTATION RESEARCH

3. DEFINITION OF IMPLEMENTATION o implementation refers to what a program consist of when it is delivered in a particular setting (Durlak and Dupre, 2008) o characteristics of intervention itself and characteristics of the intervention support system o it is important to differentiate between factors affecting implementation quality (Fixsen et al., 2005, 2009) and implementation aspects (Durlak and Dupre, 2008)

4. WHY WE MONITOR IMPLEMENTATION To know what actually happened To provide feedback for quality To strengthen the conclusions made about outcomes To examine the change process To understand the internal dynamics of program To advance knowledge for replication To strengthen the quality of evaluation

5. ORGANIZATIONAL CAPACITY ATTITUDES TOWARDS THE INTERVENTION TRAINING AND KNOWLEDGE SUPPORT FOR IMPLEMENTER MONITORING SYSTEM CHARACTERISTICS OF A PROGRAM IMPLEMENTER'S SKILL PROGRAM STANDARDIZATION ATTITUDES TOWARDS THE INTERVENTION IMPLEMENTATION FACTORS - CAPACITY FOR PROGRAM IMPLEMENTATION IMPLEMENTATION QUALITY FIDELITY PARTICIPAN T RESPONSIVE NESS DOSAGE QUALITY OF PROGRAM DELIVERY PERCEIVED PROGRAM IMPACT INDICATORS OF IMPLEMENTATION QUALITY

6. Research project »Preffi – Quality assurance in the Region of Istria« Department of Health and Social Care, Region of Istria and Faculty of Education and Rehabilitation Sciences, University of Zagreb Incorporation of science-based principles in prevention practice in Istria through collaboration of scientists and practitioners

7. TRAINING PROGRAM Principles of science-based practice Logic modelling  Implementatio  Evaluation  Advocacy Interactive group education and Individual consultation ORGANIZATION MANAGERS PROGRAM AUTHORS PROGRAM DELIVERERS

8. The general aim of this research was to study implementation processes and their outcomes in prevention programs in Croatia.

9. 1.To construct valid and reliable measures of implementation quality. 2.To explore the level and variation of implementation quality 3.To explore the differences in perception of implementation quality between program managers, implementers and participants. 4.To explore the relationships of implementation factors and indicators of implementation quality. 5.To test the impact of the Training for Prevention on the level of implementation quality in experimental group

10. SAMPLE: 24 mental health promotion and prevention programs MANAGERSIMPLEMENTERS IMPLEMENTATION FACTORS Standardization Implementers’ skills Attitudes Training Support Monitoring Standardization Attitudes Training Support Monitoring IMPLEMENTERSPARTICIPANTS INDICATORS OF IMPLEMENTATION QUALITY Fidelity Quality of delivery Responsiveness Perceived program impact Dosage Quality of delivery Responsiveness Perceived program impact Parenting Positive development and SEL Substance abuse

11. MEASURES o Implementation Factors Questionnaire for Program Managers o Implementation Factors Questionnaire for Program Implementers o Indicators of Implementation Quality Questionnaire for Program Implementers o Indicators of Implementation Quality Questionnaire for Program Participants o The Preffi 2.0 instrument

12. June 2010 - December 2010 - January 2011 February 2011 - February 2011 - January 2011 January 2011 March 2011 December 2011 3 assessors independently assess 24 written programs’ proposals Division 24 programs : experimental (N=12) and control condition (N=12) EXPERIMENTAL Training for Prevention to experimental condition Mid- Assessment implementation quality Focus groups Evaluate Initial set of items 19 PROGRAM MANAGERS 50 PROGRAM DELIVERERS 434 PROGRAM PARTICIPANTS CONSTRUCTION OF SCALES FOR IMPLEMENTATION QUALITY ASSESSMENT Implementation of 24 programs Post- Assessment Implementation quality

13. June 2010 - December 2010 - January 2011 February 2011 - February 2011 - January 2011 January 2011 March 2011 December 2011 3 assessors independently assess 24 written programs’ proposals Division 24 programs : experimental (N=12) and control condition (N=12) Mid- Assessment implementation quality Focus groups Evaluate Initial set of items Implementation of 24 programs Post- Assessment Implementation quality STUDY OF IMPLEMENTATION QUALITY 19 PROGRAM MANAGERS 50 PROGRAM DELIVERERS 434 PROGRAM PARTICIPANTS EXPERIMENTAL Training for Prevention to experimental condition

14. 19 PROGRAM MANAGERS 50 PROGRAM DELIVERERS 434 PROGRAM PARTICIPANTS June 2010 - December 2010 - January 2011 February 2011 - February 2011 - January 2011 January 2011 March 2011 December 2011 3 assessors independently assess 24 written programs’ proposals Division 24 programs : experimental (N=12) and control condition (N=12) Mid- Assessment implementation quality Focus groups Evaluate Initial set of items Implementation of 24 programs Post- Assessment Implementation quality STUDY ON THE TRAINING FOR PREVENTION IMPACT 22 PROGRAM MANAGERS 55 PROGRAM DELIVERERS 744 PROGRAM PARTICIPANTS EXPERIMENTAL Training for Prevention to experimental condition

15. RESULTS

16. STUDY OF IMPLEMENTATION QUALITY

17. Reports on implementation factors across 24 programs show that views of managers and programs implementers are rather similar MAN: Lowest ratings for standardization, training and monitoring IMP: Lowest ratings for standardization and training

18. Indicators of implementation quality are really high – group results all over value of 3 Overall group results show that program implementers give similar although higher ratings than participants

19. STUDY OF THE TRAINING FOR PREVENTION IMPACT

20. Levels of implementation factors reported by managers and the effect of the Training for Prevention at post-intervention POST INTERVENTIONINTERVENTION EFFECT CONTINT BETA (SE) PEFFECT SIZE Standardization M SD 2,89 0,43 2,05 0,81 -0.84 (0.28) <.001** -1.35 Implementers’ skills M SD 3,61 0,47 3,76 0,39 0.15 (0.18) 0.420.35 Attitudes M SD 2,97 0,37 2,74 0,46 -0.23 (0.18) 0.21-0.55 Training M SD 2,95 0,72 2,39 0,50 -0.56 (0.27).05*-0.92 Support M SD 3,47 0,63 3,25 0,40 -0.21 (0.22).36-0.42 Monitoring M SD 3,14 0,70 2,35 0,29 -0.79 (0.22).01** -1.59 N11 Managers from experimental conditions give more critical answers about implementation factors than those from control = CHANGE OF CRITERIA

21. Levels of implementation factors reported by implementers and the effect of the Training for Prevention at post-intervention POST INTERVENTIONINTERVENTION EFFECT CONTINT BETA (SE) PEFFECT SIZE Standardization M SD 3,10 0,54 2,85 0,59 -0.34 (0.23) 0.15-0.43 Attitudes M SD 3,04 0,41 2,83 0,29 -0.13 (0.13) 0.31-0.59 Training M SD 3,07 0,66 2,85 0,59 -0.22 (0.17) 0.21-0.36 Support M SD 3,44 0,47 3,49 0,69 0.07 (0.25) 0.770.09 Monitoring M SD 2,97 0,57 2,80 0,60 -0.38 (0.25) 0.14-0.29 N3322 Implementers from experimental conditions give more critical answers about implementation factors than those from control = CHANGE OF CRITERIA

22. Levels of implementer’s ratings on indicators of implementation quality and the effect of the Training for Prevention at post-intervention POST INTERVENTIONINTERVENTION EFFECT CONTINT BETA (SE) PEFFECT SIZE Fidelity M SD 3,29 0,37 3,22 0,36 -0.02 (0.13) 0.85-0.19 Quality M SD 3,55 0,40 3,49 0,36 -0.03 (0.14) 0.82-0.16 Responsiveness M SD 3,36 0,41 3,49 0,36 0.12 (0.15) 0.450.34 Perceived program impact M SD 3,34 0,48 3,25 0,48 0.02 (0.19) 0.93-0.19 N3322 At post-test, program implementers from the experimental group did not report improved indicators of implementation quality in comparison with the control group

23. Levels of participant’s ratings on indicators of implementation quality and the effect of the Training for Prevention at post-intervention POST INTERVENTIONINTERVENTION EFFECT CONTINTBETAPEFFECT SIZE Dosage M SD 92,33 18,00 99,45 2,13 6.96 (4.41).130.71 Quality M SD 3,43 0,52 3,64 0,34 0.17 (0.07).02* 0.49 Responsiveness M SD 3,09 0,67 3,50 0,41 0.26 (0.12).04*0.76 Perceived program impact M SD 2,94 0,74 3,23 0,62 0.19 (0.12).14 0.43 N391353 At post-test, participants from the experimental group report on higher levels of implementation quality, two of them being significant - quality of delivery and responsiveness.

24. Moderator analyses have shown that Training for Prevention is more effective for SHORT PROGRAMS

25. Moderator analyses have shown that Training for Prevention is more effective for PROGRAMS WHERE MANAGER IS NOT ACTIVE

26. CRUCIAL CONDITIONS FOR SUCCESFUL IMPLEMENTATION

27. Investment in implementation factors ORGANIZATIONAL CAPACITY ATTITUDES TOWARDS THE INTERVENTION TRAINING AND KNOWLEDGE SUPPORT FOR IMPLEMENTER MONITORING SYSTEM CHARACTERISTICS OF A PROGRAM IMPLEMENTER'S SKILL PROGRAM STANDARDIZATION ATTITUDES TOWARDS THE INTERVENTION

28. THANK YOU FOR YOUR ATTENTION mnovak@erf.hr