1. The Importance of Fidelity as a Means for Enhancing and Assessing Program Quality A Case Study of the RESPECT Program

2. What is fidelity? Adherence to program design, both in terms of the amount and quality of what is delivered Whether a program or treatment is being implemented as planned “Methodological strategies used to monitor and enhance the reliability and validity of behavioral interventions.” (Bellg et al., 2004)

3. Why is fidelity important? Ability to justify results An aspect of evidence-based research If you want to replicate or compare your program (Smith et al., 2007) A way to measure AND enhance program quality

4. The RESPECT Program Three-year implementation in four public middle schools in Boston, MA. Adventure approach to social/emotional skill building By the end of third year, all teachers and administrators on board and implementing the program Research and implementation funded primarily though outside sources

5. RESPECT Program Components Full Value Concepts Adventure activities Goal setting Quick Class Assessment tools (Initial teacher/administrative training and ongoing consultation)

6. RESPECT Program outcomes Increased SEL skills Decreased behavioral incidents Increased time on task Increased academic performance

7. RESPECT Program Measures Social Skills Rating Scale (SSRS) Perceived Competency of Functioning Inventory (PCFI) (Self-efficacy) Grades and state-mandated test scores Disciplinary incidents Attendance Compared with two control schools

8. Status of findings Still in progress Generally, second year data shows more promise (6 th grade teachers were into their second year/7 th grade students in Year 2 had received RESPECT for two years) Without accurate data showing level of fidelity to the model harder to state the meaning of the data

9. Factors that influence fidelity Program design – Level of complexity – Level of articulation of program components – Time and materials Staff – Training – Supervision – Level of “buy in” and perception of effectiveness

10. How to measure fidelity Direct observation – Develop protocols for what constitutes evidence of program components Self-report/self-monitor – Develop questionnaires for practitioners to rate themselves on quantity and quality of implementation Interviews

11. References Bellg, A. J., B. Borrelli, et al. (2004). Enhancing Treatment Fidelity in Health Behavior Change Studies: Best Practices and Recommendations From the NIH Behavior Change Consortium. Health Psychology 23(5): 443-451. Gresham, F. M., MacMillan, D. L., Beebe-Frankenberger, M. E., Bocian, K. M. (2000). Treatment Integrity in Learning Disabilities Intervention Research: Do We Really Know How Treatments Are Implemented? Learning Disabilities Research and Practice 15(4), 198-205. Hill, L. G., K. Maucione, et al. (2007). A focused approach to assessing program fidelity. Prevention Science 8(1): 25-34. Lane, K. L., K. M. Bocian, et al. (2004). Treatment Integrity: An Essential--But Often Forgotten--Component of School-Based Interventions. Preventing School Failure, 48(3), 36. Melde, C., F.-A. Esbensen, et al. (2006). Addressing Program Fidelity Using Onsite Observations and Program Provider Descriptions of Program Delivery. Evaluation Review 30(6): 714. Orwin, R. G. (2000). Assessing program fidelity in substance abuse health services research. Addiction 95: S309-s327. Rimm-Kaufman, S.E. & Chiu, Y.I. (2007). Promoting Social and Academic Competence in the Classroom: An Intervention Study Examining the Contribution of the Responsive Classroom Approach. Psychology in the Schools, 44(4). Smith, S. W., A. P. Daunic, et al. (2007). Treatment Fidelity in Applied Educational Research: Expanding the Adoption and Application of Measures to Ensure Evidence-Based Practice. Education and Treatment of Children 30(4ov): 121.