1. Toward a Technology of Treatment Integrity Ronnie Detrich Wing Institute APBS, Denver, Colorado 2011

2. Goals for Today Review the link between data-based decision making and treatment integrity. Discuss the dimensions of treatment integrity. Review methods for assuring treatment integrity.

3. Data-based Decision Making and Treatment Integrity Data-based decision making at heart of PBS and RtI. The impact of PBS and RtI depends on the effectiveness of specific interventions. The effectiveness of interventions is a function of the integrity with which they are implemented. The quality of decisions about effects of an intervention is directly linked to the quality of implementation.

4. Data-based Decision Making and Treatment Integrity Student data provides feedback about progress. If we know about adequacy of treatment integrity then can make decisions:  Adequacy of intervention  Adequacy of implementation  If implementation is inadequate then focus should be on improving implementation.  If implementation is adequate then focus should be on changing intervention so student can succeed.

5. Grade Level Standard Aim Line Trend Line Data-based Decision Making and Treatment Integrity

6. Positive Negative High Low Continue InterventionChange Intervention Unknown reason Intervention problem? Implementation problem? Other life changes? Unknown intervention? Intervention is effective? Outcome Integrity Positive Negative High Low

7. Assumptions about Treatment Integrity Students cannot benefit from interventions they do not experience. Unless educators know what they are supposed to be doing they cannot do more of it.

8. Dimensions of Treatment Integrity (Dane & Schneider, 1998) Exposure (Dosage): the extent to which participants are exposed to the intervention as prescribed.  Curricula usually prescribe frequency and duration of exposure that is necessary for benefit.  Ex: 3/week for 30 minutes/session.  Failing to satisfy either can impact student benefit.  Ex: 1/week for 30 minutes.

9. Dimensions of Treatment Integrity (Dane & Schneider, 1998) Adherence: the extent to which the components of an intervention are delivered as prescribed.  Most commonly measured dimension.  It is necessary but not sufficient to produce benefits.  Adherence with low dosage not likely to produce positive outcomes.

10. Dimensions of Treatment Integrity (Dane & Schneider, 1998) Quality of delivery: qualitative measure of how well the intervention is implemented.  Importance has been acknowledged for years.  Have not developed good measures or how to influence it.  Possible measures through social validity methods:  Enthusiasm  Sincerity  Variations in inflection and content of speech.

11. Variables that Influence Treatment Integrity Organizational  Systems that are perceived by teachers to be supportive and provide strong leadership have higher levels of integrity. Personal characteristics  Burnout: Inverse relation between measures of teacher burnout and treatment integrity.  Personal efficacy: Positive correlation between measures of personal efficacy and treatment integrity.

12. Variables that Influence Treatment Integrity Characteristics of Intervention  Complexity  As complexity increases integrity decreases.

13. Relationship between Complexity, Precision, and Treatment Integrity Be as precise as necessary but no more. complexity precision Catch’em being good Check in Check out Individualized support plan Integrity

14. Variables that Influence Treatment Integrity Characteristics of Intervention  Perceived effectiveness.  Acceptability.  Contextual fit.  Necessary resources available.

15. Variables that Influence Treatment Integrity Training  Training is necessary but not sufficient.  Not all training is equal.

16. OUTCOMES (% of Participants who demonstrate knowledge, demonstrate new skills in a training setting, and use new skills in the classroom) TRAINING COMPONENTS Knowledge Skill Demonstration Use in the Classroom Theory and Discussion 10% 5%0%..+Demonstration in Training 30%20%0% …+ Practice & Feedback in Training 60% 5% …+ Coaching in Classroom 95% Joyce and Showers, 2002 Not All Training is Equal

17. What We Know About Treatment Integrity It is estimated that drug prevention programs are implemented with integrity only 19% of the time. (Hallfors & Godette, 2002)  Programs have extensive research base as being effective.  This may be a generous estimate. No reason to assume that other programs are immune to poor implementation.

18. What We Know About Treatment Integrity Most of what we know has been developed at the level of individual student support plans. SET is a measure of treatment integrity at the school level.  Does not address what is happening at the level of the individual classrooms or individual students.

19. What Do We Know About Treatment Integrity and Student Behavior? Different levels of integrity result in different levels of student behavior. (Wilder, Atwell, & Wine, 2006) High integrity followed by declines in integrity has limited disruptive effect on student behavior. (Northup, Fisher, Kahng, Harrel, & Kurtz, 1997) Low levels of integrity followed by increases in integrity do not produce the same level of student response as when integrity high from the beginning. (Groskreutz,, Groskreutz, & Higbee, 2011)

20. What Do We Know About Treatment Integrity and Student Behavior? Implications  Make sure that integrity is high at the beginning of intervention.  It is better to start with high levels of integrity and let it decline than to start with low integrity and try to increase it.  Maximizes impact of intervention.

21. What Do We Know About Integrity of Interventions at the Universal Level? Kovaleski, Gickling, Morrow, & Swank (1999)  Evaluated high vs low implementation of Instructional Support Teams (IST).  School-wide organizational change.  Students benefited from IST processes only when implemented with high fidelity.  Implementing with low fidelity resulted in no better outcomes for students than control group not exposed to IST processes.  Having structures in place was not sufficient to assure high fidelity.  Fidelity assessed one time per year.

22. What Do We Know About Integrity of Interventions at the Universal Level? Horner (2005)  Effect of high fidelity vs low fidelity on office discipline referrals.  Schools that implemented with high fidelity had 25% fewer office referrals for major rule violations than schools that did not meet fidelity criterion.  Fidelity measures taken 2 times per year.

23. How Do We Assure High Levels of Integrity? Technology for assuring treatment integrity is emerging. Most of the research is at level of individual support plans. Feedback is most common approach.

24. Mortenson & Witt, 1998

25. Effective Performance Feedback More frequent the feedback better effects (Jones, Wickstrom, & Friman, 1997Mortensen & Witt, 1998). Daily better than weekly. Immediate better than delayed. Immediate more preferred than delayed.

26. Limitations of Performance Feedback Requires direct observation. May be too resource intensive to implement at large scale.  Would require significant restructuring to implement effectively.

27. Alternative Approaches: Teacher Self Report Teacher rate their own implementation.  Teacher ratings tend to overestimate accuracy of implementation ( Wickstrom, Jones, Lafleaur, & Witt).  Teachers rated integrity at 54% accurate.  Direct observation= 4%.  Integrity may be increased by rating immediately following intervention session.  Integrity may be improved by having teacher score video tape.  It may be possible to teach teachers to rate more accurately (see self-evaluation literature).

28. Alternative Approaches: Quizzes Quizzes (Detrich et al., 2001)  Staff quizzed weekly on elements of multi-component individualized behavior support plans.  Given feedback on quiz but no feedback on actual implementation of support plan.  4 versions of the quiz. One question per element of the plan (student preferences, antecedent interventions, teaching replacement behavior, responding to misbehavior).

30. Alternative Approaches: Quizzes Advantages:  Easier to implement than direct observation.  May be easy to implement at scale when standard protocols are utilized. Limitations  Does not produce maintenance effects in the absence of quizzes.

31. Alternative Approaches: Video Modeling Has been utilized to train a standard protocol as well as individualized interventions.  Problem solving curriculum.  Discrete trial instruction.  Functional assessment.

32. Alternative Approaches: Video Modeling Advantages:  For standard protocols can be used to train many implementers.  Demonstration of implementation can be standardized.  Flexible scheduling of training. Limitations  Has not been evaluated at large scale.  Coaching resources required to assure modeling was effective.

33. Alternative Approaches: Checklists Checklists make sure we get the routine things right. Gawande (2009) Widely used in airlines industry.  Becoming more common in medicine. Specify what to do when.  Before instructional session.  During session.  Following session. Most effective if require some type of active confirmation that step has occurred.

34. Alternative Approaches: Checklists Advantages:  Standardize implementation.  Increase integrity especially adherence. Limitations:  Upfront effort to develop.  Often low acceptability.

35. Alternative Approaches: Intensive Professional Development University faculty/coach assigned to school for full year (Klingner, Vaughn, Hughes, & Arguelles, 1999).  Provided in-depth training on three reading programs.  Classroom based coaching.  Problem solving implementation.  Produced moderate levels of treatment integrity at three years follow-up.

36. Alternative Approaches: Intensive Professional Development Advantages:  Highly trained resource on site. Limitations  Difficult to replicate at large scale.  Expensive.

37. Unresolved Issues Adoption or adaptation?  Must we implement exactly as prescribed or can we adjust to fit local circumstances? Research suggests that programs are almost always adapted. Presumably adapted to improve outcomes.  Some adaptation for other reasons:  Better fit teaching style.  Do not like some elements of program.

38. Unresolved Issues What does this mean for treatment integrity?  Are adaptations systematic?  If so, then we can assess integrity.  What parts of program can we adapt without doing harm to effectiveness?  Core elements?  If adapted is program still research-based?  If teachers allowed to adapt then program more acceptable.  Teacher’s made better adaptation of reading programs if they were well grounded in principles. (Klingner, Vaughn, Hughes, & Arguellas, 1999).

39. Treatment Integrity and PBS Regardless of the level of the intervention, it is necessary to know that it was implemented with integrity.  High integrity is necessary in a data based decision making approach. Integrity should be assessed at the same level that the intervention is being evaluated.

40. Treatment Integrity and PBS A program or intervention is a set of protocols that guides behavior of the adult.  If protocols are not followed then by definition the program has not been implemented or sustained. PBS is an excellent model for making decisions about when, where, and how to intervene.  Intervention without process for assuring integrity is likely to result in wasted effort.

41. Taking Treatment Integrity Measurement to Scale: Data-based Decision Making Insufficient resources to frequently observe all educators working with students to determine adequacy of implementation. One solution: Measure student behavior and analyze at different units of analysis:  School  Classroom  Individual student Any level there is gap between obtained and expected performance assess levels of treatment integrity.

42. Taking Treatment Integrity Measurement to Scale: Data-based Decision Making Requires that data are routinely collected across all levels.  RtI:  CBM on all students 3/year.  PBS  Continuous measurement of Office Discipline Referrals. Systems have to be in place to assure integrity data- based decision making process.

43. Where are We? Implementation is where good interventions go to die unless there is active plan to assess and influence integrity. Research-based approaches to influencing treatment integrity are emerging. We have ethical obligation to allocate resources to influencing treatment integrity.

44. Thank you Copies may be downloaded at winginstitute.org