1. DAY 4 Special Topics in Single-Case Design Intervention Research Tom Kratochwill

2. Goals of the Presentation
Review general measurement issues in single-case design research;
Feature the importance of intervention integrity monitoring and promotion;
Feature the importance of intervention intensity;
Discuss features and challenges of social validity assessment in intervention research.

3. General Considerations in Measurement

4. Choice of the Dependent Variable
A major issue in single-case design intervention research is that dependent variable assessment must be repeated across time;
Traditional norm-referenced measurement does not easily lend itself to repeated assessment (sometimes called indirect assessment; e.g., self-report checklists and rating scales, informant report checklists and rating scales);
Specification of the conditions of measurement should be noted (e.g., natural versus analogue, human observes versus automated recording, etc.).
Shapiro, E. S., & Kratochwill, T. R. (Eds.) (2000). Conducting school-based assessments of child and adolescent behavior. New York: Guilford.

5. Measurement Systems Commonly Used in Single-Case Design Research
Automated Recording (e.g., galvanic skin response, heart rate, blood pressure);
Permanent Products (e.g., academic responses in math, reading, spelling, number of clothes items on the floor);
Direct Observation (e.g., assessment of behavior as it occurs as noted by human observers, video recording, computer-based/phone-based apps or systems).

6. Quality of Assessment
Quality of assessment is typically determined by reliability and validity of measurement.
Some clarification of terms for reliability:
Reliability of effect (usually established through replication of the intervention in the same or repeated investigation);
Reliability of the procedures (sometimes referred to as intervention fidelity of integrity).
Reliability of measurement (usually determined by assessment agreement measures and required in the WWC Pilot Standards);

7. Remember WWC Standard 2: Inter-Assessor Agreement
Each Outcome Variable for Each Case Must be Measured Systematically by More than One Assessor.
Researcher Needs to Collect Inter-Assessor Agreement:
In each phase
On at least 20% of the data points in each condition (i.e., baseline, intervention)
Rate of Agreement Must Meet Minimum Thresholds:
(e.g., 80% agreement or Cohen’s kappa of 0.60)
If No Outcomes Meet These Criteria, Study Does Not Meet Design Standards.

8. Observer Agreement in the WWC Pilot Standards
The WWC Panel not specify the method of observer agreement (e.g., using only measures that control for chance agreement).
Hartmann, Barrios, and Wood (2004) reported over 20 measures to assess interobserver agreement.
A good practice is to report measures that control for chance agreement and report agreement for occurrences and non-occurrences of the dependent variable.
Good to assess accuracy: the degree to which observer data reflect actual performance or some true measure of behavior (e.g., video performance of the behavior).

9. Artifact, Bias, and Complexity of Assessment
Researcher needs to consider factors that can obscure observer agreement:
Direct assessment by observers can be reactive;
Observers can drift in their assessment of the dependent variable;
Observers can have expectancies for improvement that are conveyed by the researcher;
Coding systems can be too complex and lead to error.

10. Intervention Integrity

11. Intervention Integrity: A Developing Standard
Assessing (and especially promoting) intervention or treatment Integrity is a more recent and developing consideration in intervention research (see Sanetti & Kratochwill, 2014).
Sanetti, L. M., & Kratochwill, T. R. (Eds.) (2014). Treatment Integrity: A Foundation for Evidence-Based Practice in Applied Psychology. Washington, DC: American Psychological Association.

12. Representative Features of Intervention Integrity Data
Across Design Intervention Phases
Across Therapists/Intervention Agents
Across Situations
Across Sessions
Across Cases

13. Where does intervention integrity fit?
1. Screening/ assessment data suggest prevention/intervention is warranted
Where does intervention integrity fit?
2. Evidence-based intervention selected & implemented
3. Program outcomes (SO) assessed
3. Intervention integrity (II) assessed
Data reviewed
5a. Continue intervention
+ SO
+ II
4. Data-based decisions
5b. Continue intervention / promote intervention integrity
+ SO - II
-SO
- II
5c. Implement intervention strategies to promote intervention integrity
- SO
+ II
5d. Change intervention

14. Treatment Integrity: Definition
“Fidelity of implementation is traditionally defined as the determination of how well an intervention is implemented in comparison with the original program design during an efficacy and/or effectiveness study” (O’Donnell, 2008, p.33)
Five distinct dimensions of intervention integrity were proposed: adherence, exposure, quality of delivery, participant responsiveness, and program differentiation (Dane & Schneider, 1998)
Treatment integrity encompasses three different aspects: treatment adherence, therapist competence, and treatment differentiation (Waltz, Addis, Koerner, &Jacobson, 1993).
The degree to which treatment is delivered as intended (Yeaton & Sechrest, 1981)
Treatment integrity is the extent to which required intervention components are delivered as designed in a competent manner while proscribed procedures are avoided by an interventionist trained to deliver the intervention in a particular setting to a particular population (Perepletchikova, 2014).
Implementation is defined as a specified set of activities designed to put into practice an activity or program of known dimensions (Sanetti & Kratochwill, 2009)
“The degree to which an intervention program is implemented as planned’’ (Gresham et al., 1993, p. 254).
The ‘‘extent to which patients follow the instructions that are given to them for prescribed treatments’’ (Haynes et al., 2002, p. 2).
“Implementation is defined as a specified set of activities designed to put into practice an activity or program of known dimensions” (Fixsen et al. 2005, p.5).

15. Intervention Integrity Data Definition of Intervention
Intervention Integrity in Research [Sanetti, L. M. H., Gritter, K. L., Dobey, L. M. (2011). Treatment integrity of interventions with children in the school psychology literature from 1995 to School Psychology Review, 40, 72-84].
Intervention Integrity Data
Definition of Intervention

16. Intervention Integrity Assessment in School Psychology Practice [Cochran, W. S., & Laux, J. M. (2008). A survey investigating school psychologists' measurement of treatment integrity in school‐based interventions and their beliefs about its importance. Psychology in the Schools, 45, ]

17. Implementation Supports
Negative reinforcement
Intervention manual
Test driving interventions
Expert consultation
Intervention scripts
Video support
Intervention choice
Performance feedback
Implementation planning
Classroom Check Up
Motivational interviewing
Instructional coaching
Participant modeling
Treatment planning protocol
Role play
Prompts
Collaborative consultation
Self-monitoring
Direct training
Sanetti & Collier-Meek (2014)

18. Perspectives on intervention integrity
Performance Feedback
(Noell et al., 1997, 2002, 2005)
Negative Reinforcement
(DiGennaro et al., 2005, 2007)
Behavior Performance Deficit
/ text/skype cues

19. Contributions from related fields
Psychology
Education
Contributions from related fields
Medicine
Health Psychology
Prevention Science
Intervention integrity is more complicated than a behavior performance deficit.

20. Shifting conceptualization of treatment integrity
Prevention/Intervention programs require implementers to change their behavior
Implementing interventions with a high level of integrity can be considered an adult behavior change process.

21. Mental Health and Education Related Fields
Health Action Process Approach(HAPA)
Theory of Reasoned Action
Transtheoretical Approach
Behavioral theory

22. Focus on the HAPA Looked for a theory that…
Is conceptually clear and consistent,
Is parsimonious but would enable us to describe, explain, and predict behavior change.
Explicates behavior performance, not just development of behavioral intention.
Enabled us to design effective interventions that would produce change in the predicted behaviors.
Has empirical support and practical utility.

23. Breast self-examination
Empirical Support
Healthy eating
Exercise Behaviors
Breast self-examination
Seat belt use
Dental flossing
See Schwarzer, R.( 2008) Modeling health behavior change: How to predict and modify the adoption and maintenance of health behaviors. Applied Psychology, 57,

24. Project PRIME Planning Realistic intervention Implementation and
Maintenance by
Educators
Funded by the Institute for Education Sciences, U.S. Department of Education (R324A100051)

25. Project PRIME
PRIME was designed originally to prevent teachers’ level of intervention implementation from declining.
Developed a system of supports to facilitate teachers’ intervention implementation.
Delivered through a problem-solving consultation model (Bergan & Kratochwill, 1990; Kratochwill & Bergan, 1990).

26. Three components of PRIME:
Implementation Planning:
A detailed guide walks consultants and consultees through the process of
Action planning
Coping planning

27. Action Planning
Specifies how and under what circumstances an intended intervention action is to be completed
Intention to implement
Initiation of implementation

28. Action Planning Continued…
Completion of Action Plan helps to
Define the intervention components and steps
All aspects of the intervention are accounted for by a component (e.g., a social skills lesson)
Each step corresponds to behavior(s) performed to implement an intervention component (e.g., introducing the topic of the social skills lesson)
Detail logistical planning on implementation
Identify potential resource barriers

29. Action Planning Continued…
Logistical planning questions answered for each implementation step include:
Where will the step be implemented?
How often will the step be implemented?
For how long will the step be implemented?
What resources (materials, time, space, personnel, etc.) are needed to complete this step?
Throughout implementation Action Plan can be updated to reflect changes in implementation

30. Coping Planning Completion of Coping Plan helps to:
Identify up to 4 of the most significant barriers to intervention implementation
Develop “coping” strategies to address identified barriers
Barriers to implementation are listed
e.g., major changes in class schedule due to statewide testing
Throughout implementation the coping plan is updated to reflect changes in implementation

31. Three components of PRIME:
2. Implementation Beliefs Assessment (IBA)
Assesses implementer’s
Outcome expectancies
Self-efficacy
Implementation
Maintenance
Recovery

32. Three components of PRIME:
3. Strategies to Increase Implementation Intention and Self-efficacy
Eight strategies have been identified. Detailed guides walk consultants through strategy implementation.
Strategies include:
Participant modeling
Role play

33. Multi-Tiered Implementation Supports (MTIS)
Intensive strategies that typically require ongoing support
Tier 3
Tier 2
Tier 1
Selected strategies based on the implementer’s intervention integrity and stage of learning
Feasible and widely relevant implementation support strategies that can be easily embedded into typical programs

34. MTIS: Tier 1 Strategies Tier 2 Tier 1 Tier 3 Intervention manual
Intervention scripts
Collaborative/Expert consultation
Instructional coaching
Intervention choice
Test driving interventions
Direct training
Tier 3
Tier 2
Tier 1

35. MTIS: Tier 1 & 2 Strategies
Implementation Planning
Treatment planning protocol
Tier 3
Tier 2
Tier 1

36. MTIS: Tier 2 Strategies Tier 2 Tier 1 Tier 3 Role play
Participant modeling
Tier 3
Tier 2
Tier 1

37. MTIS: Tier 2 & 3 Strategies
Motivational interviewing
Self-monitoring
Prompts
Video support
Tier 3
Tier 2
Tier 1

38. MTIS: Tier 3 Strategies Tier 2 Tier 1 Tier 3 Performance feedback
Performance feedback with negative reinforcement
Tier 3
Tier 2
Tier 1

39. Take away messages
Intervention integrity is important, and cannot be assumed in prevention/intervention program implementation;
Intervention integrity can be conceptualized as a behavior change process;
Related fields can inform prevention/ intervention science practice in psychology and education;
Multiple strategies implemented in a tiered model may be necessary to promote intervention integrity.

40. Intervention Intensity

41. Intervention Intensity: Definition and Dimensions
Intervention intensity typically refers to intervention strength and traditionally, the most commonly recognized dimension is dose of the intervention (Codding & Lane, 2015).
Coding, R. S. & Lane, K. L. (2015). A spotlight on treatment intensity: An important and often overlooked component of intervention inquiry. Journal of Behavioral Education, 24, 1–10

42. Dimensions of Intervention Intensity
Dose
Learning trial/session
Session length
Session frequency
Length of intervention
Cumulative intensity
Group Size

43. Dimensions of Intervention Intensity (Continued)
Intervention Design
Form
Positive corrective feedback
Pace of instruction
Opportunities to practice or respond
Transitions between subjects or classes
Goal specificity
Intervention complexity

44. Dimensions of Intervention Intensity (Continued)
Interventionist Characteristics
Experience, knowledge, skills
Implementation Characteristics
Intervention management and planning
Materials and tangible resources required
Deviation from classroom routines

45. Social Validity in Single-Case Design Research

46. Social Validity in Intervention Research
Social Validity involves three questions about an intervention (Kazdin, 2011):
Are the goals of the intervention relevant to the person’s life?
Is the intervention acceptable to “consumers” and others involved in the procedures?
Are the outcomes of the intervention important (i.e., do changes make a difference in lives of persons involved)?
Kazdin, A. E. (2011). Single-case research designs: Methods for clinical and applied settings. New York:
Oxford.

47. Social Validity in Intervention Research
Three social validation methods can be used in intervention research:
Social Comparison
Subjective Evaluation
Sustainability of the Intervention

48. Social Validity in Intervention Research
Social Comparison
Normative social comparison data can be used when such information provides a good benchmark for positive functioning;
Typically social comparison involves identification of a “peer group” similar to the client;
The peer group would consist of persons who are functioning in an adequate or positive manner;
Sometimes standardized assessment instruments can be used for social comparison purposes (e.g., checklists and rating scales).

49. Social Validity in Intervention Research
Some challenges with social comparison:
It is not so easy to establish normative comparisons;
Normative comparisons may be unrealistic;
Normative range of functioning may be an impossible goal given the level of impairment;
Normative goals may not reflect overall quality of life;
Standardized instruments used for normative comparisons may not provide accurate data (e.g., checklists and rating scales; Reid & Maag, 1994).
Reid, R., &Maag, J. W. (1994). How many fidgets in a pretty much: A critique of behavior rating scales for identifying students with ADHD. Journal of School Psychology, 32,

50. Social Validity in Intervention Research
Subjective Evaluation
Involves the assessment by significant others who have knowledge of the client and can make a judgment of the need for intervention and the outcomes of intervention;
Individuals rendering a “subjective” judgment may be parents, teachers, or professionals who have expert status in an area;
Specific goals may be established and serve as the basis for intervention and a benchmark for outcome evaluation.

51. Social Validity in Intervention Research
Some challenges with subjective evaluation:
Global ratings may be biased;
Persons completing the rating (s) may perceive a small change as major when in fact, not much of significance has occurred;
Subjective ratings may not correspond to other outcome data (e.g., direct observation).

52. Social Validity in Intervention Research
Sustainability
The degree to which the effects of a intervention are sustained over time (Kennedy, 2005). The assessment is a measure of how long the program stays in place or is adopted.
Kennedy, C. H. (2005). Single-case designs for educational research. Boston: Allyn and Bacon.

53. Social Validity in Intervention Research
Some Challenges with sustainability
Requires a long time to conduct the analysis;
Factors other than consumer acceptance may influence ratings/evaluation of sustainability;
Sustainability is an indirect index of social validity.

54. Questions and Discussion