DAY 4 Special Topics in Single-Case Design Intervention Research Tom Kratochwill

Goals of the Presentation Review general measurement issues in single-case design research; Feature the importance of intervention integrity monitoring and promotion; Feature the importance of intervention intensity; Discuss features and challenges of social validity assessment in intervention research.

General Considerations in Measurement

Choice of the Dependent Variable A major issue in single-case design intervention research is that dependent variable assessment must be repeated across time; Traditional norm-referenced measurement does not easily lend itself to repeated assessment (sometimes called indirect assessment; e.g., self-report checklists and rating scales, informant report checklists and rating scales); Specification of the conditions of measurement should be noted (e.g., natural versus analogue, human observes versus automated recording, etc.). Shapiro, E. S., & Kratochwill, T. R. (Eds.) (2000). Conducting school-based assessments of child and adolescent behavior. New York: Guilford.

Measurement Systems Commonly Used in Single-Case Design Research Automated Recording (e.g., galvanic skin response, heart rate, blood pressure); Permanent Products (e.g., academic responses in math, reading, spelling, number of clothes items on the floor); Direct Observation (e.g., assessment of behavior as it occurs as noted by human observers, video recording, computer-based/phone-based apps or systems).

Quality of Assessment Quality of assessment is typically determined by reliability and validity of measurement. Some clarification of terms for reliability: Reliability of effect (usually established through replication of the intervention in the same or repeated investigation); Reliability of the procedures (sometimes referred to as intervention fidelity of integrity). Reliability of measurement (usually determined by assessment agreement measures and required in the WWC Pilot Standards);

Remember WWC Standard 2: Inter-Assessor Agreement Each Outcome Variable for Each Case Must be Measured Systematically by More than One Assessor. Researcher Needs to Collect Inter-Assessor Agreement: In each phase On at least 20% of the data points in each condition (i.e., baseline, intervention) Rate of Agreement Must Meet Minimum Thresholds: (e.g., 80% agreement or Cohen’s kappa of 0.60) If No Outcomes Meet These Criteria, Study Does Not Meet Design Standards.

Observer Agreement in the WWC Pilot Standards The WWC Panel not specify the method of observer agreement (e.g., using only measures that control for chance agreement). Hartmann, Barrios, and Wood (2004) reported over 20 measures to assess interobserver agreement. A good practice is to report measures that control for chance agreement and report agreement for occurrences and non-occurrences of the dependent variable. Good to assess accuracy: the degree to which observer data reflect actual performance or some true measure of behavior (e.g., video performance of the behavior).

Artifact, Bias, and Complexity of Assessment Researcher needs to consider factors that can obscure observer agreement: Direct assessment by observers can be reactive; Observers can drift in their assessment of the dependent variable; Observers can have expectancies for improvement that are conveyed by the researcher; Coding systems can be too complex and lead to error.

Intervention Integrity

Intervention Integrity: A Developing Standard Assessing (and especially promoting) intervention or treatment Integrity is a more recent and developing consideration in intervention research (see Sanetti & Kratochwill, 2014). Sanetti, L. M., & Kratochwill, T. R. (Eds.) (2014). Treatment Integrity: A Foundation for Evidence-Based Practice in Applied Psychology. Washington, DC: American Psychological Association.

Representative Features of Intervention Integrity Data Across Design Intervention Phases Across Therapists/Intervention Agents Across Situations Across Sessions Across Cases

Where does intervention integrity fit? 1. Screening/ assessment data suggest prevention/intervention is warranted Where does intervention integrity fit? 2. Evidence-based intervention selected & implemented 3. Program outcomes (SO) assessed 3. Intervention integrity (II) assessed Data reviewed 5a. Continue intervention + SO + II 4. Data-based decisions 5b. Continue intervention / promote intervention integrity + SO - II -SO - II 5c. Implement intervention strategies to promote intervention integrity - SO + II 5d. Change intervention

Treatment Integrity: Definition “Fidelity of implementation is traditionally defined as the determination of how well an intervention is implemented in comparison with the original program design during an efficacy and/or effectiveness study” (O’Donnell, 2008, p.33) Five distinct dimensions of intervention integrity were proposed: adherence, exposure, quality of delivery, participant responsiveness, and program differentiation (Dane & Schneider, 1998) Treatment integrity encompasses three different aspects: treatment adherence, therapist competence, and treatment differentiation (Waltz, Addis, Koerner, &Jacobson, 1993). The degree to which treatment is delivered as intended (Yeaton & Sechrest, 1981) Treatment integrity is the extent to which required intervention components are delivered as designed in a competent manner while proscribed procedures are avoided by an interventionist trained to deliver the intervention in a particular setting to a particular population (Perepletchikova, 2014). Implementation is defined as a specified set of activities designed to put into practice an activity or program of known dimensions (Sanetti & Kratochwill, 2009) “The degree to which an intervention program is implemented as planned’’ (Gresham et al., 1993, p. 254). The ‘‘extent to which patients follow the instructions that are given to them for prescribed treatments’’ (Haynes et al., 2002, p. 2). “Implementation is defined as a specified set of activities designed to put into practice an activity or program of known dimensions” (Fixsen et al. 2005, p.5).

Intervention Integrity Data Definition of Intervention Intervention Integrity in Research [Sanetti, L. M. H., Gritter, K. L., Dobey, L. M. (2011). Treatment integrity of interventions with children in the school psychology literature from 1995 to 2008. School Psychology Review, 40, 72-84]. Intervention Integrity Data Definition of Intervention

Intervention Integrity Assessment in School Psychology Practice [Cochran, W. S., & Laux, J. M. (2008). A survey investigating school psychologists' measurement of treatment integrity in school‐based interventions and their beliefs about its importance. Psychology in the Schools, 45, 499-507. https://doi.org/10.1002/pits.20319 ]

Implementation Supports Negative reinforcement Intervention manual Test driving interventions Expert consultation Intervention scripts Video support Intervention choice Performance feedback Implementation planning Classroom Check Up Motivational interviewing Instructional coaching Participant modeling Treatment planning protocol Role play Prompts Collaborative consultation Self-monitoring Direct training Sanetti & Collier-Meek (2014)

Perspectives on intervention integrity Performance Feedback (Noell et al., 1997, 2002, 2005) Negative Reinforcement (DiGennaro et al., 2005, 2007) Behavior Performance Deficit Email/ text/skype cues

Contributions from related fields Psychology Education Contributions from related fields Medicine Health Psychology Prevention Science Intervention integrity is more complicated than a behavior performance deficit.

Shifting conceptualization of treatment integrity Prevention/Intervention programs require implementers to change their behavior Implementing interventions with a high level of integrity can be considered an adult behavior change process.

Mental Health and Education Related Fields Health Action Process Approach(HAPA) Theory of Reasoned Action Transtheoretical Approach Behavioral theory

Focus on the HAPA Looked for a theory that… Is conceptually clear and consistent, Is parsimonious but would enable us to describe, explain, and predict behavior change. Explicates behavior performance, not just development of behavioral intention. Enabled us to design effective interventions that would produce change in the predicted behaviors. Has empirical support and practical utility.

Breast self-examination Empirical Support Healthy eating Exercise Behaviors Breast self-examination Seat belt use Dental flossing See Schwarzer, R.( 2008) Modeling health behavior change: How to predict and modify the adoption and maintenance of health behaviors. Applied Psychology, 57, 1-29. https://doi.org/10.1111/j.1464-0597.2007.00325.x

Project PRIME Planning Realistic intervention Implementation and Maintenance by Educators www.implementationscience.uconn.edu Funded by the Institute for Education Sciences, U.S. Department of Education (R324A100051)

Project PRIME PRIME was designed originally to prevent teachers’ level of intervention implementation from declining. Developed a system of supports to facilitate teachers’ intervention implementation. Delivered through a problem-solving consultation model (Bergan & Kratochwill, 1990; Kratochwill & Bergan, 1990).

Three components of PRIME: Implementation Planning: A detailed guide walks consultants and consultees through the process of Action planning Coping planning

Action Planning Specifies how and under what circumstances an intended intervention action is to be completed Intention to implement Initiation of implementation

Action Planning Continued… Completion of Action Plan helps to Define the intervention components and steps All aspects of the intervention are accounted for by a component (e.g., a social skills lesson) Each step corresponds to behavior(s) performed to implement an intervention component (e.g., introducing the topic of the social skills lesson) Detail logistical planning on implementation Identify potential resource barriers

Action Planning Continued… Logistical planning questions answered for each implementation step include: Where will the step be implemented? How often will the step be implemented? For how long will the step be implemented? What resources (materials, time, space, personnel, etc.) are needed to complete this step? Throughout implementation Action Plan can be updated to reflect changes in implementation

Coping Planning Completion of Coping Plan helps to: Identify up to 4 of the most significant barriers to intervention implementation Develop “coping” strategies to address identified barriers Barriers to implementation are listed e.g., major changes in class schedule due to statewide testing Throughout implementation the coping plan is updated to reflect changes in implementation

Three components of PRIME: 2. Implementation Beliefs Assessment (IBA) Assesses implementer’s Outcome expectancies Self-efficacy Implementation Maintenance Recovery

Three components of PRIME: 3. Strategies to Increase Implementation Intention and Self-efficacy Eight strategies have been identified. Detailed guides walk consultants through strategy implementation. Strategies include: Participant modeling Role play

Multi-Tiered Implementation Supports (MTIS) Intensive strategies that typically require ongoing support Tier 3 Tier 2 Tier 1 Selected strategies based on the implementer’s intervention integrity and stage of learning Feasible and widely relevant implementation support strategies that can be easily embedded into typical programs

MTIS: Tier 1 Strategies Tier 2 Tier 1 Tier 3 Intervention manual Intervention scripts Collaborative/Expert consultation Instructional coaching Intervention choice Test driving interventions Direct training Tier 3 Tier 2 Tier 1

MTIS: Tier 1 & 2 Strategies Implementation Planning Treatment planning protocol Tier 3 Tier 2 Tier 1

MTIS: Tier 2 Strategies Tier 2 Tier 1 Tier 3 Role play Participant modeling Tier 3 Tier 2 Tier 1

MTIS: Tier 2 & 3 Strategies Motivational interviewing Self-monitoring Prompts Video support Tier 3 Tier 2 Tier 1

MTIS: Tier 3 Strategies Tier 2 Tier 1 Tier 3 Performance feedback Performance feedback with negative reinforcement Tier 3 Tier 2 Tier 1

Take away messages Intervention integrity is important, and cannot be assumed in prevention/intervention program implementation; Intervention integrity can be conceptualized as a behavior change process; Related fields can inform prevention/ intervention science practice in psychology and education; Multiple strategies implemented in a tiered model may be necessary to promote intervention integrity.

Intervention Intensity

Intervention Intensity: Definition and Dimensions Intervention intensity typically refers to intervention strength and traditionally, the most commonly recognized dimension is dose of the intervention (Codding & Lane, 2015). Coding, R. S. & Lane, K. L. (2015). A spotlight on treatment intensity: An important and often overlooked component of intervention inquiry. Journal of Behavioral Education, 24, 1–10

Dimensions of Intervention Intensity Dose Learning trial/session Session length Session frequency Length of intervention Cumulative intensity Group Size

Dimensions of Intervention Intensity (Continued) Intervention Design Form Positive corrective feedback Pace of instruction Opportunities to practice or respond Transitions between subjects or classes Goal specificity Intervention complexity

Dimensions of Intervention Intensity (Continued) Interventionist Characteristics Experience, knowledge, skills Implementation Characteristics Intervention management and planning Materials and tangible resources required Deviation from classroom routines

Social Validity in Single-Case Design Research

Social Validity in Intervention Research Social Validity involves three questions about an intervention (Kazdin, 2011): Are the goals of the intervention relevant to the person’s life? Is the intervention acceptable to “consumers” and others involved in the procedures? Are the outcomes of the intervention important (i.e., do changes make a difference in lives of persons involved)? Kazdin, A. E. (2011). Single-case research designs: Methods for clinical and applied settings. New York: Oxford.

Social Validity in Intervention Research Three social validation methods can be used in intervention research: Social Comparison Subjective Evaluation Sustainability of the Intervention

Social Validity in Intervention Research Social Comparison Normative social comparison data can be used when such information provides a good benchmark for positive functioning; Typically social comparison involves identification of a “peer group” similar to the client; The peer group would consist of persons who are functioning in an adequate or positive manner; Sometimes standardized assessment instruments can be used for social comparison purposes (e.g., checklists and rating scales).

Social Validity in Intervention Research Some challenges with social comparison: It is not so easy to establish normative comparisons; Normative comparisons may be unrealistic; Normative range of functioning may be an impossible goal given the level of impairment; Normative goals may not reflect overall quality of life; Standardized instruments used for normative comparisons may not provide accurate data (e.g., checklists and rating scales; Reid & Maag, 1994). Reid, R., &Maag, J. W. (1994). How many fidgets in a pretty much: A critique of behavior rating scales for identifying students with ADHD. Journal of School Psychology, 32, 339-354.

Social Validity in Intervention Research Subjective Evaluation Involves the assessment by significant others who have knowledge of the client and can make a judgment of the need for intervention and the outcomes of intervention; Individuals rendering a “subjective” judgment may be parents, teachers, or professionals who have expert status in an area; Specific goals may be established and serve as the basis for intervention and a benchmark for outcome evaluation.

Social Validity in Intervention Research Some challenges with subjective evaluation: Global ratings may be biased; Persons completing the rating (s) may perceive a small change as major when in fact, not much of significance has occurred; Subjective ratings may not correspond to other outcome data (e.g., direct observation).

Social Validity in Intervention Research Sustainability The degree to which the effects of a intervention are sustained over time (Kennedy, 2005). The assessment is a measure of how long the program stays in place or is adopted. Kennedy, C. H. (2005). Single-case designs for educational research. Boston: Allyn and Bacon.

Social Validity in Intervention Research Some Challenges with sustainability Requires a long time to conduct the analysis; Factors other than consumer acceptance may influence ratings/evaluation of sustainability; Sustainability is an indirect index of social validity.

Questions and Discussion