Institute of Education Sciences Summer Research Training Institute: Single-Case Intervention Design and Analysis June 19-23, 2017 The Concourse Hotel Madison, Wisconsin

Introductions and General Overview (Tom Kratochwill, Joel Levin, and Rob Horner) Welcome and Introductions Goals and Objectives Schedule and Logistics of the Meeting Group Activity Sessions and Individual Consultation Options Resources on Single-Case Design and Analysis

Single-Case Design Institute Faculty Thomas R. Kratochwill, PhD (University of Wisconsin-Madison) Joel R. Levin, PhD (University of Arizona) Robert Horner (University of Oregon) Wendy Machalicek, PhD John Ferron, PhD (University of South Florida) Ann Kaiser, PhD (Vanderbilt University) Carnac the Magnificent Statistician, PhD, PsyD, JD, MD, DDS, and NCM (Psychic University)

A Special Tribute to our Colleague and Friend William R. Shadish (1949-2016)

Welcome Participants! Welcome to Wisconsin

Assumptions for Participants: Scholars with some knowledge and expertise in traditional and single-case research methods (some basic and many advanced); Some with less experience, and maybe some skepticism about utility of single-case research methods; Commitment to the role of science in society and the importance of intervention research, and evidence-based practices in psychology, education, and related fields.

Goals and Objectives of the Single-Case Design Institute Review Logic and Foundations of Single-Case Designs in Intervention Research; Review “pilot” Single-Case Design Standards WWC design standards (Single-Case Designs) WWC evidence criteria (Visual Analysis); Provide an overview of approaches to visual and statistical analysis and effect size measures within single-case intervention research; Review criteria for documenting evidence-based practices using single-case intervention research methods; Each participant will leave the Institute with a logic model (and more) for a single-case research study (or program of study) that fits their grant/research agenda.

Schedule and Logistics of the Meeting Agenda for the Institute Breaks, Lunch, and Dinner Break-out Sessions/Small Group Activities Individual Project Consultations

Group Activity Sessions and Individual Project Consultation Options Format for Small Group Activities Small Group Break-Out Rooms: (Rooms are reserved for the week) Schedule for Individual Project Consultations

Some Goals Group Activity: Personal Research Consultation: Participants are assigned to a collaborative group to build a logic model for a single-case study, design a study, and apply visual analysis, statistical analysis, and effect size measures. Personal Research Consultation: Each participant can meet with a faculty member(s) and develop a logic model (and more) for a single-case research study (or program of study) that fits their grant/research agenda.

Resources on Single-Case Design and Analysis Required Readings Recommended Readings Additional Resources Books, Monographs, and Journals Faculty as a Resource and Follow Up

Follow up to Institute Thomas R. Kratochwill, PhD Wisconsin Center for Education Research 1025 West Johnson Street University of Wisconsin-Madison Madison, Wisconsin 53706 E-Mail: tomkat@education.wisc.edu Robert Horner, PhD University of Oregon 1235 University of Oregon Eugene, OR 97403-1235 E-Mail: robh@uoregon.edu Joel R. Levin, PhD College of Education University of Arizona Tucson, Arizona 85721 E-Mail: jrlevin@email.arizona.edu

Institute of Educational Sciences (IES) Dr. Sarah Brasiel, National Center for Special Education Research

Logic and Foundations of Single-Case Intervention Research Joel R. Levin and Thomas R. Kratochwill University of Wisconsin-Madison

Brief Review of the Principles of Scientifically Credible Intervention Research and Transition to Single-Case Intervention Research Joel Levin

Logic and Foundations of Single-Case Intervention Research Tom Kratochwill

Options for single-case research in development of effective interventions: Useful in the iterative development of interventions (e.g., replication research). Documentation of experimental effects that can help understand and define the mechanism for change, not just the occurrence of change. Analysis of interventions targeting low-incidence populations (e.g., individual as unit of analysis, individuals with disabilities). Useful for pilot research to assess the effect size needed for other research methods (e.g., RCTs).

Options for single-case research in development of effective interventions (Continued): Useful for fine-grained analysis of “weak and non-responders” (Negative Results; to be discussed later in the Institute). Useful in RCT/group research when unique participants (e.g., non-responders) are further assessed to determine what modifications in the intervention may need to occur.

Purposes and Fundamental Assumptions of Single-Case Intervention Research Methods Defining features of SCDs Core design types Internal validity and the role of replication “True” Single-Case Applications and the WWC Pilot Standards (design standards and evidence criteria) Classroom-Based Applications (design and evidence credibility)

Features of Single-Case Research Methods Single-Case Research will have Four Features: Independent variable Dependent variable Focus is on functional relation (causal effect) Dimension(s) of predicted change (e.g., level, trend, variability, score overlap)

Design Examples Reversal/Withdrawal Designs Multiple Baseline Designs Alternating Treatment Designs Others: Changing Criterion Non-Concurrent Multiple Baseline Multiple Probe

Additional Considerations Operational definition of dependent variable (DV) Measure of DV is valid, reliable, and addresses the dimension(s) of concern. Operational definition of independent variable (IV) Core features of IV are defined and measured to document treatment integrity (Sanetti & Kratochwill, 2014). Unit of IV implementation Group versus individual unit.

Background on Single-Case Designs: Defining Features Design: Repeated measurement of an outcome before, during, and/or after active manipulation of independent variable Often Used in Applied and Clinical Fields Allows study of low prevalence disorders where otherwise would need large sample for statistical power (Odom, et al., 2005). Sometimes more palatable to service providers because SCDs do not include a no-treatment comparison group. What are researchers doing in literature reviews and studies?

Literature Reviews Wong and colleagues (2015) conducted a review of intervention practices for children with autism. Of 456 studies identified in their review, 89% used SCDs and only 11% used between-groups research designs. Wong, C., Odom, S. L., Hume, K. A., Cox, A. W., Fettig, A., Kucharczyk, S., … Schultz, T. R. (2015). Evidence-based practices for children, youth, and young adults with autism spectrum disorder: A comprehensive review. Journal of Autism and Developmental Disorders, 45(7), 1951–1966. doi:10.1007/s10803-014-2351-z

Literature Reviews Conroy, Dunlap, Clarke, and Alter (2005) conducted a systematic review of positive behavioral interventions for children with challenging behavior. They identified 62 single-case studies and 1 between-group design. Conroy, M. A., Dunlap, G., Clarke, S., & Alter, P. J. (2005). A descriptive analysis of positive behavioral intervention research With young children with challenging behavior. Topics in Early Childhood Special Education, 25(3), 157–166. doi:10.1177/02711214050250030301

Descriptive Analysis Hammond and Gast (2010) reviewed 196 randomly identified journal issues (from 1983-2007) containing 1,936 articles (a total of 556 single-case designs were coded). Multiple baseline designs were reported more often than withdrawal designs and these were more often reported across individuals and groups. Hammond, D., & Gast, D. L. (2010). Descriptive analysis of single subject research designs: 1983-2007. Education and Training in Autism and Developmental Disabilities, 45(2), 187–202.

Literature Reviews that Include Single-Case Design Kiuhara, Kratochwill, and Pullen (in press) reported over 70 literature reviews (e.g., meta-analysis) of single-case research studies from 2006-2016. More recent reviews have adopted various standards for critique of single-case design research. Kiuhara, S. A., Kratochwill, T. R., & Pullen, P. C. (in press). Single-case design intervention research: Applications in special education. In J. M. Kauffman, D. P. Hallahan, and P. C. Pullen (Eds.). Handbook of special education (2nd ed.). New York: Routledge.

Types of Research Questions that Can be Answered with Single-Case Design Types Evaluate Intervention Effects Relative to Baseline For example: Does Multi-systemic Therapy reduce the level of problem behavior for students with emotional behavior disorders? Compare Relative Effectiveness of Interventions For example: Is “function-based behavior support” more effective than “non-function-base support” at reducing the level and variability of problem behavior for this participant? Compare Single- and Multi-Component Interventions For example: Does adding Performance Feedback to Basic Teacher Training improve the fidelity with which instructional skills are used by new teachers in the classroom?

Some Further Examples of SCD Research Questions that Might be Addressed Is a certain teaching procedure functionally related to an increase in the level of social initiations by young children with autism? Is time delay prompting or least-to-most prompting more effective in increasing the level of self-help skills performed by young children with severe intellectual disabilities? Is the pacing of reading instruction functionally related to increased level and slope of reading performance (as measured by ORF) for third graders? Is Adderal (at clinically prescribed dosage) functionally related to increased level of attention performance on the Attention Network Test for elementary age students with Attention Deficit Disorder?

Single-Case Designs are Experimental Designs Like RCTs, purpose is to document causal relationships Control for major threats to internal validity Document effects for specific individuals / settings Replication (across studies) required to enhance external validity Can be distinguished from case studies

Ambiguous Temporal Precedence Selection History Maturation Testing Single-Case Design Standards were Developed to Address Threats to Internal Validity (when the unit of analysis is the individual) Ambiguous Temporal Precedence Selection History Maturation Testing Instrumentation Additive and Interactive Effects of Threats 31

Additional Threats to Validity May Occur When Clusters are the Unit of Analysis Examples: Cluster Selection Cluster Composition Within-Cluster Variability Attrition of Within-Cluster Participants and of Clusters Within-Cluster Extraneous Variables Across-Cluster Contagion Effects

Distinctions Between Experimental Single-Case Design and Clinical Case Study Research

Some Characteristics of Traditional Case Study Research Often (and traditionally) characterized by narrative description of case, treatment, and outcome variables Typically lack a formal design with replication but can involve a “basic effect” design format (e.g., A/B) Methods have been suggested to improve drawing valid inferences from case study research [e.g., Kazdin, A. E. (2011). Single-case research designs: Methods for clinical and applied settings(2nd ed.). New York: Oxford University Press]

Some Considerations That May Improve Case Study Research Type of data Assessment occasions Planned vs. ex post facto Projections of performance Treatment effect size Treatment effect impact Number of participants/replication Standardization of treatment Integrity of treatment Randomization

Questions and Discussion