Principles of Scientifically Credible Intervention Research Brief Review and Transition to Single-Case Intervention Research Joel R. Levin University of Arizona

Focus of My Presentations Throughout the Week To introduce and discuss recent methodological and data-analysis innovations that serve to elevate the perceived low “scientific credibility” of single-case intervention research. Ultimately, to compare the research validity characteristics of a traditional “group” randomized trials study and a “new age” single- case intervention study in the context of a hypothetical research grant competition.

Focus of This Presentation Considerations that researchers need to make in the way of their methodologies, procedures, measures, and statistical analyses when investigating whether or not causal relationships exist between educational interventions and outcome measures.

Brief Review of Common Education Research Methodologies Nonintervention Research –Descriptive/observational/case study (includes ethnographic research) –Self-report/survey/questionnaire studies –Correlational Intervention Research –Nonexperimental (e.g., nonequivalent control group) –Quasi-experimental –Experimental Note: Different degrees of plausibility/credibility about causal relationships are associated with these different methodologies.

Review and Preview The Punchline: “Scientifically Credible” Intervention Research What are the basic defining characteristics in conventional “group” intervention research? And what about the basic defining characteristics in single-case intervention research?

Selected Research Validities Internal validity ("randomization" = random assignment to treatments or interventions; needed to argue for cause-effect relationships) External validity (random selection = random sampling of participants; needed to generalize from sample to population) Construct validity (the relationship between experimental operations and underlying psychological traits/processes) Implementation validity (treatment integrity and “acceptability”) Statistical-conclusion validity (appropriate statistical analyses and assumptions associated with them) Shadish, W. R., Cook, T. D., & Campbell, D.T. (2002). Experimental and quasi-experimental designs for generalized causal inference. Boston: Houghton Mifflin.

Alternative Research Methodologies Gaining Scientific Respectability Nonequivalent control group designs based on comprehensive matching (Aickin, 2009; Shadish, 2011) Regression discontinuity analysis (Shadish, 2011) Prompted optional randomization trial? (Flory & Karlawish, 2012) Single-case time-series designs (Dallery, Cassidy, & Raiff, 2013; Kratochwill & Levin, 2014; Plavnick & Ferreri, 2013) Aickin, M. (2009, Jan. 1). A simulation study of the validity and efficiency of design-adaptive allocation to two groups in the regression situation. International Journal of Biostatistics, 5(1): Article 19. Retrieved from Dallery, J. Cassidy, R. N., & Raiff, B. R. (2013). Single-case experimental designs to evaluate novel technology-based health interventions. Journal of Medical Internet Research, 15(2). Retrievable from Flory, J.. & Karlawish, J. (2012). The prompted optional randomization trial: A new design for comparative effectiveness research. American Journal of Public Health, 102(12), e8-e10. Kratochwill, T. R., & Levin, J. R. (Eds.). (2014). Single-case intervention research: Methodological and statistical advances. Washington, DC: American Psychological Association. Plavnick, J. B., & Ferreri, S, J. (2013). Single-case experimental designs in educational research: A methodology for causal analyses in teaching and learning. Educational Psychology Review, 25, Shadish, W. R. (2011). Randomized controlled studies and alternative designs in outcome studies: Challenges and opportunities. Research on Social Work Practice, 21,

Time Out For a Research “Case” Case studies –observational/naturalistic intensive study of individual “cases” –no designed/planned intervention –Freud, Piaget, Skinner; brain-injury cases; cases of phenomenal mnemonists –recent efforts to change the “Rodney Dangerfield” perceptions of this mode of inquiry (e.g., Dattilio, Edwards, & Fishman, 2010) Case-based reasoning –arguments and inferences derived from legal and medical cases –now used in higher-education courses “Make a Case” studies –the anecdotal evidence “research” currently on display in some of our professional organizations Dattilio, F. M., Edwards, D. J. A., & Fishman, D. B. (2010). Case studies within a mixed methods paradigm: Toward a resolution of the alienation between researcher and practitioner in psychotherapy research. Psychotherapy Theory, Research, Practice, Training, 47,

Single-Case Intervention Research: Preview of Things to Come in the Days Ahead (see also Horner & Spalding, 2010) Historically, referred to as “single-subject,” “N = 1,” “interrupted time-series,” and currently “single- participant” or “single-case” research Involves carefully planned researcher-administered interventions and comparison conditions Applies both to individual participants and aggregates or “clusters” (e.g., pairs, small groups, classrooms, schools, communities) For purposes of assessing an intervention effect, each case serves as its own control.

Characteristics of Single-Case Intervention Research Adaptations of interrupted time-series designs that have the potential to provide rigorous evaluations of the effects resulting from experimental interventions Single-case designs involve repeated, systematic measurement of an outcome measure before, during, and after the implementation of an intervention. The ratio of outcome measures to the number of cases usually is large enough to distinguish single- case designs from other longitudinal designs (e.g., traditional pretest-posttest and general repeated- measures designs). Kratochwill, T. R., Hitchcock, J., Horner, R. H., Levin, J. R., Odom, S. L., Rindskopf, D. M & Shadish, W. R. (2010). Single-case designs technical documentation. Retrieved from the What Works Clearinghouse website:

Characteristics of Single-Case Intervention Research The outcome measure is taken repeatedly within and across different levels of the researcher-manipulated intervention variable. These different levels are referred to as “phases” and minimally include a baseline (A) phase and an intervention (B) phase. A central goal of single-case intervention research is to determine whether there is a change in the outcome measure(s) following introduction of the intervention. Evidential support for an intervention effect is achieved through various forms of replication within the particular study. From Kratochwill et al. (2010)

Days/Weeks/Months/Sessions Traditional Basic Design (AB)

Different Patterns of Baseline-to-Intervention Phase Change Kratochwill, T. R., & Levin, J. R. (1978). What time-series designs may have to offer educational researchers. Contemporary Educational Psychology, 3,

Intervention Research Evidence Standards American Psychological Association’s Reporting Standards American Psychologist, 2008, 63, p The What Works Clearinghouse’s Standards For traditional “group” intervention designs: doc.aspx?docid=19&tocid=1 For single-case intervention designs:

Methodological Standards for Single-Case Research Separate credibility standards should be applied for each type of intervention research, conventional group and single-case –Guidelines are in effect for conventional group RCTs (Moher, Schulz, & Altman (2001; APA Publications and Communications Board Working Group on Journal Article Reporting Standards, 2008) –Standards have been developed for traditional single-case intervention designs based on individual cases (Kratochwill et al., 2010) –The same needs to be done for both randomized single- case intervention designs and single-case designs in which the “cases” consist of aggregates or “clusters” (e.g., small groups, classrooms, schools, communities) Moher, D., Schulz, K. F., & Altman, D. G. (2001). The CONSORT statement: Revised recommendations for improving the quality of reports of parallel-group randomized trials. Annals of Internal Medicine, 134, 657–662. APA Publications and Communications Board Working Group on Journal Article Reporting Standards. (2008). Reporting standards for research in psychology: Why do we need them? What might they be? American Psychologist, 63,

Recurring Themes Throughout This Institute Can/should single-case intervention designs be held accountable for similar scientific-credibility standards as are applied to conventional “group” research? Can various randomization schemes be incorporated into single-case intervention designs to improve their scientific credibility? If so, what are the resulting logistical (and ethical) consequences of charting a single-case RCT course?

Whys and Wherefores Single-Case Intervention Research? “My only recommendation would be to spend some time describing the basic tenets of single case experimental design. What are the reasons why someone would choose to apply it? What are the unique contributions of this methodology? Basically, it would help us ‘sell’ these designs to potential group reviewers.” Attendee of the 2011 IES-Sponsored Single-Case Design-and-Analysis Institute

Why Single Case? Scarce resources with respect to the number of cases that can be recruited and/or treated Scarce resources with respect to a focus on low-incidence populations Adequate resources with respect to the number of measures that can be taken Fewer logistical hurdles/practical constraints Intensive study of individual cases As either a precursor or a complement to a “group” randomized intervention study

Cliffhanger Conclusion