Joel R. Levin University of Arizona Principles of Scientifically Credible Intervention Research Brief Review and Transition to Single-Case Intervention Research Joel R. Levin University of Arizona

Focus of My Presentations Throughout the Week To introduce and discuss recent methodological and data-analysis innovations that serve to elevate the perceived low “scientific credibility” of single-case intervention research. Ultimately, to compare the research validity characteristics of a “traditional” multiple-subject randomized trials study and a “new age” single-case intervention study in the context of a hypothetical research grant competition.

Focus of This Presentation Considerations that researchers need to make in the way of their methodologies, procedures, measures, and statistical analyses when investigating whether or not causal relationships exist between educational interventions and outcome measures.

Brief Review of Common Education Research Methodologies Nonintervention Research Descriptive/observational/case study (includes ethnographic research) Self-report/survey/questionnaire studies Correlational Intervention Research Nonexperimental (e.g., nonequivalent control group) Quasi-experimental Experimental Note: Different degrees of plausibility/credibility about causal relationships are associated with these different methodologies.

Review and Preview The Punchline: Conducting “Scientifically Credible” Intervention Research What are the critical defining attributes of conventional (multiple-participant) intervention research? And what about the critical defining attributes of single-case intervention research?

Selected Research Validities • Internal validity ("randomization" = random assignment to treatments or interventions); needed to argue for cause-effect relationships • External validity (random selection = random sampling of participants); needed to generalize from sample to population − replication affords a useful proxy • Construct validity (the relationship between experimental operations and underlying psychological traits/processes) • Implementation validity (treatment integrity and “acceptability”) • Statistical-conclusion validity (appropriate statistical analyses and assumptions associated with them) Shadish, W. R., Cook, T. D., & Campbell, D.T. (2002). Experimental and quasi-experimental designs for generalized causal inference. Boston: Houghton Mifflin .

Alternative Research Methodologies Gaining Scientific Respectability Nonequivalent control group designs based on comprehensive matching (Aickin, 2009; Shadish, 2011) Regression discontinuity analysis (Shadish, 2011) Prompted optional randomization trial? (Flory & Karlawish, 2012) Single-case time-series designs (Dallery, Cassidy, & Raiff, 2013; Kratochwill & Levin, 2014; Plavnick & Ferreri, 2013) Aickin, M. (2009, Jan. 1). A simulation study of the validity and efficiency of design-adaptive allocation to two groups in the regression situation. International Journal of Biostatistics, 5(1): Article 19. Retrieved from www.ncbi.nlm.nih.gov/pmc/articles/PMC2827888/ Dallery, J. Cassidy, R. N., & Raiff, B. R. (2013). Single-case experimental designs to evaluate novel technology-based health interventions. Journal of Medical Internet Research, 15(2). Retrievable from www.jmir.org/2013/2/e22/. Flory, J.. & Karlawish, J. (2012). The prompted optional randomization trial: A new design for comparative effectiveness research. American Journal of Public Health, 102(12), e8-e10. Kratochwill, T. R., & Levin, J. R. (Eds.). (2014). Single-case intervention research: Methodological and statistical advances. Washington, DC: American Psychological Association. Plavnick, J. B., & Ferreri, S, J. (2013). Single-case experimental designs in educational research: A methodology for causal analyses in teaching and learning. Educational Psychology Review, 25, 549-569. Shadish, W. R. (2011). Randomized controlled studies and alternative designs in outcome studies: Challenges and opportunities. Research on Social Work Practice, 21, 636-643.

Time Out For a Research “Case” Case studies observational/naturalistic intensive study of individual “cases” no designed/planned intervention Freud, Piaget, Skinner; brain-injury cases; cases of phenomenal mnemonists and geniuses efforts to change the “Rodney Dangerfield” perceptions of this mode of inquiry (e.g., Dattilio, Edwards, & Fishman, 2010) Dattilio, F. M., Edwards, D. J. A., & Fishman, D. B. (2010). Case studies within a mixed methods paradigm: Toward a resolution of the alienation between researcher and practitioner in psychotherapy research. Psychotherapy Theory, Research, Practice, Training, 47, 427-441.

Time Out For a Research “Case” Case studies observational/naturalistic intensive study of individual “cases” no designed/planned intervention Freud, Piaget, Skinner; brain-injury cases; cases of phenomenal mnemonists efforts to change the “Rodney Dangerfield” perceptions of this mode of inquiry (e.g., Dattilio, Edwards, & Fishman, 2010) Case-based reasoning arguments and inferences derived from legal and medical cases now used in higher-education courses “Make a Case” studies the anecdotal evidence “research” currently on display in some of our professional organizations Dattilio, F. M., Edwards, D. J. A., & Fishman, D. B. (2010). Case studies within a mixed methods paradigm: Toward a resolution of the alienation between researcher and practitioner in psychotherapy research. Psychotherapy Theory, Research, Practice, Training, 47, 427-441.

Facts, Alternative Facts, and Fiction “As David Quammen [science writer and explorer] wrote, ‘Faith comforts, but data persuade: What are called data today, for Darwin were facts’.” Wesson, R. (2017, p. 208), Darwin’s first theory. Pegasus: New York.

Theories, Hypotheses, Predictions, and Post Hoc “ Explanations” Theory  Hypothesis  Prediction  Post Hoc “Explanation” (“Postdiction”) 

Post Hoc “Explanation” (From Platt’s Classic “Strong Inference” Article) Some cynics tell a story, which may be apocryphal, about the theoretical chemist who explained to his class, "And thus we see that the C-Cl bond is longer in the first compound than in the second because the percent of ionic character is smaller."

Post Hoc “Explanation” A voice from the back of the room said, "But Professor X, according to the Table, the C-Cl bond is shorter in the first compound.“ "Oh, is it?" said the professor. "Well, that's still easy to understand, because the double-bond character is higher in that compound."

Post Hoc “Explanation” To the extent that this kind of story is accurate, a "theory" of this sort is not a theory at all, because it does not exclude anything. It predicts everything, and therefore does not predict anything. It becomes simply a verbal formula which the graduate student repeats and believes because the professor has said it so often.

Post Hoc “Explanation” This is not science, but faith; not theory, but theology. Whether it is hand-waving or number-waving or equation-waving, a theory is not a theory unless it can be disproved. That is, unless it can be falsified by some possible experimental outcome. Platt, J. R. (1964). Strong inference. Science, 146, 347-353.

Single-Case Intervention Research: Preview of Things to Come in the Days Ahead (see also Horner & Spalding, 2010) Historically, referred to as “single-subject,” “N = 1,” “interrupted time-series,” and currently “single-participant” or “single-case” research “Case” applies both to individual participants and aggregates or “clusters” (e.g., pairs, small groups, classrooms, schools, communities) Involves carefully planned researcher-administered interventions and comparison conditions For purposes of assessing an intervention effect, each case serves as its own control.

Why Single Case (as Compared to Conventional “Group”)? Scarce resources with respect to the number of cases that can be recruited and/or treated Scarce resources with respect to a focus on low-incidence populations Adequate resources with respect to the number of outcome measures that can be taken Fewer logistical hurdles/practical constraints Intensive study of individual cases As either a precursor or a complement to a “group” randomized intervention study

Characteristics of Single-Case Intervention Research Adaptations of interrupted time-series designs that have the potential to provide rigorous evaluations of the effects resulting from experimental interventions Single-case designs involve repeated, systematic measurement of an outcome measure before, during, and after the implementation of an intervention. The ratio of outcome measures to the number of cases usually is large enough to distinguish single-case designs from other longitudinal designs (e.g., traditional pretest-posttest and general repeated-measures designs). Kratochwill, T. R., Hitchcock, J., Horner, R. H., Levin, J. R., Odom, S. L., Rindskopf, D. M & Shadish, W. R. (2010). Single-case designs technical documentation. Retrieved from the What Works Clearinghouse website: http://ies.ed.gov/ncee/wwc/pdf/wwc_scd.pdf

Characteristics of Single-Case Intervention Research The outcome measure is taken repeatedly within and across different levels of the researcher-manipulated intervention variable. These different levels are referred to as “phases” and minimally include a baseline (A) phase and an intervention (B) phase. From Kratochwill et al. (2010)

Days/Weeks/Months/Sessions Traditional Basic Design (AB)

Characteristics of Single-Case Intervention Research The outcome measure is taken repeatedly within and across different levels of the researcher-manipulated intervention variable. These different levels are referred to as “phases” and minimally include a baseline (A) phase and an intervention (B) phase. A central goal of single-case intervention research is to determine whether there is a change in the outcome measure(s) following introduction of the intervention. Evidential support for an intervention effect is achieved through various forms of replication within the particular study. From Kratochwill et al. (2010)

Different Patterns of Baseline-to-Intervention Phase Change Kratochwill, T. R., & Levin, J. R. (1978). What time-series designs may have to offer educational researchers. Contemporary Educational Psychology, 3, 273-329.

Intervention Research Evidence Standards American Psychological Association’s Reporting Standards American Psychologist, 2008, 63, p. 839-851 The What Works Clearinghouse’s Standards For traditional “group” intervention designs: http://ies.ed.gov/ncee/wwc/references/idocviewer/doc.aspx?docid=19&tocid=1 For single-case intervention designs: http://ies.ed.gov/ncee/wwc/pdf/wwc_scd.pdf

Methodological Standards for Single-Case Research Separate “credibility” standards should be applied for each type of intervention research, conventional group and single-case Guidelines are in effect for conventional group RCTs (Moher, Schulz, & Altman (2001; APA Publications and Communications Board Working Group/Task Force on Journal Article Reporting Standards, 2008; 2018)

Methodological Standards for Single-Case Research − “Credibility” standards have been developed for traditional single-case intervention designs based on individual cases (Kratochwill et al., 2010) − The same needs to be done for both randomized single-case intervention designs and single-case designs in which the “cases” consist of aggregates or “clusters” (e.g., small groups, classrooms, schools, communities) Moher, D., Schulz, K. F., & Altman, D. G. (2001). The CONSORT statement: Revised recommendations for improving the quality of reports of parallel-group randomized trials. Annals of Internal Medicine, 134, 657–662. APA Publications and Communications Board Task Force (2018). Journal Article Reporting Standards for Quantitative Research in Psychology. American Psychologist, 73, 3-25. APA Publications and Communications Board Working Group on Journal Article Reporting Standards. (2008). Reporting standards for research in psychology: Why do we need them? What might they be? American Psychologist, 63, 839-851.

Recurring Themes Throughout This Institute Can/should single-case intervention designs be held accountable for similar scientific-credibility standards as are applied to conventional “group” research? Can various randomization schemes be incorporated into single-case intervention designs to improve their scientific credibility? If so, what are the resulting logistical (and ethical) consequences of charting a single-case intervention-research course?