Single-Case Intervention Research Training Institute Madison, WI - June, 2018 James E. Pustejovsky pusto@austin.utexas.edu Effect size measures for single-case designs: between-case standardized mean differences

Parametric between-case effect sizes Within-case standardized mean difference is not on the same scale as SMD from between-groups designs (e.g., between-subjects randomized trial). Shadish, Rindskopf, & Hedges (2008) asked: Can we estimate a SMD based on the data from a single-case design that IS in the same metric a SMD from a between-groups design? Why do this? (Shadish, Hedges, Horner, & Odom, 2015) Translation of single-case research for researchers who work primarily with between-groups designs. Comparison of results from single-case studies and between-groups studies, for purposes of understanding the utility and limitations of each type of design. Synthesis involving both single-case and between-groups designs.

Between-case SMD What is the SMD from a between-groups experiment? These quantities can be estimated from single-case data using a hierarchical model that describes variation within and between participants. But we’ll need to have a sample of multiple participants (bare minimum of 3, more for more complex models).

Estimating between-case SMDs: The broad strategy: Develop a hierarchical model that describes a) the functional relationship for each case and b) how the outcome and functional relationship vary across cases. Use the hierarchical model to imagine a hypothetical between-subjects experiment with the same population of participants, same treatment, same outcomes. Calculate the between-case SMD for the hypothetical experiment. Imagine running a hypothetical experiment. In a sense, you could think of this as “simulating” data for a set of participants

Estimating BC-SMDs: Basic model Hedges, Pustejovsky, and Shadish proposed BC-SMD estimators for a basic hierarchical linear model HPS (2012): Treatment reversal (ABAB) designs HPS (2013): Multiple baseline/multiple probe designs Assumptions: Baseline is stable (no baseline trend). Intervention effect is immediate (no intervention-phase trend). The outcome is normally distributed around mean level for each case, with variance σ2. Within-case errors follow a first-order auto-regressive process (serial dependence). The baseline level for each case is normally distributed, with variance τ2. The treatment effect is constant across cases.

Estimating BC-SMDs: Basic model Moment estimation Web app: https://jepusto.shinyapps.io/scdhlm/ R package: http://jepusto.github.io/getting-started-with-scdhlm SPSS macro: http://faculty.ucmerced.edu/wshadish/software/software-meta-analysis-single-case-design/dhps-version-march-7-2015 Shadish, Hedges, and Pustejovsky (2014) includes worked examples Restricted maximum likelihood estimation (recommended) Both methods produce estimates of BC-SMD (corrected for small-sample bias) and accompanying standard error.

Rodriguez & Anderson (2014) example (using REML estimation)

More flexible models for BC-SMDs Pustejovsky, Hedges, and Shadish (2014) extend the basic model to allow for less restrictive assumptions: Variability of individual treatment effects Baseline time trends (constant or varying across participants) Time trends in treatment phase (constant or varying across participants) More flexible models require more cases For models with time trends, also need to specify a focal follow-up time

Barton-Arwood, Wehby, & Falk (2005) Barton-Arwood, Wehby, & Falk (2005). Reading instruction for elementary-age students with emotional and behavioral disorders: Academic and behavioral outcomes Intervention consisted of Horizon Fast Track reading program and Peer-Assisted Learning Strategies

Barton-Arwood, Wehby, & Falk (2005) Effect size calculations Allow for baseline and intervention time trends, both varying across cases. Focal follow-up time after 13 sessions. BC-SMD estimate of 0.91 (SE = 0.93). Substantial heterogeneity in intervention time trends.

Limitations of between-case SMD Describes an average effect across a set of cases Conceals potential individual heterogeneity Inherent consequence of comparability with between-groups effect sizes. Technical limitations Only available for treatment reversal (ABAB) and multiple baseline/multiple probe across participant designs. Requires at least 3 participants, preferably more. Assumes normally distributed, interval-scale outcomes. More work needed on evaluating model selection, model fit Use between-case effect sizes as a complement to (not a replacement for) within-case effect size measures Between-groups effect sizes only provide information about average effects, rather than individual effects.

References Beretvas, S. N., & Chung, H. (2008). An evaluation of modified R2-change effect size indices for single-subject experimental designs. Evidence-Based Communication Assessment and Intervention, 2(3), 120–128. doi:10.1080/17489530802446328 Busk, P. L., & Serlin, R. C. (1992). Meta-analysis for single-case research. In T. R. Kratochwill & J. R. Levin (Eds.), Single-Case Research Design and Analysis: New Directions for Psychology and Education (pp. 187–212). Hillsdale, NJ: Lawrence Erlbaum Associates, Inc. Campbell, J. M. (2003). Efficacy of behavioral interventions for reducing problem behavior in persons with autism: a quantitative synthesis of single-subject research. Research in Developmental Disabilities, 24(2), 120–138. doi:10.1016/S0891-4222(03)00014-3 Campbell, J. M., & Herzinger, C. V. (2010). Statistics and single subject research methodology. In D. L. Gast (Ed.), Single Subject Research Methodology in Behavioral Sciences (pp. 417–450). New York, NY: Routledge. Gingerich, W. J. (1984). Meta-analysis of applied time-series data. Journal of Applied Behavioral Science, 20(1), 71–79. doi:10.1177/002188638402000113 Hedges, L. V, Pustejovsky, J. E., & Shadish, W. R. (2012). A standardized mean difference effect size for single case designs. Research Synthesis Methods, 3, 224–239. doi:10.1002/jrsm.1052 Hedges, L. V, Pustejovsky, J. E., & Shadish, W. R. (2013). A standardized mean difference effect size for multiple baseline designs across individuals. Research Synthesis Methods, 4(4), 324–341. doi:10.1002/jrsm.1086 Kahng, S., Iwata, B. a, & Lewin, A. B. (2002). Behavioral treatment of self-injury, 1964 to 2000. American Journal of Mental Retardation : AJMR, 107(3), 212–221. doi:10.1352/0895-8017(2002)107<0212:BTOSIT>2.0.CO;2 Maggin, D. M., Swaminathan, H., Rogers, H. J., O’Keeffe, B. V, Sugai, G., & Horner, R. H. (2011). A generalized least squares regression approach for computing effect sizes in single-case research: Application examples. Journal of School Psychology, 49(3), 301–321. doi:10.1016/j.jsp.2011.03.004 Marquis, J. G., Horner, R. H., Carr, E. G., Turnbull, A. P., Thompson, M., Behrens, G. A., … Doolabh, A. (2000). A meta-analysis of positive behavior support. In R. Gersten, E. P. Schiller, & S. Vaughan (Eds.), Contemporary Special Education Research: Syntheses of the Knowledge Base on Critical Instructional Issues (pp. 137–178). Mahwah, NJ: Lawrence Erlbaum Associates. Pustejovsky, J. E., Hedges, L. V, & Shadish, W. R. (2014). Design-comparable effect sizes in multiple baseline designs: A general modeling framework. Journal of Educational and Behavioral Statistics, 39(5), 368–393. doi:10.3102/1076998614547577 Pustejovsky, J. E. (2015). Measurement-comparable effect sizes for single-case studies of free-operant behavior. Psychological Methods, 20(3), 342–359. doi:10.1037/met0000019 Pustejovsky, J. E., & Swan, D. M. (2015). Four methods for analyzing partial interval recording data, with application to single-case research. Multivariate Behavioral Research, 50(3), 365–380. doi:10.1080/00273171.2015.1014879 Shadish, W. R., Rindskopf, D. M., & Hedges, L. V. (2008). The state of the science in the meta-analysis of single-case experimental designs. Evidence-Based Communication Assessment and Intervention, 2(3), 188–196. doi:10.1080/17489530802581603 Shadish, W. R., Hedges, L. V, & Pustejovsky, J. E. (2014). Analysis and meta-analysis of single-case designs with a standardized mean difference statistic: A primer and applications. Journal of School Psychology, 52(2), 123–147. doi:10.1016/j.jsp.2013.11.005 Shadish, W. R., Hedges, L. V, Horner, R. H., & Odom, S. L. (2015). The role of between-case effect size in conducting, interpreting, and summarizing single-case research. Washington, DC. Retrieved from http://ies.ed.gov/ncser/pubs/2015002/ Rodriguez, B. J., & Anderson, C. M. (2014). Integrating a social behavior intervention during small group academic instruction using a total group criterion intervention. Journal of Positive Behavior Interventions, 16(4), 234–245. doi:10.1177/1098300713492858