1. Single-Case Intervention Research Training Institute
Madison, WI - June, 2018
James E. Pustejovsky
Effect size measures for single-case designs: between-case standardized mean differences

2. Parametric between-case effect sizes
Within-case standardized mean difference is not on the same scale as SMD from between-groups designs (e.g., between-subjects randomized trial).
Shadish, Rindskopf, & Hedges (2008) asked: Can we estimate a SMD based on the data from a single-case design that IS in the same metric a SMD from a between-groups design?
Why do this? (Shadish, Hedges, Horner, & Odom, 2015)
Translation of single-case research for researchers who work primarily with between-groups designs.
Comparison of results from single-case studies and between-groups studies, for purposes of understanding the utility and limitations of each type of design.
Synthesis involving both single-case and between-groups designs.

3. Between-case SMD What is the SMD from a between-groups experiment?
These quantities can be estimated from single-case data using a hierarchical model that describes variation within and between participants.
But we’ll need to have a sample of multiple participants (bare minimum of 3, more for more complex models).

4. Estimating between-case SMDs:
The broad strategy:
Develop a hierarchical model that describes a) the functional relationship for each case and b) how the outcome and functional relationship vary across cases.
Use the hierarchical model to imagine a hypothetical between-subjects experiment with the same population of participants, same treatment, same outcomes.
Calculate the between-case SMD for the hypothetical experiment.
Imagine running a hypothetical experiment. In a sense, you could think of this as “simulating” data for a set of participants

5. Estimating BC-SMDs: Basic model
Hedges, Pustejovsky, and Shadish proposed BC-SMD estimators for a basic hierarchical linear model
HPS (2012): Treatment reversal (ABAB) designs
HPS (2013): Multiple baseline/multiple probe designs
Assumptions:
Baseline is stable (no baseline trend).
Intervention effect is immediate (no intervention-phase trend).
The outcome is normally distributed around mean level for each case, with variance σ2.
Within-case errors follow a first-order auto-regressive process (serial dependence).
The baseline level for each case is normally distributed, with variance τ2.
The treatment effect is constant across cases.

6. Estimating BC-SMDs: Basic model
Moment estimation
Web app:
R package:
SPSS macro:
Shadish, Hedges, and Pustejovsky (2014) includes worked examples
Restricted maximum likelihood estimation (recommended)
Both methods produce estimates of BC-SMD (corrected for small-sample bias) and accompanying standard error.

7. Rodriguez & Anderson (2014) example (using REML estimation)

8. More flexible models for BC-SMDs
Pustejovsky, Hedges, and Shadish (2014) extend the basic model to allow for less restrictive assumptions:
Variability of individual treatment effects
Baseline time trends (constant or varying across participants)
Time trends in treatment phase (constant or varying across participants)
More flexible models require more cases
For models with time trends, also need to specify a focal follow-up time

9. Barton-Arwood, Wehby, & Falk (2005)
Barton-Arwood, Wehby, & Falk (2005). Reading instruction for elementary-age students with emotional and behavioral disorders: Academic and behavioral outcomes
Intervention consisted of Horizon Fast Track reading program and Peer-Assisted Learning Strategies

10. Barton-Arwood, Wehby, & Falk (2005) Effect size calculations
Allow for baseline and intervention time trends, both varying across cases.
Focal follow-up time after 13 sessions.
BC-SMD estimate of 0.91 (SE = 0.93).
Substantial heterogeneity in intervention time trends.

11. Limitations of between-case SMD
Describes an average effect across a set of cases
Conceals potential individual heterogeneity
Inherent consequence of comparability with between-groups effect sizes.
Technical limitations
Only available for treatment reversal (ABAB) and multiple baseline/multiple probe across participant designs.
Requires at least 3 participants, preferably more.
Assumes normally distributed, interval-scale outcomes.
More work needed on evaluating model selection, model fit
Use between-case effect sizes as a complement to (not a replacement for) within-case effect size measures
Between-groups effect sizes only provide information about average effects, rather than individual effects.

12. References
Beretvas, S. N., & Chung, H. (2008). An evaluation of modified R2-change effect size indices for single-subject experimental designs. Evidence-Based Communication Assessment and Intervention, 2(3), 120–128. doi: /
Busk, P. L., & Serlin, R. C. (1992). Meta-analysis for single-case research. In T. R. Kratochwill & J. R. Levin (Eds.), Single-Case Research Design and Analysis: New Directions for Psychology and Education (pp. 187–212). Hillsdale, NJ: Lawrence Erlbaum Associates, Inc.
Campbell, J. M. (2003). Efficacy of behavioral interventions for reducing problem behavior in persons with autism: a quantitative synthesis of single-subject research. Research in Developmental Disabilities, 24(2), 120–138. doi: /S (03)
Campbell, J. M., & Herzinger, C. V. (2010). Statistics and single subject research methodology. In D. L. Gast (Ed.), Single Subject Research Methodology in Behavioral Sciences (pp. 417–450). New York, NY: Routledge.
Gingerich, W. J. (1984). Meta-analysis of applied time-series data. Journal of Applied Behavioral Science, 20(1), 71–79. doi: /
Hedges, L. V, Pustejovsky, J. E., & Shadish, W. R. (2012). A standardized mean difference effect size for single case designs. Research Synthesis Methods, 3, 224–239. doi: /jrsm.1052
Hedges, L. V, Pustejovsky, J. E., & Shadish, W. R. (2013). A standardized mean difference effect size for multiple baseline designs across individuals. Research Synthesis Methods, 4(4), 324–341. doi: /jrsm.1086
Kahng, S., Iwata, B. a, & Lewin, A. B. (2002). Behavioral treatment of self-injury, 1964 to American Journal of Mental Retardation : AJMR, 107(3), 212–221. doi: / (2002)107<0212:BTOSIT>2.0.CO;2
Maggin, D. M., Swaminathan, H., Rogers, H. J., O’Keeffe, B. V, Sugai, G., & Horner, R. H. (2011). A generalized least squares regression approach for computing effect sizes in single-case research: Application examples. Journal of School Psychology, 49(3), 301–321. doi: /j.jsp
Marquis, J. G., Horner, R. H., Carr, E. G., Turnbull, A. P., Thompson, M., Behrens, G. A., … Doolabh, A. (2000). A meta-analysis of positive behavior support. In R. Gersten, E. P. Schiller, & S. Vaughan (Eds.), Contemporary Special Education Research: Syntheses of the Knowledge Base on Critical Instructional Issues (pp. 137–178). Mahwah, NJ: Lawrence Erlbaum Associates.
Pustejovsky, J. E., Hedges, L. V, & Shadish, W. R. (2014). Design-comparable effect sizes in multiple baseline designs: A general modeling framework. Journal of Educational and Behavioral Statistics, 39(5), 368–393. doi: /
Pustejovsky, J. E. (2015). Measurement-comparable effect sizes for single-case studies of free-operant behavior. Psychological Methods, 20(3), 342–359. doi: /met
Pustejovsky, J. E., & Swan, D. M. (2015). Four methods for analyzing partial interval recording data, with application to single-case research. Multivariate Behavioral Research, 50(3), 365–380. doi: /
Shadish, W. R., Rindskopf, D. M., & Hedges, L. V. (2008). The state of the science in the meta-analysis of single-case experimental designs. Evidence-Based Communication Assessment and Intervention, 2(3), 188–196. doi: /
Shadish, W. R., Hedges, L. V, & Pustejovsky, J. E. (2014). Analysis and meta-analysis of single-case designs with a standardized mean difference statistic: A primer and applications. Journal of School Psychology, 52(2), 123–147. doi: /j.jsp
Shadish, W. R., Hedges, L. V, Horner, R. H., & Odom, S. L. (2015). The role of between-case effect size in conducting, interpreting, and summarizing single-case research. Washington, DC. Retrieved from
Rodriguez, B. J., & Anderson, C. M. (2014). Integrating a social behavior intervention during small group academic instruction using a total group criterion intervention. Journal of Positive Behavior Interventions, 16(4), 234–245. doi: /