1. Single-Case Effect Size and Meta-Analytic Measures
Part I (yesterday):
Overview
Non-Overlap Measures of Effect Size
Part II (today):
Parametric Measures of Effect Size
Meta-Analytic Methods

2. Alternative Effect Sizes: Parametric
A series of parametric effect sizes developed to overcome noted concerns with PND

3. Within-Case Standardized Mean Difference

4. 2. Account for Baseline Trends +a +c 3. Sensitive to Size of Effect --
PND
PEM
ECL
NAP
TauU d
Stable - +
2. Account for Baseline Trends +a +c
3. Sensitive to Size of Effect --
4. Known Sampling Distribution +b +d
5. Comparability
6. No Extreme Values
aAssuming trend is linear and can be extrapolated
bAssuming independence
cSome technical reservations
dAssuming independence, normality, homogeneity of variance

5. Log Response Ratio (R)
Appropriate for outcomes on a ratio scale (e.g., counts)
The natural logarithm is used to make the range less restricted
If the means are equal then R = 0
Should correct for small sample size:

6. 2. Account for Baseline Trends +a +c 3. Sensitive to Size of Effect --
PND
PEM
ECL
NAP
TauU d R
Stable - +
2. Account for Baseline Trends +a +c
3. Sensitive to Size of Effect --
4. Known Sampling Distribution +b +d
5. Comparability
6. No Extreme Values
aAssuming trend is linear and can be extrapolated
bAssuming independence
cSome technical reservations
dAssuming independence, normality, homogeneity of variance

7. Within-Case Standardized Regression Coefficient

8. Within-Case Standardized Regression Coefficient:
Options for when to estimate the effect:
Time centered so that 0 corresponds to the first intervention observation
Time centered so that 0 corresponds to the 3rd intervention observation
Options for linear or curvilinear trends:
Options for different assumptions about errors:
Independent, normally distributed, common variance: OLS
First order autoregressive: GLS; Bayesian; ReML

9. 2. Account for Baseline Trends +a +c +e 3. Sensitive to Size of Effect
PND
PEM
ECL
NAP
TauU d R β
Stable - +
2. Account for Baseline Trends +a +c +e
3. Sensitive to Size of Effect --
4. Known Sampling Distribution +b +d +f
5. Comparability
6. No Extreme Values
7. Easy to Estimate
aAssuming trend is linear and can be extrapolated
bAssuming independence
cSome technical reservations
dAssuming independence, normality, homogeneity of variance
eAssuming form of trend is known and can be extrapolated
fAssuming specific error structure (e.g., first order autoregressive)

10. Between-Case Standardized Regression Coefficient
Basic Multilevel Modeling Approach
Within-case variance Between-case variance

11. Between-Case Standardized Regression Coefficient
Problem with the Basic Multilevel Modeling Approach
Between-case variance is very biased so effect size is very biased (> 10%) when we use restricted maximum likelihood (ReML) for estimation.

12. Between-Case Standardized Regression Coefficient
Options:
Non-multilevel (Hedges, Pustejovsky, & Shadish, 2012, 2013)
- No trends
Multilevel – Bayesian Estimation (Swaminathan, Rogers, & Horner, 2014)
Multilevel – ReML with Small Sample Adjustments (Pustejovsky, Hedges, & Shadish, 2014)
Between-case variance is very biased so effect size is very biased (> 10%) if we use traditional multilevel modeling estimation methods: maximum likelihood or restricted maximum likelihood.

13. Between-Case Standardized Regression Coefficient
ReML with small sample size adjustment (Pustejovsky et al., 2014)
1. Run a multilevel model that is appropriate to the data
2. Estimate the effect based on the appropriate regression coefficients and variance components

14. Between-Case Standardized Regression Coefficient
ReML with small sample size adjustment (Pustejovsky et al., 2014)
3. Adjust the effect size estimate for small sample size
where υ is the degrees of freedom
4. Estimate the approximate sampling variance in

15. Between-Case Standardized Regression Coefficient
Run the following lines in R from the command prompt:
install.packages("devtools") library(devtools) setInternet2(use = TRUE) download.file(" destfile = "scdhlm_0.2.1.zip")
install.packages("scdhlm_0.2.1.zip", repo = NULL)
library(scdhlm)
vignette("Estimating-effect-sizes")

16. 2. Account for Baseline Trends +a +c +e 3. Sensitive to Size of Effect
PND
PEM
ECL
NAP
TauU d R β δ
Stable - +
2. Account for Baseline Trends +a +c +e
3. Sensitive to Size of Effect --
4. Known Sampling Distribution +b +d +f
5. Comparability
6. No Extreme Values
7. Easy to Estimate
8. Can Estimate with N=1
aAssuming trend is linear and can be extrapolated
bAssuming independence
cSome technical reservations
dAssuming independence, normality, homogeneity of variance
eAssuming form of trend is known and can be extrapolated
fAssuming specific error structure (e.g., first order autoregressive)

17. Effect Size Recommendations
Should you provide effect sizes when you write up a primary study?
If so, which one or ones?

18. Meta-Analysis of the Effect Sizes
Find effect size weights
more precise effect sizes get more weight
Make a data set with all the effect sizes and their weights
Estimate the average effect size
Use multilevel modeling to account for nesting of effect sizes within studies
Look for factors that moderate the size of the effect
Add case and study characteristics as predictors in the multilevel model

19. Find Effect Size Weights
If you pick an effect size with a sampling distribution, then
Find the standard error for the effect size,
Find the sampling variance for the effect size,
Find the weights, which are the inverse of the sampling variance

20. Make Data Set Study Case ES W X Z 1 .8 11.11 30 2 .6 15.44 3 .9 18.983 .9
18.98 4 .7
5.45 45 5 .5
7.32 6
5.99 7
6.49

21. Multilevel Model of Effect Sizes to Find Average
Level-1 Sampling error:
Level-2 Participants:
Level-3 Studies:

22. Add Predictors to Model to Explore Moderation
Level-1 Sampling error:
Level-2 Participants:
Level-3 Studies:

23. Example Meta-Analysis
Topic: The impact of choice-making on problem behaviors
Original Reference: Shogren, Faggella-Luby, Bae, & Wehmeyer (2004)
Number of Single-Case Studies: 13
Number of Cases: 31
Effect Size Used: (originally PND, in this reanalysis )
Moderators: Male (1=male; 0=female)
Order (1 = task order; 0 = either/or)
Website with Data, SAS & R code, and Output:

24. SAS Code PROC MIXED NOCLPRINT NOITPRINT COVTEST method=reml;
CLASS study case;
WEIGHT prec;
MODEL g = /SOLUTION ddfm=satterth;
RANDOM intercept /SUB = study;
RANDOM intercept /SUB=case(study);
PARMS /HOLD = 3;
run;

25. SAS Output

26. SAS Code with Moderators
PROC MIXED NOCLPRINT NOITPRINT COVTEST method=reml;
CLASS study case;
WEIGHT prec;
MODEL g = male order /SOLUTION ddfm=satterth;
RANDOM intercept /SUB = study;
RANDOM intercept /SUB=case(study);
PARMS /HOLD = 3;
run;

27. SAS Output

28. Publication Bias
Publication Bias occurs when the outcome of a study influences the chance that it is published.
If small effects are less likely to be published, meta-analyses will overestimate the average effect.
Do you think this happens in single-case research?

29. Cumulative Meta-Analysis
Study 1
Study 2
Study 3
Study 4
Study 5
Study 6
Study 7
Study 8
Study 9
Study 10
Study 11
Study 12
Study 13
Study 14
Study 15
-.5 .5
1.0

30. Collaborative Care vs Standard Care: Depression Outcomes
(Gilbody & Fletcher, 2006)
Wilkinson et al, 1993
Callahan et al, 1994
Katon et al, 1995
Blanchard et al, 1995
Katon et al, 1996
Mann et al, 1999
Peveler et al, 1999
Katon et al, 1999
Coleman et al, 1999
Wells et al, 2000
Hunkeler et al, 2000
Simon et al, 2000
Finley et al, 2000
Whooley et al, 2000
Katzelnick et al, 2000
Rost et al, 2001
Katon et al, 2001
Unutzer et al, 2001
Swindle et al, 2003
Brook et al, 2003
Akerblad et al, 2003
Araya et al, 2003
Oslin et al, 2003
Datto et al, 2003
Rickles et al, 2004
Bruce et al, 2004
Cappocia et al, 2004
Simon et al, 2004
-1.0
-.5 .5

31. References
Busk, P. L, & Serlin, R. C. (1992). Meta-analysis for single-case research. In T. R. Kratochwill & J. R. Levin (Eds.). Single- case research design and analysis: New directions for psychology and education. Hillsdale, NJ: LEA.
Hedges, L. V., Pustejovsky, J. E., & Shadish, W. R. (2013). A standardized mean difference effect size for multiple for multiple baseline designs across individuals. Research Synthesis Methods, 4,
Hedges, L. V., Pustejovsky, J. E., & Shadish, W. R. (2012). A standardized mean difference effect size for single case designs. Research Synthesis Methods, 3,
Gilbody, S. Bower, P., Fletcher, J. Richards, D., & Sutton, A. J. (2006). Collaborative care for depression: A cummmulative meta-analysis and review of longer-term outcomes. Archives of Internal Medicine, 166,
Maggin, D. M., Swaminathan, H., Rogers, H. J., O’Keeffe, B. V., Sugai, G., & Horner, R. H. (2011). A generalized least squares regression approach for computing effect sizes in single-case research: Application examples. Journal of School Psychology,
Moeyaert, M., Ugille, M., Ferron, J., Beretvas, S. N., & Van den Noortgate, W. (2013). The three-level synthesis of standardized single-subject experimental data: A Monte Carlo simulation study. Multivariate Behavioral Research, 48,
Pustejovsky, J. E. (2015). Measurement-comparable effect sizes for single-case studies of free-operant behavior. Psychological Methods, 20,
Pustejovsky, J. E., & Ferron, J. (in press). Research synthesis and meta-analysis of single-case research. In J. M. Kauffman, D. P. Hallahan, and P. C. Pullen (Eds), Handbook of Special Education, 2nd edition (pp. ) New York: Routledge.
Pustejovsky, J. E., Hedges, L. V., & Shadish, W. R. (2014). Design-comparable effect sizes in multiple baseline designs: A general modeling framework. Journal of Educational and Behavioral Statistics, 39,

32. References
Shadish, W. R., Hedges, L. V., & Pustejovsky, J. E. (2014). Analysis and meta-analysis of single-case designs with a standardized mean difference statistic: A primer with applications. Journal of School Psychology, 52,
Shogren, K. A., Faggella-Luby, M. N., Bae, S. J., & Wehmeyer, M. L. (2004). The effect of choice-making as an intervention for problem behavior: A meta-analysis. Journal of Positive Behavior Interventions, 6,
Swaminathan, H., Rogers, H. J., & Horner, R. H. (2014). An effect size measure and Bayesian analysis of single-case designs. Journal of School Psychology, 52,
Ugille, M., Moeyaert, M., Beretvas, S. N., Ferron, J., & Van den Noortgate, W. (2012). Multilevel meta-analysis of single- subject experimental designs: A simulation study. Behavior Research Methods, 44,
Van den Noortgate, W., & Onghena, P. (2008). A multilevel meta-analysis of single-subject experimental designs. Evidence-Based Communication Assessment and Intervention, 2,