1. Statistical Models for the Analysis of Single-Case Intervention Data Introduction to:  Regression Models  Multilevel Models

2. Why consider statistical models? Can provide effect size estimates and confidence intervals for those estimates. e.g., We are 95% confident the immediate shift in level for Jenny was an increase of between 11 and 14 minutes of time spent reading, or between 1.5 and 2.0 standard deviations

3. Regression Imagine a scatter plot showing the relationship between motivation and achievement.

4. Regression allows us to summarize the relationship between the variables.

5. Often when we think of regression we think of each data point coming from a different individual, but all the observations could come from the same individual.

6. Rise Run What is the rate of change for Jody?

7. For single-case studies we expect a discontinuity BaselineIntervention

8. What is the shift in level for Jody? Effect =

9. What is the immediate shift in level and the shift in slope for Jody? -5 -4 -3 -2 -1 0 1 2 3 4 5

10. Issues to Keep in Mind You may have to choose at what point in time you should calculate the effect size

11. You may want to standardize the effect size If so, what SD should be use?

12. There needs to be a match between the trajectory specified in the model and what is seen in the data Effect = b 1 ? This seems incorrect

13. What is seen may require specification of a complex growth trajectory Do you think specification tends to be easier when there are more or less observations in a phase?

14. Correct model specification requires more than just correctly specifying the growth trajectory Should you assume the errors (e i ): are independent? have common variance? are normally distributed?

15. Cindy Lucy George John Imagine we have multiple cases

16. A separate regression could be obtained for each case Or a multilevel analysis could be run

17. Multilevel Model Multilevel models allow us to answer additional questions: What is the average treatment effect? What is the average treatment effect? Does the size of the effect vary across participants? Does the size of the effect vary across participants? What factors relate to effect size? What factors relate to effect size?

18. What is the average effect for the participants? Average Effect = γ 10 Average Baseline Level = γ 00

19. Does the size of the effect vary across participants?

20. What factors relate to effect size? ADD Non-ADD

21. Issues to Keep in Mind There still needs to be a match between the trajectory specified in the model and what is seen in the data Effect = b 1 ? This seems incorrect

22. Correct model specification requires assumptions about multiple error terms Should you assume the errors (e ij, r 0j, r 1j ) are independent? Normally distributed?

23. If one standardizes the effect size, what SD should be used for standardization Within case variance? Between case variance?

24. Imagine we have multiple studies 24

25. Multilevel models were developed for large sample size conditions, but single-case applications tend to have a very small number of cases. Given small sample sizes the variances (e.g. variance in the treatment effect across participants) will generally be more poorly estimated than the averages (e.g. the average treatment effect).

26. Example Analysis Summarize results from 5 studies that examined the effect of intervention on autistic children’s speech DV: Percent intervals with child speech IV: Intervention based on increased parent verbalizations Design: Multiple baseline across participants

27. Laski, K. E., Charlop, M. H., & Schreibman, L. (1988). Training parents to use the natural language paradigm to increase their autistic children’s speech. Journal of Applied Behavior Analysis, 21, 391-400.

28. One child from Laski et al. 28

29. 2-Level Model for Laski et al. 29 Parameter EstimatedEstimateSEp Fixed Effects Average Baseline Level ( θ 00 )34.178.16.0030 Average Treatment Effect ( θ 10 )33.575.00.0001 Variance Components Variance in Baseline Level ( )577.8299.1.0267 Variance in Treatment Effects ( )185.4110.5.0468 Covariance u 0 & u 1 ( )-294.1110.5.0838 Variance Within Person ( )147.318.7<.0001

30. Software Code: SAS, R 30 SAS: proc mixed covtest; class Case; model Y= D / solution ddfm=sat; random intercept D / sub=Case type=un; R: twolevel <- lmer(Y ~ D + (1 + D | Case), data2) summary(twolevel)

31. 3-Level Model for All Five Studies 31

32. 3-Level Model Results 32 Parameter EstimatedEstimate SEp Fixed Effects Average Baseline Level ( γ 000 )18.716.31.0345 Average Treatment Effect ( γ 100 )31.729.37.0309 Variance Components Between Study Variance in Baseline Level ( )128.2128.3.1588 Between Study Variance in Treatment Effects ( )377.2320.7.1198 Covariance v 0 & v 1 ( )165.5169.8.3296 Within Study Variance in Baseline Level ( )316.9103.6.0011 Within Study Variance in Treatment Effects ( )222.281.8.0033 Covariance u 0 & u 1 ( )-47.870.1.4956 Variance Within Person ( )328.715.7<.0001

33. Software Code: SAS, R 33 SAS: proc mixed covtest; class Study Case; model Y= D / solution ddfm=sat; random intercept D / sub=Study type=un; random intercept D / sub=Case(Study) type=un; R: threelevel <- lmer(Y ~ D + (1 + D | Study:Case) + (1 + D | Study), data3) summary(threelevel)

34. Statistical models (regression and multilevel) provide a flexible approach for estimating treatment effects from single-case data, but care must be taken to ensure the model being used is consistent with the data being analyzed. Conclusion

35. Applications and Illustrations Baek, E., & Ferron, J. M. (2013). Multilevel models for multiple-baseline data: Modeling across participant variation in autocorrelation and residual variance. Behavior Research Methods, 45, 65-74. Baek, E. K., Moeyaert, M., Petit-Bois, M., Beretvas, S. N., Van den Noortgate, W., & Ferron, J. M. (2014). The use of multilevel analysis for integrating single-case experimental design results within a study and across studies. Neuropsychological Rehabilitation, 24, 590-606. Ferron, J. M., Moeyaert, M., Van den Noortgate, W., & Beretvas, S. N. (in press). Estimating casual effects from multiple-baseline studies: Implications for design and analysis. Psychological Methods. Moeyaert, M., Ferron, J., Beretvas, S. N., & Van den Noortgate, W. (2014). From a single-level analysis to a multilevel analysis of single-case experimental designs. Journal of School Psychology, 52, 191-211. Moeyaert, M., Ugille, M., Ferron, J., Onghena, P., Heyvaert, M., Beretvas, S. N., & Van den Noortgate, W. (in press). Estimating intervention effects across different types of single-subject experimental designs: Empirical illustration. School Psychology Quarterly. Rindskopf, D., & Ferron, J. (2014). Using multilevel models to analyze single-case design data. In T. R. Kratochwill & J. R. Levin (Eds.), Single-Case Intervention Research: Statistical and Methodological Advances (pp. 221-246). American Psychological Association. Shadish, W.R., Kyse, E.N., & Rindskopf, D.M. (2013). Analyzing data from single-case designs using multilevel models: new applications and some agenda items for future research. Psychological Methods, 18, 385-405. Van den Noortgate, W., & Onghena, P. (2003). Combining single-case experimental data using hierarchical linear models. School Psychology Quarterly, 18, 325-346. Van den Noortgate, W., Onghena, P. (2007). The aggregation of single-case results using hierarchical linear models. The Behavior Analyst Today, 8(2), 196-209. Van den Noortgate, W., & Onghena, P. (2008). A multilevel meta-analysis of single-subject experimental designs. Evidence-Based Communication Assessment and Intervention, 2, 142-151.