1. Meta-Analysis of Single Subject Designs William R. Shadish University of California, Merced Natasha Beretvas University of Texas, Austin Eden Nagler CUNY, The Graduate Center

2. Presentations William Shadish: “Meta-Analysis of Single Subject Designs: An Overview” Natasha Beretvas: “Effect Sizes for Meta- Analysis of Single Subject Designs” Eden Nagler: “Analyzing Data from Small N Designs using Multilevel Models”

3. Single Subject Designs (SSDs) A time series of observations on one person. The repeated introduction, removal, and/or reintroduction of treatment. Demonstration of causality through showing that the effect is present when the treatment is present, but absent otherwise. Many different variations, for example:

4. A Hypothetical SSD

5. McClannahan et al., 1990 Home 3 Home 2 Home 1 No Feedback Mean Personal Appearance Score Sessions Feedback

6. Meta-Analysis of Single Subject Designs IES awarded two grants on this topic: –Shadish & Rindskopf –Beretvas A general interest in using single subject designs as a source of evidence for “Evidence Based Practice” But methods for the synthesis of such designs are diverse, with no consensus about which method is best.

7. Shadish and Rindskopf Grant Review the Literature and Create Annotated Bibliographies (DONE) –Past Meta-Analyses of Single Subject Designs –Methods for Meta-Analyses of Single Subject Designs Identify Sample Data Sets for Meta-Analysis of Single Subject Designs (DONE) Creation of Coding Protocols (IN PROCESS) Analytic and Computer Simulation of Pertinent Statistics (YEARS 2 and 3)

8. Past Meta-Analyses of Single Subject Designs The annotated bibliography is at: http://faculty.ucmerced.edu/wshadish/Past %20MA%20of%20SSDs.htm http://faculty.ucmerced.edu/wshadish/Past %20MA%20of%20SSDs.htm We found 22 such meta-analyses Published between 1985 and 2005, with increasing frequency in the last decade. Using a wide variety of methods:

9. Methods for Meta-Analysis of Single Subject Designs We located 33 scholarly works on the topic, which are posted as an annotated bibliography at http://faculty.ucmerced.edu/wshadish/Meth ods%20for%20MA%20of%20SSD.htm http://faculty.ucmerced.edu/wshadish/Meth ods%20for%20MA%20of%20SSD.htm They were published between 1971 and 2005 As you can see, the frequency of this work has increased significantly over time:

11. Sample Data Sets One small and one large data set: –16 studies reporting SSD’s to improve reading. Annoted Bibliography is at: http://faculty.ucmerced.edu/wshadish/Reading%20Im provement.htm http://faculty.ucmerced.edu/wshadish/Reading%20Im provement.htm We are currently extracting data from these –86 studies reporting SSD’s for treatment of learning disabilities. Annoted Bibliography is at : http://faculty.ucmerced.edu/wshadish/Learning%20Di sabilities.htm http://faculty.ucmerced.edu/wshadish/Learning%20Di sabilities.htm

12. Two Kinds of Data, Two Kinds of Problems Multiple Studies, each of which uses a different SSD (the hard problem) –Different participants –Different dependent variables (though perhaps conceptually related) –Different (related) treatments –Different time scales –Different designs: AB, ABAB, ABC, etc. Multiple SSDs that use the same design within one study on multiple participants –The easy problem –Probably analyze with HLM or LGM

13. Which Leads Nicely Into the Next Talks Natasha Beretvas will talk about the first problem Eden Nagler will talk about the second problem. Then we will have time for open discussion.