1. A Little History

2. Cotes
In the early 18th century, the English mathematician Roger Cotes computed weighted averages of measurements made by different astronomers

3. Pearson (1904) The Effects of a Typhoid Vaccine on
Incidence of Typhoid
Fatalities from Typhoid
Pearson averaged the correlations from six studies, and concluded they were positive but not high enough to adopt the vaccine.

4. R.A. Fisher 1944 Source of the idea of cumulating probability values
“When a number of quite independent tests of significance have been made, it sometimes happens that although few or none can be claimed individually as significant, yet the aggregate gives an impression that the probabilities are on the whole lower than would often have been obtained by chance” (p. 99).
Source of the idea of cumulating probability values

5. W.G. Cochran 1953
Discusses a method of averaging means across independent studies
Laid-out much of the statistical foundation that modern meta-analysis is built upon (e.g., inverse variance weighting and homogeneity testing)

6. Problems with Narrative Reviews
Coping with an increasingly large literature
Memory biases (recall for confirming evidence)
Vulnerability of narrative summaries to bias
Though bias can never be eliminated from science
One person’s bias is another person’s judgment.
Failure to weight studies by sample size
Tendency to rely on significance tests rather than effect size
“Box score” counts can be misleading

7. Cooper and Rosenthal (1980)
Seven studies given to 41 people
Who were then randomly assigned to a narrative review or a meta-analysis condition (technically, combined p-values).
Studies had ambiguous results—two clearly rejected the null, the other five reported no effect.
Meta-analytic group more likely to perceive support for the hypothesis and a larger effect based on the combined p-value, e.g.,
68% of the meta-analytic group considered rejecting the null, compared to only 27% of the narrative reviewers.

8. Why not the p-value approach?
There is a long history of methods for combining probability levels.
However, this approach does not given a measure of effect size because of its dependency on sample size.
And it ignores the direction of the effect.
Since a form of a combined probability is yielded by effect-size meta-analysis (the p value associated with the average effect), the combined probability method has fallen out of favor.

9. Gene Glass (1976): Primary, Secondary, and Meta-Analysis
Primary Analysis: The analysis of primary study data by the person who gathered it.
Secondary Analysis: The analysis of primary study data by someone else.
Meta-Analysis: The analysis of summary statistics (e.g., means) from primary studies (typically, no access to individual data from primary studies).

10. The Key Problem That Glass Solved
How to summarize results from many studies when each study used different measures in different metrics (with different means and standard deviations) to measure outcome. For depression, for example:
Beck Depression Inventory
MMPI Depression Scale
Investigator-generated scales

11. The Solution: Standardized Effect Sizes

12. Major Approaches: Social Sciences
Glass: The pioneer, but not widely used today (unweighted, failure to aggregate effect sizes to the study level, use of control group standard deviation)
Hunter and Schmidt: Widely used in industrial-organizational psychology, focus on correlations (e.g., validity generalization), uses many adjustments (e.g., for reliability of measures, for restriction of range), less formally statistical.
Hedges and Olkin: Most widely used approach (the focus of most of this workshop), most formally statistical, most general.

13. Major Approaches: Medicine
Even though meta-analysis began in education and psychology, it has spread widely now to many other fields.
Particularly well-developed in medicine
The Cochrane Collaboration
Evidence-Based Practice
Lots more money than the social sciences, so able to fund many more people both to use and to develop methods for meta-analsyis
Methods generally follow Hedges and Olkin,
Often independently developed
Many major developments beyond Hedges and Olkin, e.g.,
Bayesian methods (Spiegelhalter)
Methods for dichotomous outcome effect sizes

14. So You Want to Do a Meta-Analysis? Time and Costs

15. Meta-analysis uses the same basic steps as all research
Formulating problem (are there enough studies)
Gathering studies (file drawer problem)
Coding studies (creating data)
Computing effect sizes (how to do for nonstandard data)
Analyzing data (lots of options)
Interpreting and presenting results (graphics)
Some people call this full process “systematic review” reserving “meta-analysis” for analyzing data

16. Costs of Doing Meta-Analysis
Meta-Analytic work is time-consuming and needs a substantial budget—though not as high as many primary studies.
Steinberg et al. (1997) Am J Epid
Identified tasks and costs for doing a meta-analysis on 11 studies of effects of oral contraceptives on ovarian cancer
Not fully representative of meta-analysis because studies drawn from a prior meta-analysis of individual patient data rather than a literature search.
Here are the cost and time estimates:

17. About 1000 hours of time over people
About $50K

18. More on Time
Allen and Olkin (1999) JAMA: “Estimating Time to Conduct a Meta-Analysis From Number of Citations Retrieved”.
Documented time on tasks on 37 meta-analyses conducted by a private company specializing in meta-analysis.
Median time was 1110 hours
Range hours…
Then they graphed total number of citations initially retrieved against total time to do the meta-analysis:

19. The 721 hours constant reflects the fact that there are some constant time commitments required at startup.
Add about 15 minutes for each citation retrieved, slightly less as more and more citations are found.

20. More on Time Component mean (SD) times were
Pre-analysis search, retrieval and database development: 558 (337) hours
Includes protocol development, searches, library retrieval, abstract management, study matrix construction, paper screening and blinding, data extraction and quality scoring, data entry, and data matrix construction.
Statistical analysis: 144 (106) hours
Report and manuscript writing: 206 (125) hours
Other (administrative): 201 (193) hours
Includes proposal development, project-specific correspondence, project meetings and administration, project management, and training

21. Startup Time
If you have never done a meta-analysis before, the time allocated should be increased considerably.
Particularly underestimated tasks:
Training and monitoring of coders
Learning new statistical methods
Best Advice: Make your first one small.

22. Computer Programs
They abound (see Will’s web page at
We will focus primarily on R
But also see
Lipsey and Wilson’s SPSS macros
They also have macros for SAS and Stata that work essentially the same way.
Go to
The Stata metadialog add-on package.
Instructions for downloading are in your handout package.
See Sterne, J.A.C. (2009). Meta-Analysis in Stata. College Station, TX: Stata Press.
Comprehensive Meta-Analysis

23. Gathering Studies

24. The Question The first step is deciding on the question to be asked:
E.g., Shadish and Baldwin 2005: What are the effects of behavioral marital therapy (BMT) compared to no treatment?
The question initially need not be too specific—better to be inclusive at first and then exclude later.
You may discover operationalizations of constructs that you did not anticipate
Broader inclusion helps explore generalization

25. The Literature Search: What?
Inclusion Exclusion Criteria
Which treatments, people, outcomes? For example, in Shadish’s meta-analysis of BMT:
Including Cognitive Behavioral?
Effects on anything? Just on divorce or the Marital Attitude Survey?
Excluding studies of “enrichment”?
Which designs: e.g., RCTs only?
Language restrictions?
Limited to certain years?
Both published and unpublished sources?

26. The Literature Search: Where?
The Bibliographies of Past Meta-Analyses and Literature Reviews.
The Bibliography of every pertinent study retrieved
Standard Electronic Databases
E.g., PubMed, Web of Science, PsycInfo, Dissertation Abstracts International
What keywords, how far back?
Hand Searches of Journals (Recent Years)
Funding agencies for final reports.
A serious search for unpublished studies?
Contacts with Experts
Registries of Studies Initiated
Archives of convention programs and papers
Keep a record of keywords used:
Keep a flow chart and record of inclusions-exclusions: