A Little History

Cotes In the early 18th century, the English mathematician Roger Cotes computed weighted averages of measurements made by different astronomers

Pearson (1904) The Effects of a Typhoid Vaccine on Incidence of Typhoid Fatalities from Typhoid Pearson averaged the correlations from six studies, and concluded they were positive but not high enough to adopt the vaccine.

R.A. Fisher 1944 Source of the idea of cumulating probability values “When a number of quite independent tests of significance have been made, it sometimes happens that although few or none can be claimed individually as significant, yet the aggregate gives an impression that the probabilities are on the whole lower than would often have been obtained by chance” (p. 99). Source of the idea of cumulating probability values

W.G. Cochran 1953 Discusses a method of averaging means across independent studies Laid-out much of the statistical foundation that modern meta-analysis is built upon (e.g., inverse variance weighting and homogeneity testing)

Problems with Narrative Reviews Coping with an increasingly large literature Memory biases (recall for confirming evidence) Vulnerability of narrative summaries to bias Though bias can never be eliminated from science One person’s bias is another person’s judgment. Failure to weight studies by sample size Tendency to rely on significance tests rather than effect size “Box score” counts can be misleading

Cooper and Rosenthal (1980) Seven studies given to 41 people Who were then randomly assigned to a narrative review or a meta-analysis condition (technically, combined p-values). Studies had ambiguous results—two clearly rejected the null, the other five reported no effect. Meta-analytic group more likely to perceive support for the hypothesis and a larger effect based on the combined p-value, e.g., 68% of the meta-analytic group considered rejecting the null, compared to only 27% of the narrative reviewers.

Why not the p-value approach? There is a long history of methods for combining probability levels. However, this approach does not given a measure of effect size because of its dependency on sample size. And it ignores the direction of the effect. Since a form of a combined probability is yielded by effect-size meta-analysis (the p value associated with the average effect), the combined probability method has fallen out of favor.

Gene Glass (1976): Primary, Secondary, and Meta-Analysis Primary Analysis: The analysis of primary study data by the person who gathered it. Secondary Analysis: The analysis of primary study data by someone else. Meta-Analysis: The analysis of summary statistics (e.g., means) from primary studies (typically, no access to individual data from primary studies).

The Key Problem That Glass Solved How to summarize results from many studies when each study used different measures in different metrics (with different means and standard deviations) to measure outcome. For depression, for example: Beck Depression Inventory MMPI Depression Scale Investigator-generated scales

The Solution: Standardized Effect Sizes  

Major Approaches: Social Sciences Glass: The pioneer, but not widely used today (unweighted, failure to aggregate effect sizes to the study level, use of control group standard deviation) Hunter and Schmidt: Widely used in industrial-organizational psychology, focus on correlations (e.g., validity generalization), uses many adjustments (e.g., for reliability of measures, for restriction of range), less formally statistical. Hedges and Olkin: Most widely used approach (the focus of most of this workshop), most formally statistical, most general.

Major Approaches: Medicine Even though meta-analysis began in education and psychology, it has spread widely now to many other fields. Particularly well-developed in medicine The Cochrane Collaboration Evidence-Based Practice Lots more money than the social sciences, so able to fund many more people both to use and to develop methods for meta-analsyis Methods generally follow Hedges and Olkin, Often independently developed Many major developments beyond Hedges and Olkin, e.g., Bayesian methods (Spiegelhalter) Methods for dichotomous outcome effect sizes

So You Want to Do a Meta-Analysis? Time and Costs

Meta-analysis uses the same basic steps as all research Formulating problem (are there enough studies) Gathering studies (file drawer problem) Coding studies (creating data) Computing effect sizes (how to do for nonstandard data) Analyzing data (lots of options) Interpreting and presenting results (graphics) Some people call this full process “systematic review” reserving “meta-analysis” for analyzing data

Costs of Doing Meta-Analysis Meta-Analytic work is time-consuming and needs a substantial budget—though not as high as many primary studies. Steinberg et al. (1997) Am J Epid Identified tasks and costs for doing a meta-analysis on 11 studies of effects of oral contraceptives on ovarian cancer Not fully representative of meta-analysis because studies drawn from a prior meta-analysis of individual patient data rather than a literature search. Here are the cost and time estimates:

About 1000 hours of time over people About $50K

More on Time Allen and Olkin (1999) JAMA: “Estimating Time to Conduct a Meta-Analysis From Number of Citations Retrieved”. Documented time on tasks on 37 meta-analyses conducted by a private company specializing in meta-analysis. Median time was 1110 hours Range 216-2518 hours… Then they graphed total number of citations initially retrieved against total time to do the meta-analysis:

The 721 hours constant reflects the fact that there are some constant time commitments required at startup. Add about 15 minutes for each citation retrieved, slightly less as more and more citations are found.

More on Time Component mean (SD) times were Pre-analysis search, retrieval and database development: 558 (337) hours Includes protocol development, searches, library retrieval, abstract management, study matrix construction, paper screening and blinding, data extraction and quality scoring, data entry, and data matrix construction. Statistical analysis: 144 (106) hours Report and manuscript writing: 206 (125) hours Other (administrative): 201 (193) hours Includes proposal development, project-specific correspondence, project meetings and administration, project management, and training

Startup Time If you have never done a meta-analysis before, the time allocated should be increased considerably. Particularly underestimated tasks: Training and monitoring of coders Learning new statistical methods Best Advice: Make your first one small.

Computer Programs They abound (see Will’s web page at http://faculty.ucmerced.edu/wshadish/software/other-meta-analysis-software-sites) We will focus primarily on R But also see Lipsey and Wilson’s SPSS macros They also have macros for SAS and Stata that work essentially the same way. Go to http://mason.gmu.edu/~dwilsonb/ma.html. The Stata metadialog add-on package. Instructions for downloading are in your handout package. See Sterne, J.A.C. (2009). Meta-Analysis in Stata. College Station, TX: Stata Press. Comprehensive Meta-Analysis

Gathering Studies

The Question The first step is deciding on the question to be asked: E.g., Shadish and Baldwin 2005: What are the effects of behavioral marital therapy (BMT) compared to no treatment? The question initially need not be too specific—better to be inclusive at first and then exclude later. You may discover operationalizations of constructs that you did not anticipate Broader inclusion helps explore generalization

The Literature Search: What? Inclusion Exclusion Criteria Which treatments, people, outcomes? For example, in Shadish’s meta-analysis of BMT: Including Cognitive Behavioral? Effects on anything? Just on divorce or the Marital Attitude Survey? Excluding studies of “enrichment”? Which designs: e.g., RCTs only? Language restrictions? Limited to certain years? Both published and unpublished sources?

The Literature Search: Where? The Bibliographies of Past Meta-Analyses and Literature Reviews. The Bibliography of every pertinent study retrieved Standard Electronic Databases E.g., PubMed, Web of Science, PsycInfo, Dissertation Abstracts International What keywords, how far back? Hand Searches of Journals (Recent Years) Funding agencies for final reports. A serious search for unpublished studies? Contacts with Experts Registries of Studies Initiated Archives of convention programs and papers Keep a record of keywords used: Keep a flow chart and record of inclusions-exclusions: