1. Meta-analysis & psychotherapy outcome research

2. What is a meta-analysis? How is a meta-analysis conducted?
Overview
What is a meta-analysis?
How is a meta-analysis conducted?
Robinson et al, 1990: Is psychotherapy effective as a treatment for depression?
Meta-analysis

3. What is a meta-analysis?
"Meta-analysis refers to the analysis of analyses the statistical analysis of a large collection of analysis results from individual studies for the purpose of integrating the findings. It connotes a rigorous alternative to the casual, narrative discussions of research studies [= traditional review papers] which typify our attempts to make sense of the rapidly expanding research literature." (G. Glass, 1976)
By comparing results from many different studies, we can look for general conclusions in domains where conclusions of individual studies may be uncertain and/or disputed, because they are subject to many variables, or because the literature is a big mess, offering selective support for conflicting viewpoints
Meta-analysis

4. What is meta-analysis?
Each trial is treated as one estimate of an effect, assumed to be underlain by some global population value
This is analogous to individual questions on a psychometric test, inasmuch as each one provides some information on the construct to which it relates, but may be subject to error or contamination by itself
Meta-analysis

5. Five steps in conducting a meta-analysis
1.) Define a question that you want to answer
2.) Select studies according some specified inclusion criteria
3.) Select your statistical model (fixed effects versus random effects)
4.) Calculate summary effects
5.) Interpret the results
Meta-analysis

6. 1.) Define a question that you want to answer
The question may be posed in terms of an independent variable, or a set of commonly researched variables, or by causes and consequences of important variables.
How effective is psychotherapy for depression?
Meta-analysis

7. 2.) Select studies according some specified inclusion criteria
The purpose is to include only comparable studies of good quality
In Robinson et al, 1990:
1.) Studies from
2.) Patients suffering only and explicitly from depression
3.) Outpatients only
4.) Adults only
5.) Included a comparison of treatment versus no treatment or different types of therapy = no case histories, no pre/post designs (Why not?)
6.) Verbal psychotherapy only
Meta-analysis

8. 3.) Select your statistical model
Fixed effects: Assumes that the data are consistent with the treatment effect being constant (i.e. there is a single fixed treatment effect = no interaction between study and effect)
Random effects: Assumes that the studies included in the meta-analysis are a random sample generalizing to the domain of all similar studies (under the finding that there is a study X treatment interaction: i.e. different treatment effects in different studies)
We can still generalize, under the assumption that our studies constitute a random sampling of possible effects, but the confidence interval will be wider = less certainty in conclusions
Meta-analysis

9. What is an effect size?
A standard p-value tells you how certain you can be that two (or more) groups are really different (how likely it is that any apparent difference is really due to chance)
A p-value depends on two things: the size of the effect and the size of the sample.
You can get a significant effect either if the effect is very big (despite a small sample) or if the sample is very big (despite a small effect size).
You can't average p-values, because they do not reflect the same things in different studies
‘Effect size’ is a way of quantifying the size of the difference in standardized terms
It is the standardized mean difference between two groups
Meta-analysis

10. How does it work?
We won't consider the (rather complex) mathematical details in this class
Specialized computer programs are available
The basic idea is to convert values of of significance (i.e. t F, c, or p values) into some common format: Pearson's r, or Cohen's d (a measure of effect size = the standardized mean difference between two groups)
These common values must be corrected for random error (within each study) due to sample size, measurement error, and range restrictions (i.e. selection for studies selecting for extremes in the possible range)
It is more difficult to control for (although one can check for) publication bias (only significant results get published) and publication quality
Meta-analysis

11. How does it work?
When the disparate measures from each study are all converted to a single measure, they are directly comparable (assuming they used comparable outcome measures!)
The process is analogous to converting disparate measures (number of hockey goals scored versus number of baskets achieved) to z-scores to make them directly comparable.
The effect size measure is standardized and is essentially equivalent to a z-score
Meta-analysis

12. The problem of moderator variables
Moderator variables: Extraneous variables influencing the results in a particular study
There are mathematical ways to deal with these
Meta-analysis

13. 4.) Calculate summary effects
In Robinson et al, the mean effect size of psychotherapy compared to no treatment (37 studies) was 0.73
What does this mean?
An effect size of 0.73 means that patients who received psychotherapy had outcomes about 3/4 of a standard deviation better than those who had no treatment
The mean effect size of psychotherapy compared to waiting list was 0.84
The mean effect size of psychotherapy compared to placebo was 0.28 (p > 0.05)- What does this tell us?
Meta-analysis

14. 4.) Calculate summary effects
There was no reliable difference between types of therapy, but in individual planned comparisons cognitive, behavioral, and cognitive-behavioral were all better than 'general verbal'
The effect size comparing psychotherapy to (all) drug therapy was 0.13 (p < 0.05), but there was no difference between a combination of the two versus psychotherapy alone (d = 0.01; p > 0.05) or versus drug therapy alone (d = 0.17; p > 0.05)
Meta-analysis

15. 5.) Interpret the results
The results of this meta-analysis suggest that psychotherapy does work as a treatment for depression
BUT it does not work better than placebos
It works marginally (but significantly) better than drug therapy, but the two treatments do not have a significantly additive effect
Treatment cost- in human and dollar terms- must be factored into treatment planning
Some costs may vary between individuals: some hate drugs and others hate paying more than they need to
Meta-analysis

16. Some final thoughts on meta-analysis
Items : Testing ::
Studies : Meta-analysis
Testing : Psychological constructs
Meta-analysis : Experimental/Clinical Effects
Just as we can use psychometric testing to quantify the degree to which a construct matters for any particular purpose, we can use meta-analysis to quantify the degree to which measured effects matter for a specified purpose
Just as constructs exist in a quantifiable haze of certainty, so do treatments and effects
Meta-analysis

17. Some final thoughts on meta-analysis
It is important to distinguish significance testing from measurement of effect sizes
When we select from extremes of a normal distribution (high/low), we can often get highly reliable effects that are nevertheless of negligible import in explaining the phenomenon under study
i.e. Some variables in lexical access have highly reliable effects on RT when selected from extremes, but correlate with RT with r < 0.1
How much of the variance in RT do these highly reliable effects account for?
Meta-analysis

18. Some final thoughts on meta-analysis
More is not always better: Effects that are significant individually may be accounting for shared variance, and therefore not sum together
i.e. drug therapy and psychotherapy are both better than nothing, but but adding drugs to psychotherapy is not better than psychotherapy alone
The question you ask matters: 'Which treatment is better?' ≠ 'Which treatment should I prescribe?'
It is one thing to show that two treatments differ, but quite another to make a decision about which one is best for any particular individual
Meta-analysis