Meta-analysis & psychotherapy outcome research

What is a meta-analysis? How is a meta-analysis conducted? Overview What is a meta-analysis? How is a meta-analysis conducted? Robinson et al, 1990: Is psychotherapy effective as a treatment for depression? Meta-analysis

What is a meta-analysis? "Meta-analysis refers to the analysis of analyses . . . the statistical analysis of a large collection of analysis results from individual studies for the purpose of integrating the findings. It connotes a rigorous alternative to the casual, narrative discussions of research studies [= traditional review papers] which typify our attempts to make sense of the rapidly expanding research literature." (G. Glass, 1976) By comparing results from many different studies, we can look for general conclusions in domains where conclusions of individual studies may be uncertain and/or disputed, because they are subject to many variables, or because the literature is a big mess, offering selective support for conflicting viewpoints Meta-analysis

What is meta-analysis? Each trial is treated as one estimate of an effect, assumed to be underlain by some global population value This is analogous to individual questions on a psychometric test, inasmuch as each one provides some information on the construct to which it relates, but may be subject to error or contamination by itself Meta-analysis

Five steps in conducting a meta-analysis 1.) Define a question that you want to answer 2.) Select studies according some specified inclusion criteria 3.) Select your statistical model (fixed effects versus random effects) 4.) Calculate summary effects 5.) Interpret the results Meta-analysis

1.) Define a question that you want to answer The question may be posed in terms of an independent variable, or a set of commonly researched variables, or by causes and consequences of important variables. How effective is psychotherapy for depression? Meta-analysis

2.) Select studies according some specified inclusion criteria The purpose is to include only comparable studies of good quality In Robinson et al, 1990: 1.) Studies from 1976-1986 2.) Patients suffering only and explicitly from depression 3.) Outpatients only 4.) Adults only 5.) Included a comparison of treatment versus no treatment or different types of therapy = no case histories, no pre/post designs (Why not?) 6.) Verbal psychotherapy only Meta-analysis

3.) Select your statistical model Fixed effects: Assumes that the data are consistent with the treatment effect being constant (i.e. there is a single fixed treatment effect = no interaction between study and effect) Random effects: Assumes that the studies included in the meta-analysis are a random sample generalizing to the domain of all similar studies (under the finding that there is a study X treatment interaction: i.e. different treatment effects in different studies) We can still generalize, under the assumption that our studies constitute a random sampling of possible effects, but the confidence interval will be wider = less certainty in conclusions Meta-analysis

What is an effect size? A standard p-value tells you how certain you can be that two (or more) groups are really different (how likely it is that any apparent difference is really due to chance) A p-value depends on two things: the size of the effect and the size of the sample. You can get a significant effect either if the effect is very big (despite a small sample) or if the sample is very big (despite a small effect size). You can't average p-values, because they do not reflect the same things in different studies ‘Effect size’ is a way of quantifying the size of the difference in standardized terms It is the standardized mean difference between two groups Meta-analysis

How does it work? We won't consider the (rather complex) mathematical details in this class Specialized computer programs are available The basic idea is to convert values of of significance (i.e. t F, c, or p values) into some common format: Pearson's r, or Cohen's d (a measure of effect size = the standardized mean difference between two groups) These common values must be corrected for random error (within each study) due to sample size, measurement error, and range restrictions (i.e. selection for studies selecting for extremes in the possible range) It is more difficult to control for (although one can check for) publication bias (only significant results get published) and publication quality Meta-analysis

How does it work? When the disparate measures from each study are all converted to a single measure, they are directly comparable (assuming they used comparable outcome measures!) The process is analogous to converting disparate measures (number of hockey goals scored versus number of baskets achieved) to z-scores to make them directly comparable. The effect size measure is standardized and is essentially equivalent to a z-score Meta-analysis

The problem of moderator variables Moderator variables: Extraneous variables influencing the results in a particular study There are mathematical ways to deal with these Meta-analysis

4.) Calculate summary effects In Robinson et al, the mean effect size of psychotherapy compared to no treatment (37 studies) was 0.73 What does this mean? An effect size of 0.73 means that patients who received psychotherapy had outcomes about 3/4 of a standard deviation better than those who had no treatment The mean effect size of psychotherapy compared to waiting list was 0.84 The mean effect size of psychotherapy compared to placebo was 0.28 (p > 0.05)- What does this tell us? Meta-analysis

4.) Calculate summary effects There was no reliable difference between types of therapy, but in individual planned comparisons cognitive, behavioral, and cognitive-behavioral were all better than 'general verbal' The effect size comparing psychotherapy to (all) drug therapy was 0.13 (p < 0.05), but there was no difference between a combination of the two versus psychotherapy alone (d = 0.01; p > 0.05) or versus drug therapy alone (d = 0.17; p > 0.05) Meta-analysis

5.) Interpret the results The results of this meta-analysis suggest that psychotherapy does work as a treatment for depression BUT it does not work better than placebos It works marginally (but significantly) better than drug therapy, but the two treatments do not have a significantly additive effect Treatment cost- in human and dollar terms- must be factored into treatment planning Some costs may vary between individuals: some hate drugs and others hate paying more than they need to Meta-analysis

Some final thoughts on meta-analysis Items : Testing :: Studies : Meta-analysis Testing : Psychological constructs Meta-analysis : Experimental/Clinical Effects Just as we can use psychometric testing to quantify the degree to which a construct matters for any particular purpose, we can use meta-analysis to quantify the degree to which measured effects matter for a specified purpose Just as constructs exist in a quantifiable haze of certainty, so do treatments and effects Meta-analysis

Some final thoughts on meta-analysis It is important to distinguish significance testing from measurement of effect sizes When we select from extremes of a normal distribution (high/low), we can often get highly reliable effects that are nevertheless of negligible import in explaining the phenomenon under study i.e. Some variables in lexical access have highly reliable effects on RT when selected from extremes, but correlate with RT with r < 0.1 How much of the variance in RT do these highly reliable effects account for? Meta-analysis

Some final thoughts on meta-analysis More is not always better: Effects that are significant individually may be accounting for shared variance, and therefore not sum together i.e. drug therapy and psychotherapy are both better than nothing, but but adding drugs to psychotherapy is not better than psychotherapy alone The question you ask matters: 'Which treatment is better?' ≠ 'Which treatment should I prescribe?' It is one thing to show that two treatments differ, but quite another to make a decision about which one is best for any particular individual Meta-analysis