Meta-analysis statistical models: Fixed-effect vs. random-effects Ryan Singh, MPH H676 October 13, 2016

Two statistical models for MA Intent of MA is to get the mean of the effect sizes from each study (summary effect)-2 models: Fixed-effect model -one true effect across all studies Random-effects model -true effect varies between studies True effect size (Θ) – effect in underlying population, and the effect size that would be observed if n=∞ Observed effect (Y) – effect size that is actually observed

FIXED-EFFECT MODEL RANDOM-EFFECTS MODEL -one true effect -true effect varies -differences b/w studies due to sampling error only -differences b/w studies more than sampling error -large studies more influential in ES estimation -weight between all studies is more balanced -account for within study uncertainty -account for within study AND between study uncertainty -summary effect=common effect -summary effect=mean effect -null hypothesis=0 effect in every study -null hypothesis=0 mean effect -summary effect limited to identified population -summary effect more generalizable

Fixed-effect model All factors that could affect ES same across studies True ES is same Only variation from random (sampling) error Assume that if n=∞, then no sampling error (draw) When to use All included studies are functionally identical Goal is to get ES for identified population *Need to estimate true ES based on observed effects

Fixed-effect model To perform FE analysis: Start with observed effects to estimate population effect Observed effect is the sum of the true effect and sampling error Assign appropriate weights to all studies-inverse of each study’s variance Calculate weighted mean (summary effect) with observed ESs and weights Calculate all summary statistics (V, SE, CI, Z, p)

Precision and weighting Addresses the accuracy of the predicted/observed effect as an estimate of the true effect encompasses variance, standard error, & confidence interval Dependent on sample size, study design, & ? Weights assigned to studies to minimize within study error

Why do some meta-analysts suggest setting w equal to the sample size, rather than the inverse of the variance (according to DeCoster)? I have no clue.

Fixed-effect model What would happen to observed ES (Y) if n=∞?

Fixed-effect model What would happen to observed ES (Y) if n=∞? sampling error=0, so observed ES=true ES What is the relationship between sample size and variance?

Fixed-effect model What would happen to observed ES (Y) if n=∞? sampling error=0, so observed ES=true ES What is the relationship between sample size and variance? small sample=large variance

Random-effects model Studies not similar enough to assume true ES is same across studies Studies differ for a variety of reasons Assume true effects are normally distributed (draw picture) Goal-use all observed effects to estimate mean of all true ESs Distance b/w overall mean and observed effect in a study consists of 2 parts: Sampling error True variation in ES Observed effect is the sum of: 1) the overall mean, 2) deviation of true effect from overall mean, & 3) deviation of observed effect from true effect Variance for both needs to be considered

Random-effects model To perform RE analysis: Start with observed effects to estimate population effect Observed effect is the sum of the mean and total error Assign appropriate weights to all studies-inverse of each study’s variance Calculate weighted mean (summary effect) with observed ESs and weights Calculate all summary statistics (V, SE, CI, Z, p)

Random-effects model To calculate variance, need: Within study variance=sampling error Between study variance (τ2) AKA variance of ESs across studies Dependent on SD of distribution of true effects across studies Applies to all studies in MA Can be estimated *does not account for within study variance

Additional questions Why are smaller studies less significant in FE analysis? Conversely, why does the weight of each study in RE analysis matter? Thinking back to our readings on study inclusion, what issues arise with including less studies (due to design flaw perhaps) and the limitation of smaller analyses? OR What limitations are there with our (potentially) smaller samples for this class?

Additional questions Why are smaller studies less significant in FE analysis? better information about same ES from larger studies Conversely, why does the weight of each study in RE analysis matter? each study represents a different effect-want to represent each one Thinking back to our readings on study inclusion, what issues arise with including less studies (due to design flaw perhaps) and the limitation of smaller analyses? OR What limitations are there with our (potentially) smaller samples for this class? estimate of between study variance has poor precision

FIXED-EFFECT MODEL RANDOM-EFFECTS MODEL -one true effect -true effect varies -differences b/w studies due to sampling error only -differences b/w studies more than sampling error -large studies more influential in ES estimation -weight between all studies is more balanced -account for within study uncertainty -account for within study AND between study uncertainty -summary effect=common effect -summary effect=mean effect -null hypothesis=0 effect in every study -null hypothesis=0 mean effect -summary effect limited to identified population -summary effect more generalizable

Group Discussion Topic? How will you go about finding your effect size? Which model (FE or RE) will you be using? Questions/issues you have at this point?