Meta Analysis An Introduction

What… is… it? A “study of studies,” i.e., averaging results across studies in a given domain to get a better estimate of population parameters (Allen & Preiss, 2007; Hunter & Schmidt, 2004). Key ingredient = measures of effect size (usually Pearson’s r or Cohen’s d)

Why????????? 1. To reduce problems associated with SAMPLING ERROR at the individual study level. How big is this problem? According to Hunter, sampling is error is “massive” with sample sizes of N=100 or less, as we typically have in research. Result: VERY frequent Type II errors….

Example Monte Carlo study with population effect size (p) =.10, and 19 studies of sample sizes of N=30, N=68, and N=400. How many times did the conventional p <.05 test flag (*) an r as significant? 1/19 times, for a 95% error rate! What about for N = 68? 2/19, for an error rate of 89%. For N = 400, the error rate is STILL 47%!

Why????????? Hunter & Schmidt (1990) note that though Type I is typically 5%, Type II error rate for an average sample size (N = 80) with an average effect size (d =.40) and alpha level at p =.05 is… …still about 50%! How do we fix this? By increasing sample size in individual studies (rarely done) or through meta analysis.

But… it gets even worse…. In addition to random (sampling) error, studies also suffer from systematic problems, such as: – Measurement artifacts. – Issues of design. – Choice of sample. – Anything else that makes study results different. Fortunately, meta analysis can correct or at least account for these problems. Yay!

Points to Consider Meta analysis isn’t as easy as it may seem on the surface (see next slides), but… It provides the most accurate estimate of population parameters possible (vs. individual studies or literature reviews), and… FYI: Natural sciences also have variability in study results and most use forms of meta analysis to deal with divergent study findings.

So how can I do a meta analysis? It’s not available as an option in SPSS….  It seems like many meta analysts do the calculations by hand(!), though there is some software available for it, e.g., – Jack Hunter’s free DOS programs for “bare bones” meta analysis and other corrections he advocates for. – Commercially available software like CMA.CMA

Meta Analysis Example Sherry (2001) estimated the effect size of violent video games on aggression through meta analysis. Steps included: 1. Study selection and coding (p ) – Exhaustive lit search done for studies on video game violence and aggression, from – Of 900 initial returns, 25 studies were identified from which an effect size could be estimated. – Relevant info was recorded on coding sheets.

Meta Analysis Example 2. Effect sizes (r) estimated (p ) – Many had to be calculated from other stats. – Nonsignificant findings also had to be dealt with. 3. Overall effect size estimated (p ) – Mean effect size (weighted by sample size) and variance from all studies calculated. – Sampling error residual variance accounted for. – Variance from moderating variables accounted for. – See Table 1 (p. 420) for a typical summary.

Results Highlights Table 2 shows mean effect sizes and residual variance (p. 421) – High probability of moderating variables indicated. – Methodological variables include survey vs. experiment and type of outcome measure. – Theoretical variables include age, type of game violence, and length of game exposure. Table 3 shows how the theoretical variables correlate with effect size.

Results Highlights MULTIVARIATE STAT ALERT! These variables were entered into a multiple regression equation with effect size as the DV and moderators as IVs. Sherry’s reason? “To control for the effect of moderators on each other (e.g., suppression).” Check out the results in first full paragraph on p. 422.

What did we learn? Converting to d, overall effect size of games on aggression is.30, smaller than that found for television of.65 (Paik and Comstock, 1994). More recent games have a larger effect size. Player age also positively related to effect size. Effect size negatively related to playing time, however. Results used for theoretical advancement.

The Latest Meta Analysis Piece Theoretical assessment of evidence (2007): 1. No support for social learning theory. 2. Little support for catharsis, though not studied properly. Time finding points to it, as does declining violence at macro level. 3. Arousal (excitation transfer) and priming supported by available evidence. New model: PRIMED AROUSAL.

You Tube BREAK How prevalent is META ANALYSIS on YouTube? Not very…. Not very BONUS VIDEO tangentially related to meta analysis and the next slide: Powerthirst 2: Re- DominationPowerthirst 2: Re- Domination

Other Recent Examples Paul et al. (2007) Third Person Effect study— 32 studies (N = 45,729) indicate a substantial effect size of r =.50. Q: Moderators? A: Message, sampling, and respondent type. Other topics in Preiss et al. (2007) include effects of agenda setting, sexually explicit media, frightening media, music, health campaigns, spiral of silence, and more….

Final Considerations Shows importance of “replication, replication, REPLICATION” (Hunter 2001) in science. Some limitations, however…. Have to have a number of studies in a given domain before a meta analysis is worthwhile; otherwise, a literature review should suffice (Pfau, 2007). Q: Are there any areas you think are ripe for a meta analysis?

DO IT! Questions, comments, suggestions? Thank You.