An appetizer to meta-analysis Week 4.2

What is a meta-analysis “an analysis of analyses”

Why do a meta-analysis? A fundamental concept in statistics: VARIANCE If there is no variability in your data, you don’t need inferential statistics: t-test, ANOVA, regression, etc., and of course meta-analysis. You can simply run descriptive statistics.

Fact Even if you have a time machine, every time you run a study, you are going to have a different set of means. This implies you will also get different variances. Which implies you will get different…

Grappling with uncertainty As a practitioner, knowing whether a programme will work or not is rarely the goal. Knowing how well it will work is desirable, especially for cost-benefit analysis This is what it means by “meta-analytical thinking”.

Four Steps of Meta Analysis Identify your studies Determine eligibility of studies Inclusion: which ones to keep Exclusion: which ones to throw out Extract data from the studies Data analysis (this session)

1. Identify your studies Be methodical: plan first List of popular databases to search Pubmed/Medline/PsycINFO/Google Scholar Other strategies you may adopt Hand search (go to the library...) Personal references, and emails

How to Search for literature Formulate your question appropriately “Are social norms effective?”  what do you mean? Be smart about searching Boolean logic (AND, OR, NOT) Comb references of published articles

2. Determine eligibility GIGO: Garbage in, garbage out Crappy studies distorts your effect size estimate Need to evaluate quality of studies Meta-analysts are also good methodologists. So, you cannot rely on statisticians to do meta-analysis for you. Beware of file drawer effect

2. Determine eligibility Keep the ones with relevant high levels of evidence good quality Is there a good control group? Are the DVs robust (valid and reliable)? Is the manipulation valid? Were there methodological flaws? Attrition rate?

ARE THE STUDIES ELIGIBLE FOR MA (STEP I)? Action Plan ARE THE STUDIES ELIGIBLE FOR MA (STEP I)? NO DISCARD YES Extract the relevant data Enter into a spreadsheet the relevant data

3. What data do you extract? Different type of meta-analysis requires different ‘raw ingredients’. But two things are common: Effect size estimate (d, r, RR, count, etc.) Sample size per condition  needed for calculating inverse variance weighting Example: Mean differences {mean1, sd1, mean2, sd2, n1, n2}  d

Different types of meta-analysis Largely depends on the type of effect size you are dealing with Mean difference between groups (continuuous DV) # of deaths between group (count proportion DV) Mean likelihood difference (RR DV) The math differs, but the idea is always the same. Hint: type ?meta after loading library(meta)to see the various types.

4. Analyze data Combine data to arrive at a summary, 3 measures Effect size estimate With 95% confidence intervals Two plots Forest Plot Funnel Plot (not useful if you have few studies)

Weighted means Large studies should be given more weight Large studies have smaller standard errors. Why? Recall the formula. That’s why it’s called inverse variance weighting. (The math is not too complicated, but not worth knowing in my opinion.)

A realistic timeframe for meta-analysis Search for published papers: 1-2 mths Code papers: 5 per day, 3-6 mths+ Analyze data: 5 mins – 1 mth Forest and funnel plots: 2 mins - 12 hrs Write up results: 3-6 mths

Resource in R for meta-analysis https://www.r-bloggers.com/practicing-meta-analytic-thinking-through-simul Google: kovalchik_meta_tutorial.pdf

What have we learnt in the last three stats classes? Develop a lifelong learning attitude You will not learn many techniques in UG or Grad school. Strong base + good attitude = competence In-depth understanding of math is not needed to do stuff You don’t need to know how atoms work to be a carpenter. You just have to have faith that atoms work in a certain way ≡ have faith in statisticians who devise certain techniques