1. An appetizer to meta-analysis
Week 4.2

2. What is a meta-analysis
“an analysis of analyses”

3. Why do a meta-analysis? A fundamental concept in statistics: VARIANCE
If there is no variability in your data, you don’t need inferential statistics: t-test, ANOVA, regression, etc., and of course meta-analysis.
You can simply run descriptive statistics.

4. Fact
Even if you have a time machine, every time you run a study, you are going to have a different set of means.
This implies you will also get different variances.
Which implies you will get different…

5. Grappling with uncertainty
As a practitioner, knowing whether a programme will work or not is rarely the goal.
Knowing how well it will work is desirable, especially for cost-benefit analysis
This is what it means by “meta-analytical thinking”.

6. Four Steps of Meta Analysis
Identify your studies
Determine eligibility of studies
Inclusion: which ones to keep
Exclusion: which ones to throw out
Extract data from the studies
Data analysis (this session)

7. 1. Identify your studies Be methodical: plan first
List of popular databases to search
Pubmed/Medline/PsycINFO/Google Scholar
Other strategies you may adopt
Hand search (go to the library...)
Personal references, and s

8. How to Search for literature
Formulate your question appropriately
“Are social norms effective?”  what do you mean?
Be smart about searching
Boolean logic (AND, OR, NOT)
Comb references of published articles

9. 2. Determine eligibility
GIGO: Garbage in, garbage out
Crappy studies distorts your effect size estimate
Need to evaluate quality of studies
Meta-analysts are also good methodologists.
So, you cannot rely on statisticians to do meta-analysis for you.
Beware of file drawer effect

10. 2. Determine eligibility
Keep the ones with
relevant
high levels of evidence
good quality
Is there a good control group?
Are the DVs robust (valid and reliable)?
Is the manipulation valid?
Were there methodological flaws?
Attrition rate?

11. ARE THE STUDIES ELIGIBLE FOR MA (STEP I)?
Action Plan
ARE THE STUDIES ELIGIBLE FOR MA (STEP I)? NO
DISCARD
YES
Extract the relevant data
Enter into a spreadsheet the relevant data

12. 3. What data do you extract?
Different type of meta-analysis requires different ‘raw ingredients’.
But two things are common:
Effect size estimate (d, r, RR, count, etc.)
Sample size per condition  needed for calculating inverse variance weighting
Example: Mean differences
{mean1, sd1, mean2, sd2, n1, n2}  d

13. Different types of meta-analysis
Largely depends on the type of effect size you are dealing with
Mean difference between groups (continuuous DV)
# of deaths between group (count proportion DV)
Mean likelihood difference (RR DV)
The math differs, but the idea is always the same.
Hint: type ?meta after loading library(meta)to see the various types.

14. 4. Analyze data Combine data to arrive at a summary, 3 measures
Effect size estimate
With 95% confidence intervals
Two plots
Forest Plot
Funnel Plot (not useful if you have few studies)

15. Weighted means Large studies should be given more weight
Large studies have smaller standard errors. Why? Recall the formula.
That’s why it’s called inverse variance weighting. (The math is not too complicated, but not worth knowing in my opinion.)

16. A realistic timeframe for meta-analysis
Search for published papers: 1-2 mths
Code papers: 5 per day, 3-6 mths+
Analyze data: 5 mins – 1 mth
Forest and funnel plots: 2 mins - 12 hrs
Write up results: 3-6 mths

17. Resource in R for meta-analysis
Google: kovalchik_meta_tutorial.pdf

18. What have we learnt in the last three stats classes?
Develop a lifelong learning attitude
You will not learn many techniques in UG or Grad school. Strong base + good attitude = competence
In-depth understanding of math is not needed to do stuff
You don’t need to know how atoms work to be a carpenter. You just have to have faith that atoms work in a certain way ≡ have faith in statisticians who devise certain techniques