Meta-analysis of observational studies Nicole Vogelzangs Department of Psychiatry & EMGO + institute

Outline RCTs vs. observational studies Data extraction Meta-analysis –Combining results (statistical pooling) –Studying sources of heterogeneity (subgroup analysis and meta-regression analysis) Observational II-2

RCT vs. observational research hypothesis therapyetiology, diagnosis, prognosis methodology cleardivers, different designs validity design highvarying risk of confounding lowhigh analyses clearadjustment for confounding univariablemultivariable RCT Observational research Observational II-3

Heterogeneity More heterogeneity in observational studies compared with RCTs: –Design –Population (less strict selection criteria) –Different ways of assessing exposure and disease –Adjustment for confounding Studying the sources of heterogeneity is an important aim of analyses in reviews of observational studies Observational II-4

An example Depression as a risk factor for the onset of type 2 diabetes mellitus - Knol, Twisk et al. Diabetologica 2006 Selection criteria –Longitudinal studies on depression and onset of DM2 –Exclusion: studies including prevalent DM2 cases (only persons ‘at risk’) Insufficient data to calculate RR, OR or HR Observational II-5

Publication bias? Studies with large SE (small N) and weaker association appear absent Observational II-6

Outline RCTs vs. observational studies Data extraction Meta-analysis –Combining results (statistical pooling) –Studying sources of heterogeneity (subgroup analysis and meta-regression analysis) Observational II-7

Data extraction For each study: 1.Effect estimate (association determinant – outcome) For confounding corrected effect! 2.Variance or standard error (SE) of the effect estimate (=> weight in meta-analysis) 3.Information on potential sources of heterogeneity Observational II-8

1. Effect estimate Effect estimate based on multivariable analysis, adjusted for all possible confounders (age, sex, BMI, blood pressure, etc.) Cohort study: –Logistic regression: OR –Cox regression: hazard ratio (HR ≈ RR) Patient-control study –Logistic regression: OR Observational II-9

Are OR and RR/HR exchangeable? Only when risk of outcome is small! Odds = p / 1-p large risk, e.g. p=0.30: odds = 0.30 / 0.70 = 0.43 small risk, e.g. p=0.01: odds = 0.01 / 0.99 ≈ 0.01 => when risk is large, OR is overestimation of RR alcohol consumption - bladder cancer: OR is OK computer job - neck/shoulder complaints: OR not OK Observational II-10

2. Standard error of the effect estimate SE often not reported Calculate from –confidence interval –p-value (need sufficient decimals; p <.05 is insufficient) Observational II-11

3. Sources of heterogeneity Characteristics of study population Method of measurement for exposure to risk factor / prognostic factor Method of measurement outcome or disease Aspects of design Analysis Observational II-12

Sources of heterogeneity - example Population: differences in age, gender, potential confounders Measurement of exposure: method for assessment of depression (questionnaire, interview, diagnosis care provider) Outcome: method for assessment diabetes (screening or self-report) Observational II-13

Sources of heterogeneity - example Aspects of design: cohort study –Duration of follow-up? –Response / drop-out rate? –Which confounders are measured? –Which methods are used to exclude diabetes patients at baseline? Analysis –How is exposure defined? (score on depression scale or dichotomous [cut-off]) –Which confounders are used in analysis? Observational II-14

Reducing heterogeneity Some study differences can be canceled out by converting published data: –Defining exposure (e.g. merge several depression categories into yes/no depression) –Degree of adjustment for confounding => Possibilities strongly depend on reporting Observational II-15

Outline RCTs vs. observational studies Data extraction Meta-analysis –Combining results (statistical pooling) –Studying sources of heterogeneity (subgroup analysis en meta-regression analysis) Observational II-16

Meta-analysis Combining results (statistical pooling) Enter for each study: –effect estimate –weight (1 / SE 2 ) Fixed effects model (often not realistic) Random effects model Study influence of sources of heterogeneity –Subgroup analysis –Meta-regression analysis Observational II-17

Meta-analysis - example RR (REM): 1.37 ( ) Q-test: p = 0.02 Observational II-18

Study heterogeneity: subgroup analysis 1 characteristic at once (e.g. duration of follow-up) Not too many categories (e.g. > or < 5 year) Stratify for these categories Pool effect estimates within each category Observational II-19

Subgroup analysis - example RR Yes (n = 3)No (n = 6) Exclusion undetected diabetes at baseline Observational II-20

Study heterogeneity: meta-regression Several characteristics can be studied simultaneously (e.g. follow-up duration, design type, age of population) Efficient and more power Also possible for ordinal and continuous measures Observational II-21

Meta-regression What is meta-regression? Weighted linear regression Studies are subject of analysis Dependent variable (Y) = effect estimate of study Weight = 1 / SE 2 Independent variables (X) = sources of heterogeneity (effect modifiers) Y = b 0 + b 1 X + unexplained variance Observational II-22

Dependent variable (outcome) Linear regression: dependent variable (Y) should be +/- normally distributed OR en RR: skewed distribution Log-transformation:ln (OR) = b 0 + b 1 X ln (RR) = b 0 + b 1 X Observational II-23

Meta-regression: sleep position and sudden infant death Year of study 0 0,5 1,0 1,5 2,0 2,5 3, ln(OR) stomach vs. back Observational II-24

Meta-regression: depression and incidence of diabetes No association between duration of follow-up and effect estimate Observational II-25

Meta-regression (fictive data) Association depression and incidence diabetes 6 (fictive) cohort studies 3 studies adjust for overweight (BMI > 25), 3 do not adjust for BMI Enter for each study: –ln (RR) –weighting factor: 1 / SE 2 –confounder adjustment (yes = 1, no = 0) Observational II-26

Meta-regression - example Regression equation: ln(RR) = b 0 + b 1 * adjconf Results regression analysis (output) b 0 = 0.750, b 1 = Pooled ln(RR) for studies without adjustment for overweight (adjconf = 0) ln(RR) = (-0.250*0) = => RR = e = 2.11 Pooled ln(RR) for studies with adjustment for overweight (adjconf = 1) ln(RR) = (-0.250*1) = => RR = e = 1.65 Observational II-27

Bottlenecks meta-regression I Limited number of variables that can be studied at once: usually limited number of studies Necessary data are not always available Ecological fallacy (aggregation bias): Associations are analyzed at aggregated level: do not necessarily reflect the true association within studies Observational II-29

Ecological fallacy Effect of age within studies, not found when analyzed at aggregated level See: Thompson SG & Higgins JPT. Stat Med 2002;21: No effect of age within studies, effect found when analyzed at aggregated level Observational II-30

Bottlenecks meta-regression II For subgroup analyses and meta-regression: studying more variables increases risk of type I error (false-positive results) Restrict the number of subgroups / variables: –Hypothesis testing: a priori analyses: describe in methods –Hypothesis generating: post hoc analyses: discuss in discussion Observational II-31

Individual patient data (IPD) meta-analysis The best solution? –Request all original data –Clean data and recode if necessary –Meta-analysis and subgroup analysis / meta-regression –Very powerful –Cheaper than new trials But: –Very labor intensive –Data sometimes (often?) no longer available Observational II-33

In sum Meta-analysis of observational studies –Large heterogeneity –Prework necessary Statistical analysis –Statistical pooling: fixed/random effects model –Analysis of sources of heterogeneity (subgroup analysis, meta-regression) –IPD meta-analysis Keep in mind limitations of subgroup analysis / meta-regression! Observational II-34

Meta-analysis of observational studies THE END Observational II-35