1. Methodological quality assessment of observational studies Nicole Vogelzangs Department of Psychiatry & EMGO + institute

2. Outline Observational study designs Methodological quality assessment: –Bias and confounding –Important aspects of quality assessment –Using methodological quality in meta-analysis Causality Observational I-2

3. Research designs Etiology –Onset of symptoms or disease –Risk factors Diagnosis –Assessment of (severity) of symptoms or disease Prognosis –Course of symptoms or disease –Prognostic factors (includes treatment!) Observational I-3

4. Reasons for observational design Study of the natural course of disease Follow-up of large groups of persons Long follow-up necessary Randomization not ethical Randomization not possible –e.g. work related exposure Practical problems (efficiency, costs) Rare diseases or events –e.g. side effects Observational I-4

5. Design (general) Examples: I.Does use of Cox-2 inhibitors increase the risk of myocardial infarction as compared with non-selective anti-inflammatory drugs (NSAIDs)? II.Is there an association between stressful events during childhood and the onset of chronic pain? III.Does the risk of a chronic course of depression increase because of the presence of somatic diseases? Confounders DeterminantOutcome Observational I-5

6. Cross-sectional research All determinants and the outcome are simultaneously assessed Determinant Outcome Examples: I.Patients with joint pain in 50 general practices who have used Cox-2 inhibitors or NSAIDs in the past 5 years? Medical file research: yes/no infarction? II.Assessment of stressful events in the past and chronic pain complaints III.Assessment of depressive symptoms in the past 2 years and the presence of somatic diseases Observational I-6

7. Cohort study Select cohort Assess exposure to determinant Follow cohort in time Identify disease cases Determinant Outcome Example I: -Patients with joint pain from 50 GPs, who start medication -(Repeated) assessment of type of painkiller -Register myocardial infarction cases during 5 years -Compare incidence infarction with Cox-2 inhibitors vs. NSAIDs Observational I-7

8. Patient - control design Select disease cases Select a (healthy) control group Assess exposure to determinant (in the past) Determinant Outcome Example I: - All (new) infarctions during 1 year in 10 centers - Select controls (e.g. through general practitioners) - Compare type of painkiller between patients and controls (OR) Observational I-8

9. Outline Observational study designs Methodological quality assessment: –Bias and confounding –Important aspects of quality assessment –Using methodological quality in meta-analysis Causality Observational I-9

10. Methodological quality assessment Diversity in design: no standard checklist Existing checklists differ Adjust to topic of review Assess presence and degree of possible bias and confounding –Selection bias –Information bias –Confounding Observational I-10

11. Selection bias Inadequate selection of participants Chance to be selected depends on outcome (example: safety belt) Association between determinant and outcome is different in the study population compared with the (theoretical) source population Observational I-11

12. Assessment of risk of selection bias Does the study population correctly reflect the source population? –Clear description setting, selection procedure, selection criteria –PC: Do controls and patients stem from the same source population? –High response? Cohort: are persons ‘at risk’ being selected? Observational I-12

13. Information bias Incomparable information in patients and controls: –of the determinant –of the disease Measurements are not assessed in the same way Measurements are influenced by knowledge on determinant and/or disease status (e.g. childhood events in persons with chronic pain) Observational I-13

14. Assessment of risk of information bias Are determinant and disease measured in a standardized way? –Same method for all participants? –Definition of cut-off points and diagnostic criteria? Are good (=valid & reliable) measurements being used? Are determinant and disease assessed independently? –Blinding –Independent assessment Observational I-14

15. Confounding The association between determinant and disease is (partly) explained by other (non- mediating) determinants => confounders Confounder is associated with determinant and with outcome (and is not in the causal pathway) Determinant Outcome Confounder Observational I-15

16. Assessment of risk of confounding Are potential confounders measured? Does the design address confounding (e.g. restriction, matching) Are the statistical analyses well conducted? –Stratified analyses –Multivariable analyses (is the model described?) Observational I-16

17. Follow-up (cohort design) Drop-out of participants can distort results –High drop-out during follow-up? –Selective drop-out (drop-out related to exposure)? Is duration of follow-up sufficient? Observational I-17

18. Summary quality checklist Adequate selection procedure? High response? PC: patients and controls from same source population? Determinant and outcome similarly measured in all persons? Independent measurement of determinant and outcome? Limited drop-out during follow-up? Adequate duration of follow-up? Design deals with confounding? Analyses adjusted for confounding? Observational I-18

19. Using methodological quality in meta-analysis Use total score of checklist: Weighted for total score (pooling) –RR unweighted:1.38 [1.01-1.87] –RR weighted for quality:1.46 [1.29-1.64] Stratified analyses (pooling) –RR studies low quality:1.07 [0.89-1.29] –RR studies high quality:1.91 [1.56-2.35] Chlorination of drinking water and cancer Morris et al. Am J Publ Health 1992;82:955-63 Observational I-21

20. Study the influence of specific aspects –Stratified analyses (subgroup analyses) –Meta-regression analyses (addressing several aspects simultaneously) Aspects of methodological assessment, e.g. –Studies with high vs. low response –Cohort study vs. patient-control study –Studies with vs. without blinding Observational I-22 Using methodological quality in meta-analysis

21. Subgroup analyses Saturated fat intake and breast cancer 1.0 0.8 1.2 1.4 1.6 RR 12 case-control studies 6 cohort studies Intermittent sunlight and melanoma 1.0 0.5 1.5 2.0 2.5 OR 7 studies with blinding 9 studies no blinding Egger et al. BMJ 1998;316:140-4 Observational I-23

22. Outline Observational study designs Methodological quality assessment: –Bias and confounding –Important aspects of quality assessment –Using methodological quality in meta-analysis Causality Observational I-24

23. Association or causal relationship? Necessary and sufficient causes II. factor X  factor F  Disease factor Y  I. factor F  Disease III. factor P  factor Q  Disease factor R  Observational I-25

24. Criteria for causality (adapted from Hill) Timing: temporal relationship determinant - outcome 1 prospective cohort study 2 patient-control study 3 cross-sectional study Strength of the association / dose-response Consistency of study results Validity (methodological quality) Adequate consideration to possible confounding Plausible explanation for association Observational I-26

25. Levels of evidence (example) Strong: consistent associations found in ≥ 2 high quality cohorts Moderate: consistent associations found in ≥ 1 high quality cohort and ≥ 1 low quality cohort Weak: consistent associations found in ≥ 1 high quality cohort or in ≥ 3 low quality cohorts Inconclusive: association found in < 3 low quality cohorts Inconsistent: inconsistent findings irrespective of study quality Observational I-27

26. But … Associations are often weak (RR 0.5-2.0): precise, but spurious … Observational studies are in general quite heterogeneous It is impossible to fully exclude bias and confounding in observational research Possibly greater chance of publication bias Observational I-28

27. Spurious results … Observational I-29

28. MOOSE Meta-analysis Of Observational Studies in Epidemiology Guideline for reporting of systematic reviews of observational studies CONSORT (RCTs) and QUORUM (Sys reviews) Checklist for editors and authors –Hypothesis –Search strategy –Methodological assessment –Analyses –Discussion and conclusions Stroup et al. JAMA 2000; 283: 2008-12 Observational I-30

29. In sum Prospective cohort studies (in principal) are preferred when studying causal relationships Observational studies are sensitive for bias Caution with conclusions about presence or strength of causal relationship Methodological quality assessment is largely customized Observational I-31

30. Methodological quality assessment of observational studies THE END Observational I-32