1 Lecture 20: Non-experimental studies of interventions Describe the levels of evaluation (structure, process, outcome) and give examples of measures of each level Describe the applications of cohort and case-control designs to the evaluation of interventions. Describe advantages and disadvantages of randomization versus: - Historical controls - Simultaneous, non-randomized controls Describe the following quasi-experimental designs: - Time series (trend) design - Non-equivalent control group design

2 Design of an intervention study Study objectives: –Define intervention –Define target population –Define evaluation measures Study design: –Experimental –Non-experimental

3 Levels of evaluation STRUCTURE: –Drugs, devices, staff, equipment needed to provide intervention PROCESS: –Interaction between structure and patient/client –Adherence/compliance OUTCOMES: –Expected or unexpected results, positive or negative, e.g.: Death, disease, disability Attitudes, behaviors Costs

4 Levels of evaluation Create hypothetical diagram linking structure, process, and outcome Based on goals of study, select measures of structure, process, and/or outcome

5 Levels of evaluation: example Hypothetical diagram: –HIV/AIDS educational intervention for drug injectors (describe planned structure)  Process (attendance/quality of participation)  Outcome 1: Improved knowledge/attitudes  Outcome 2: Lower risk behaviour  Outcome 3: Lower HIV incidence rate

6 Levels of evaluation Example: –Exercise program to reduce CHD risk STRUCTURE? PROCESS? OUTCOMES?

7 Epidemiological observational study designs Cohort and case-control studies Independent and dependent variables: Studies of risk factors: – independent variable (exposure): risk factor –dependent variable: disease Studies of interventions: –independent variable (exposure): intervention –dependent variable(s): selected “outcomes” (could be measures of process and/or outcomes)

8 Cohort study Study population: –Cohorts with and without “exposure” to intervention (or different levels of exposure) –Control (unexposed) cohort - concurrent or historical confounding by changes over tine in patient population, aspects of treatment other than intervention; measurement of confounders Follow-up to measure outcomes

9 Cohort study Selection of controls: could they receive either treatment? Example: medical vs surgical treatment of CHD Some sources of bias: –Selection bias –Information bias: detection bias, other –Confounding: by indication, other

10 Examples of cohort studies Effectiveness of new cancer treatment –Historical controls Do HMOs reduce hospitalization in terminal cancer patients, during 6 months before death? –Administrative databases and tumor registry from Rochester NY –Cancer deaths in 100 pairs of HMO members and non- members –Matched by age, cancer site, months from diagnosis to death

11

12 Case-control study Study population: –Cases (with outcome) –Controls (without outcome) Limited to single, categorical outcome Data collected on prior “exposure” to intervention Some sources of bias –Selection bias –Information bias –Confounding: by indication, other

13 Case-control study: Examples Screening programs: –screening Pap test and invasive cervical cancer –screening mammography and breast cancer deaths –screening sigmoidoscopy and colon cancer deaths Vaccine effectiveness (e.g., BCG) Neonatal intensive care and neonatal deaths

14 Quasi-experimental study designs Investigator has “some control” over timing or allocation of intervention –Non-randomized or quasi-randomized trials –Non-equivalent control group designs: pre-test and post-test post-test only –Time series designs single or muliple

15 Diagramming Intervention Study (Evaluation) Designs Campbell and Stanley X = program O = measurement R = randomization

16 Randomized (Experimental) Designs Randomized pre-test post-test control group design R O 1 X O 2 R O 3 O 4 Post-test only control group design R X O 1 R O 2

17 Some Weak Observational Designs: Cross-sectional One-shot case-study X O Static group comparison: X O 1 O 3

18 Some Weak Observational Designs: Longitudinal Before-after (pre-post) study O 1 X O 2

19 Some quasi-experimental designs: with control/comparison group

20 Health insurance in Quebec 1961: universal hospital insurance – included ER care for accidents 1970: universal health insurance (Medicare) –added MD care including hospital outpatient clinics and ERs Population surveys before and after Effects on: –use of physician services by general population –physician workload –use of emergency rooms –hospitalization and surgery

21 MD visits/person/year by income (household surveys)

22 MD visits/person/year (household surveys)

23 MD visits/person/year by income (household surveys)

24 % adults with cough 2+ weeks who consulted MD (household surveys)

25 % children (<17) with tonsilitis or sore throat and fever who consulted MD (household surveys)

26 % pregnancies with visit in first trimester (household survey)

27 % Tried to contact MD before ED visit; of these, % successful (6 hospital sample)

28 Examples of pre-post non- equivalent control group design Stanford 5-city study of CHD prevention Intervention included mass media education and group interventions for high-risk 5 cities selected - similar characteristics –those with shared media market were allocated to intervention –isolated cities allocated to control group

29 Time series designs

30 Example of time series study: Tamblyn et al, 2001 Evaluation of prescription drug cost-sharing among poor and elderly Methods: –Trend study: Multiple pre- and post- measurements –Cohort study:

31 Source: Tamblyn et al, JAMA 2001, 285(4):

32 Source: Tamblyn et al, JAMA 2001, 285(4):

33 Time-series design: Home care in terminal cancer Evaluation of home-hospice programme in Rochester, NY Expansion of home-care benefits in 1978 Hypothesis: home-hospice care in last month of life reduces hospital days and costs Data sources: Linkage of tumor registry and health insurance claims databases

34

35

36 Differences between quasi-experimental and epidemiological cohort study designs Quasi-experimental designs often use ecological rather than individual level of measurement Serial cross-sectional studies over time vs follow- up of individuals: –advantages and disadvantages?