Quasi-Experimental Designs 101: What Works? The Need To Know Team January 31 – February 1, 2005 Patricia J. Martens PhD

Outline Reviewing X’s and O’s Quasi-experimental time series designs with comparison groups The Population Health Research Data Repository: what data do we have? Brainstorming ideas

Key features of study designs Artificial manipulation? (experimental or observational) Experimental: Are the groups randomly assigned to receive or not receive the intervention? (randomized controlled trial) Are the groups selected to be as similar as possible, not randomly? (quasi-experimental comparison groups)

Research Design Schema Research Designs Descriptive Analytical Experimental Randomly selected Non-random (quasi- experimental) Observational Cross- Sectional Longitudinal Case-ControlCohort Prospective Historical Prospective (Retrospective)

Key Features of Study Designs Observational: – Information collected concurrently or over a time period? (cross-sectional or longitudinal) – If over a time period, i.e. longitudinal, do you go from exposure to disease (cohort) or from disease back in time to examine exposures (case- control)? – Do you start now and go forward (prospective), or do you have a “cohort” somewhere in the past and you follow them forward (historical prospective)?

Research Design Schema Research Designs Descriptive Analytical Experimental Randomly selected Non-random (quasi- experimental) Observational Cross- Sectional Longitudinal Case-ControlCohort Prospective Historical Prospective (Retrospective)

Study design: observational Cross-sectional studies – studying all factors at once - both the hypothesized explanatory and outcome variables Prospective studies – going forward in time, following a cohort and observing the effect of exposure to a future outcome Case-control studies – going backwards in time from the cases/controls to look at differential exposures

Research Design Schema Research Designs Descriptive Analytical Experimental Randomly selected Non-random (quasi- experimental) Observational Cross- Sectional Longitudinal Case-ControlCohort Prospective Historical Prospective (Retrospective)

Study design: “What Works” proposal Randomized Controlled (Clinical) Trial – designing a specific intervention and randomly assigning people to receive it or not to receive it Quasi-experimental – using a comparison group which is not randomly assigned – Each RHA is a comparison group – A quasi-experimental time series with many comparison groups (all other RHAs in the province) Diagrammed and described by Campbell & Stanley (1963)

X is an intervention O is an outcome measure X O Let’s play X’s and O’s

O X O Let’s play X’s and O’s

O X O O Let’s play X’s and O’s

R means randomly assigned RO X O R O O (pretest-posttest control group design) Let’s play X’s and O’s

_ _ _ _ means not randomly assigned (quasi-experimental comparison) O X O O Let’s play X’s and O’s

O X O O quasi-experimental pretest- posttest design (non-randomized control group) (non-equivalent pretest-posttest comparison group design) Let’s play X’s and O’s

Hospital BFHI Compliance Scores Time (8 month interval) BFHI Compliance site Arborg Pine Falls Ten Steps and WHO Code each assigned 4 points, for total compliance of 44 control intervention Split-unit anova: p= Martens 2001 Examples of a quasi-experimental pretest- posttest comparison group study to determine effectiveness of hospital policy/education program

O O X O O Time series (quasi experiments) Let’s play X’s and O’s

Breastfeeding Initiation year proportion initiating breastfeeding 1994 Breastfeeding study: pregnant women interviewed ? ? ? Video and breastfeedng booklet completed, used in individual prenatal instruction by CHN CHN at conference, uses new techniques to address prenatal feeding intent ? ? CHN hired PC Training begun ? * p<0.05, one-tailed, adjusted for birth weight and parity * Martens 2002 Example of a quasi-experimental time series to determine effectiveness of a community-based breastfeeding strategy

Time series (quasi experiment with comparison group) O O X O O O O Let’s play X’s and O’s

From CIHR proposal submission September 2004 Example of a quasi-experimental time series with comparison groups to determine effectiveness of a regional teen pregnancy reduction program

Additions of small amounts of phosphorus to one section of ELA Lake 226 caused surface blooms of blue-green algae, and vividly demonstrated the importance of phosphate as a cause of excessive algal growth or eutrophication. This experiment spurred legislation controlling the input of phosphorus to many water bodies. A demonstration of the work of Dr. David Schindler and the Experimental Lakes project in NW Ontario

Study design: Low internal validity Anecdote/case study Pre-experimental just doing a pretest and posttest on one group and seeing its effect Cross-sectional a snapshot in time: can’t tell which comes first, but only that they are “associated”

Study design: medium internal validity Time series; Time series with qualitative layer – looking over time to see change, with information about when interventions occurred in the time frame Case-control – going backwards in time from the cases/controls to look at different exposures to possible risk factors Observational (prospective) – going forward in time, observing the effect of exposure on a cohort to a future outcome

Study design: high internal validity Randomized Controlled (clinical) Trials, RCT designing a specific intervention and randomly assigning people to receive it or not to receive it following people to observe the outcome of interest Quasi-experimental comparison group studies using a comparison group which is not randomly assigned, but very similar at onset

Internal validity Low High Cross-sectional Pre-experimental Anecdote/case study Time series with comparison Observational (prospective) Case-control Time series with qualitative layer Randomized Controlled Trials RCT Quasi-experimental comparison group studies

“There is nothing so useless as doing efficiently that which should not be done in the first place.” Peter Drucker

MCHP’s … “paperclips” “Population Health Research Data Repository” Population- Based Health Registry Hospital Home CarePharmaceuticals Cost Vital Statistics Provider Nursing Home Medical Family Services Education Immunization National surveys Census Data EA/DA level

Brainstorming: “What Works” proposal Pick (a) a policy; and (b) a program – Think of something that your region has done in the past, somewhere between 1997 and the present (hopefully, with a few years of data AFTER the onset of this) What OUTCOME measures would you think this would impact? – Think of what you would expect to see if this intervention was “working” – Are there specific target groups to which this intervention applies? (e.g. teens, people living in a certain district of your region?) – What measures of this intervention would be available through the Repository data? Brainstorm and report! (see sheet for recording)

Policy or Program Outcome Measure(s) Target Group Outcome available in Repository ? Other comments Teen pregnancy reduction Teen pregnancy rate year olds? Certain district? pregnancies or live births? Maybe birth control pill use in Rx data?