Types of study designs: from descriptive studies to randomized controlled trials Kirsten Bibbins-Domingo, PhD, MD Assistant Professor of Medicine and of Epidemiology and Biostatistics University of California, San Francisco

Objectives To understand the difference between descriptive and analytic studies To identify the strengths and weakness of different designs To be able to apply different study designs to the same research question

Descriptive vs. Analytic Test Performance Height Descriptive Questions What is the average height of UCSF students? How many UCSF student are over 6 feet tall? Analytic Questions Is student height associated with test scores? Do students greater than 6 feet tall score higher on standardized tests?

Hierarchy of Study Types?? Analytic Descriptive Case report Case series Survey Observational Cross sectional Case-control Cohort studies Experimental Randomized controlled trials Strength of evidence for causality between a risk factor and outcome

Analytic Studies Attempt to establish a causal link between a predictor/risk factor and an outcome. You are doing an analytic study if you have any of the following words in your research question: causes, leads to, compared with, more likely than, associated with, related to, similar to, correlated with, greater than, less than Predictor (risk factor) Outcome (disease)

Measures of association Risk ratio (relative risk) A A + B C C + D Disease Yes No Risk Factor A B C D

Hierarchy of Study Types Analytic Descriptive Case report Case series Survey Observational Cross sectional Case-control Cohort studies Experimental Randomized controlled trials Strength of evidence for causality between a risk factor and outcome

Research Question Is the regular consumption of Red Bull associated with improved academic performance among U.S. medical students?

Rationale “functional drink” designed for periods of mental and physical exertion. performance, concentration, memory, reaction time, vigilance, and emotional balance Taurine + glucuronolactone + caffeine

Background Alford C, Cox H, Wescott R. The effects of red bull energy drink on human performance and mood. Amino Acids. 2001;21(2):139-50. Warburton DM, Bersellini E, Sweeney E. An evaluation of a caffeinated taurine drink on mood, memory and information processing in healthy volunteers without caffeine abstinence. Psychopharmacology (Berl). 2001 Nov;158(3):322-8. Seidl R, Peyrl A, Nicham R, Hauser E. A taurine and caffeine-containing drink stimulates cognitive performance and well-being. Amino Acids. 2000;19(3-4):635-42. Horne JA, Reyner LA. Beneficial effects of an "energy drink" given to sleepy drivers. Amino Acids. 2001;20(1):83-9. Kennedy DO, Scholey AB. A glucose-caffeine 'energy drink' ameliorates subjective and performancedeficits during prolonged cognitive demand. Appetite. 2004 Jun;42(3):331-3.

Great idea, but how do you get started…. Interesting, novel, and relevant, but… You only have 25,000 dollars to start investigating this question. What is feasible?

Study Design #1 Cross-sectional study of UCSF medical students taking USMLE Step 2 Questionnaire administered when registering for USMLE 2 Primary predictor: self-report of >3 cans Red Bull per week Primary outcome: Score on USMLE Step 2

Cross-sectional study: structure Predictor (risk factor) Outcome (disease) Red Bull consumption USMLE Score time

Cross-sectional Study: Pluses + Prevalence (not incidence) + Fast/Inexpensive - no waiting! + No loss to follow up + Associations can be studied Many well-known cross-sectional studies AAMC California Health Interview Survey (NHIS, CHIS) National Health and Nutrition Exam Survey (NHANES)

Cross-sectional study: minuses - Cannot determine causality Red Bull consumption USMLE Score time

Cross-sectional study: minuses - Cannot determine causality ACE inhibitor use hospitalization for heart failure Documented DNR status in-hospital mortality time

Cross-sectional study: minuses - Cannot determine causality - Cannot study rare outcomes

What if you are interested in the rare outcome? The association between regular Red Bull consumption and… A perfect score on the USMLE – Step 2 Acceptance into a highly selective residency ANSWER: A Case-Control study

Study Design #2 A case-control study Cases: 4th year med students accepted to residency in “highly selective specialty X”. Controls: 4th year med students who applied but were not accepted. Predictor: self-reported regular Red Bull consumption

Case control studies Investigator works “backward” (from outcome to predictor) Sample chosen on the basis of outcome (cases), plus comparison group (controls) Predictor (risk factor) Outcome (disease)

Case-control study structure present CASES 4th year UCSF students who matched in “highly selective specialty X” Red Bull consumption YES Red Bull consumption NO CONTROLS 4th year students who failed to match in “highly selective specialty X” time

Case control studies Cannot yield estimates of incidence or prevalence of disease in the population (why?) Odds Ratio is statistics

Measures of association Odds ratio A D C B Disease Yes No Test A B C D

Case-control Study: pluses + Rare outcome/Long latent period + Inexpensive and efficient: may be only feasible option + Establishes association (Odds ratio) + Useful for generating hypotheses (multiple risk factors can be explored)

Case-control study-minuses Causality still difficult to establish Selection bias (appropriate controls) Caffeine and Pancreatic cancer in the GI clinic Recall bias: sampling (retrospective) Abortion and risk of breast cancer in Sweden Cannot tell about incidence or prevalence

Case-control - “the house red” Rely tampons and toxic shock syndrome: High rates of toxic shock syndrome in menstruating women Suspected OCPs or meds for PMS Cases: 180 women with TSS in 6 geographic areas Controls: 180 female friends of these patients and 180 females in the same telephone code Tampon associated with TSS (OR = 29!) Super absorbency associated with TSS (OR 1.34 per gm increase in absorbency) Led to “RELY” brand tampons being taken off the market.

Where are we? Preliminary results from our cross-sectional and case-control study suggest an association between Red Bull consumption and improved academic performance among medical students What’s missing? - strengthening evidence for a causal link between Red Bull consumption and academic performance Use results from our previous studies to apply for funding for a prospective cohort study!

Study design #3 Prospective cohort study of UCSF medical students Class of 2009 All entering medical students surveyed regarding beverage consumption Outcomes: USMLE Step 1 score, USMLE Step 2 score, match in first choice residency

Elements of a cohort study Selection of sample from population Measures predictor variables in sample Follow population for period of time Measure outcome variable Famous cohort studies Framingham Nurses’ Health Study Physicians’ Health Study Olmsted County, Minnesota Predictor (risk factor) Outcome (disease)

Prospective cohort study structure The present The future Top USMLE scorers Everyone else time

Strengths of cohort studies Know that predictor variable was present before outcome variable occurred (some evidence of causality) Directly measure incidence of a disease outcome Can study multiple outcomes of a single exposure (RR is measure of association)

Weaknesses of cohort studies Expensive and inefficient for studying rare outcomes HERS vs. WHI Often need long follow-up period or a very large population CARDIA Loss to follow-up can affect validity of findings Framingham

Other types of cohort studies Retrospective cohort Identification of cohort, measurement of predictor variables, follow-up and measurement of outcomes have all occurred in the past Much less costly than prospective cohorts Investigator has minimal control over study design

Hierarchy of Study Types Analytic Descriptive Case report Case series Survey Observational Cross sectional Case-control Cohort studies Experimental Randomized controlled trials Strength of evidence for causality between a risk factor and outcome

What distinguishes observational studies from experiments? Ability to control for confounding Confounder Predictor Outcome Examples: sex (men are more likely to drink red bull and men are more likely to match in neurosurgery) Undergraduate institution (students from southern California are more likely to drink red bull and also score higher on USMLE) Height?

But we measured all of the potential confounders……. In a prospective cohort study you can (maybe) measure all potential known confounders, but… You can’t control for unanticipated or unmeasured confounders Randomization controls for unmeasured confounding

Study design # 4 Randomized controlled trial of daily Red Bull consumption among entering UCSF medical students Class 2009 Randomized to daily consumption of Red Bull vs. daily consumption of placebo Outcomes: USMLE Step 1 score, USMLE Step 2 score, match in first choice residency

Steps in a randomized controlled trial Select participants high-risk for outcome (high incidence) Likely to benefit and not be harmed Likely to adhere Measure baseline variables Randomize Eliminates baseline confounding

Steps in a randomized controlled trial Blinding the intervention Follow subjects Adherence to protocol Lost to follow up Measure outcome Clinically important measures Adverse events

Why blind?: Co interventions Unintended effective interventions participants use other therapy or change behavior study staff, medical providers, family or friends treat participants differently Nondifferential - decreases power Differential - causes bias

Why blind?: Biased Outcome Ascertainment or adjudication If group assignment is known participants may report symptoms or outcomes differently physicians or investigators may elicit symptoms or outcomes differently Study staff or adjudicators may classify similar events differently in treatment groups Problematic with “soft” outcomes investigator judgment participant reported symptoms, scales

High Quality Randomized Trials Tamper-proof randomization Blinding of participants, study staff, lab staff, outcome ascertainment and adjudication Adherence to study intervention and protocol Complete follow-up

Why not just jump to an RCT? Expensive Feasibility concerns Ethical considerations May not be the right design for your question

Hierarchy of Study Types?? A study type of every budget, purpose and research question Analytic Descriptive Case report Case series Survey Observational Cross sectional Case-control Cohort studies Experimental Randomized controlled trials Strength of evidence for causality between a risk factor and outcome