Tests for Binary/Categorical outcomes

Binary or categorical outcomes (proportions) Outcome Variable Are the observations correlated?Alternative to the chi- square test if sparse cells: independentcorrelated Binary or categorical (e.g. fracture, yes/no) Chi-square test: compares proportions between more than two groups Relative risks: odds ratios or risk ratios Logistic regression: multivariate technique used when outcome is binary; gives multivariate-adjusted odds ratios McNemar’s chi-square test: compares binary outcome between correlated groups (e.g., before and after) Conditional logistic regression: multivariate regression technique for a binary outcome when groups are correlated (e.g., matched data) GEE modeling: multivariate regression technique for a binary outcome when groups are correlated (e.g., repeated measures) Fisher’s exact test: compares proportions between independent groups when there are sparse data (some cells <5). McNemar’s exact test: compares proportions between correlated groups when there are sparse data (some cells <5).

Binary or categorical outcomes (proportions) Outcome Variable Are the observations correlated?Alternative to the chi- square test if sparse cells: independentcorrelated Binary or categorical (e.g. fracture, yes/no) Chi-square test: compares proportions between more than two groups Relative risks: odds ratios or risk ratios (for 2x2 tables) Logistic regression: multivariate technique used when outcome is binary; gives multivariate-adjusted odds ratios McNemar’s chi-square test: compares binary outcome between correlated groups (e.g., before and after) Conditional logistic regression: multivariate regression technique for a binary outcome when groups are correlated (e.g., matched data) GEE modeling: multivariate regression technique for a binary outcome when groups are correlated (e.g., repeated measures) Fisher’s exact test: compares proportions between independent groups when there are sparse data (some cells <5). McNemar’s exact test: compares proportions between correlated groups when there are sparse data (some cells <5).

Chi-square test Probiotics groupPlacebo groupp-valueAdjusted OR(95% CI)p-value Cumulative incidence at 12 months 12/33 (36.4%)22/35 (62.9%)0.029*0.243(0.075–0.792)0.019† *Significant difference between the groups as determined by Pearson's chi-square test. †p value was calculated by multivariable logistic regression analysis adjusted for the antibiotics use, total duration of breastfeeding, and delivery by cesarean section. Kim et al. Effect of probiotic mix (Bifidobacterium bifidum, Bifidobacterium lactis, Lactobacillus acidophilus) in the primary prevention of eczema: a double-blind, randomized, placebo-controlled trial. Pediatric Allergy and Immunology. Published online October 2009.Pediatric Allergy and Immunology Table 3. Cumulative incidence of eczema at 12 months of age From an RCT of probiotic supplementation during pregnancy to prevent eczema in the infant:

Chi-square test Statistical question: Does the proportion of infants with eczema differ in the treatment and control groups? What is the outcome variable? Eczema in the first year of life (yes/no) What type of variable is it? Binary Are the observations correlated? No Are groups being compared and, if so, how many? Yes, two groups Are any of the counts smaller than 5? No, smallest is 12 (probiotics group with eczema)  chi-square test or relative risks, or both

Chi-square test of Independence Chi-square test allows you to compare proportions between 2 or more groups (ANOVA for means; chi-square for proportions).

Example 2 Asch, S.E. (1955). Opinions and social pressure. Scientific American, 193,

The Experiment A Subject volunteers to participate in a “visual perception study.” Everyone else in the room is actually a conspirator in the study (unbeknownst to the Subject). The “experimenter” reveals a pair of cards…

The Task Cards Standard lineComparison lines A, B, and C

The Experiment Everyone goes around the room and says which comparison line (A, B, or C) is correct; the true Subject always answers last – after hearing all the others’ answers. The first few times, the 7 “conspirators” give the correct answer. Then, they start purposely giving the (obviously) wrong answer. 75% of Subjects tested went along with the group’s consensus at least once.

Further Results In a further experiment, group size (number of conspirators) was altered from Does the group size alter the proportion of subjects who conform?

The Chi-Square test Conformed? Number of group members? Yes No Apparently, conformity less likely when less or more group members…

= 235 conformed out of 500 experiments. Overall likelihood of conforming = 235/500 =.47

Expected frequencies if no association between group size and conformity… Conformed? Number of group members? Yes 47 No53

Do observed and expected differ more than expected due to chance?

Chi-Square test Degrees of freedom = (rows-1)*(columns-1)=(2-1)*(5-1)=4

Chi-Square test Rule of thumb: if the chi-square statistic is much greater than it’s degrees of freedom, indicates statistical significance. Here 85>>4. Degrees of freedom = (rows-1)*(columns-1)=(2-1)*(5-1)=4

Interpretation Group size and conformity are not independent, for at least some categories of group size The proportion who conform differs between at least two categories of group size Global test (like ANOVA)  doesn’t tell you which categories of group size differ

Caveat **When the sample size is very small in any cell (<5), Fisher’s exact test is used as an alternative to the chi-square test.

Review Question 1 I divide my study population into smokers, ex-smokers, and never-smokers; I want to compare years of schooling (a normally distributed variable) between the three groups. What test should I use? a.Repeated-measures ANOVA. b.One-way ANOVA. c.Difference in proportions test. d.Paired ttest. e.Chi-square test.

Review Question 1 I divide my study population into smokers, ex-smokers, and never-smokers; I want to compare years of schooling (a normally distributed variable) between the three groups. What test should I use? a.Repeated-measures ANOVA. b.One-way ANOVA. c.Difference in proportions test. d.Paired ttest. e.Chi-square test.

Review Question 2 I divide my study population into smokers, ex-smokers, and never-smokers; I want to compare the proportions of each group that went to graduate school. What test should I use? a.Repeated-measures ANOVA. b.One-way ANOVA. c.Difference in proportions test. d.Paired ttest. e.Chi-square test.

Review Question 2 I divide my study population into smokers, ex-smokers, and never-smokers; I want to compare the proportions of each group that went to graduate school. What test should I use? a.Repeated-measures ANOVA. b.One-way ANOVA. c.Difference in proportions test. d.Paired ttest. e.Chi-square test.

Review Question 2 I divide my study population into smokers, ex-smokers, and never-smokers; I want to compare the proportions of each group that went to graduate school. What test should I use? a.Repeated-measures ANOVA. b.One-way ANOVA. c.Difference in proportions test. d.Paired ttest. e.Chi-square test.

Binary or categorical outcomes (proportions) Outcome Variable Are the observations correlated?Alternative to the chi- square test if sparse cells: independentcorrelated Binary or categorical (e.g. fracture, yes/no) Chi-square test: compares proportions between more than two groups Relative risks: odds ratios or risk ratios (for 2x2 tables) Logistic regression: multivariate technique used when outcome is binary; gives multivariate-adjusted odds ratios McNemar’s chi-square test: compares binary outcome between correlated groups (e.g., before and after) Conditional logistic regression: multivariate regression technique for a binary outcome when groups are correlated (e.g., matched data) GEE modeling: multivariate regression technique for a binary outcome when groups are correlated (e.g., repeated measures) Fisher’s exact test: compares proportions between independent groups when there are sparse data (some cells <5). McNemar’s exact test: compares proportions between correlated groups when there are sparse data (some cells <5).

Risk ratios and odds ratios Probiotics groupPlacebo groupp-valueAdjusted OR(95% CI)p-value Cumulative incidence at 12 months 12/33 (36.4%)22/35 (62.9%)0.029*0.243(0.075–0.792)0.019† *Significant difference between the groups as determined by Pearson's chi-square test. †p value was calculated by multivariable logistic regression analysis adjusted for the antibiotics use, total duration of breastfeeding, and delivery by cesarean section. Kim et al. Effect of probiotic mix (Bifidobacterium bifidum, Bifidobacterium lactis, Lactobacillus acidophilus) in the primary prevention of eczema: a double-blind, randomized, placebo-controlled trial. Pediatric Allergy and Immunology. Published online October 2009.Pediatric Allergy and Immunology Table 3. Cumulative incidence of eczema at 12 months of age From an RCT of probiotic supplementation during pregnancy to prevent eczema in the infant:

Corresponding 2x2 table TreatmentPlacebo Treatment Group Eczema

Risk ratios and odds ratios Statistical question: Does the proportion of infants with eczema differ in the treatment and control groups? What is the outcome variable? Eczema in the first year of life (yes/no) What type of variable is it? Binary Are the observations correlated? No Are groups being compared and, if so, how many? Yes, binary Are any of the counts smaller than 5? No, smallest is 12 (probiotics group with eczema)  chi-square test or relative risks, or both

Odds vs. Risk (=probability) If the risk is…Then the odds are… ½ (50%) ¾ (75%) 1/10 (10%) 1/100 (1%) Note: An odds is always higher than its corresponding probability, unless the probability is 100%. 1:1 3:1 1:9 1:99

Risk ratios and odds ratios Absolute risk difference in eczema between treatment and placebo: 36.4%-62.9%=-26.5% (p=.029, chi- square test). Risk ratio: Corresponding odds ratio:  There is a 26.5% decrease in absolute risk, a 42% decrease in relative risk, and a 66% decrease in relative odds.

Why do we ever use an odds ratio?? We cannot calculate a risk ratio from a case- control study (since we cannot calculate the risk of developing the disease in either exposure group). The multivariate regression model for binary outcomes (logistic regression) gives odds ratios, not risk ratios. The odds ratio is a good approximation of the risk ratio when the disease/outcome is rare (~<10% of the control group)

Interpretation of the odds ratio:  The odds ratio will always be bigger than the corresponding risk ratio if RR >1 and smaller if RR <1 (the harmful or protective effect always appears larger) The magnitude of the inflation depends on the prevalence of the disease.

The rare disease assumption 1 1 When a disease is rare: P(~D) = 1 - P(D)  1

The odds ratio vs. the risk ratio 1.0 (null) Odds ratio Risk ratio Odds ratio Risk ratio Odds ratio Rare Outcome Common Outcome 1.0 (null)

When is the OR is a good approximation of the RR? General Rule of Thumb: “OR is a good approximation as long as the probability of the outcome in the unexposed is less than 10%”

Binary or categorical outcomes (proportions) Outcome Variable Are the observations correlated?Alternative to the chi- square test if sparse cells: independentcorrelated Binary or categorical (e.g. patency, revision) Chi-square test: compares proportions between more than two groups Relative risks: odds ratios or risk ratios (for 2x2 tables) Logistic regression: multivariate technique used when outcome is binary; gives multivariate-adjusted odds ratios McNemar’s chi-square test: compares binary outcome between correlated groups (e.g., before and after) Conditional logistic regression: multivariate regression technique for a binary outcome when groups are correlated (e.g., matched data) GEE modeling: multivariate regression technique for a binary outcome when groups are correlated (e.g., repeated measures) Fisher’s exact test: compares proportions between independent groups when there are sparse data (some cells <5). McNemar’s exact test: compares proportions between correlated groups when there are sparse data (some cells <5).

Recall… Split-face trial: Researchers assigned 56 subjects to apply SPF 85 sunscreen to one side of their faces and SPF 50 to the other prior to engaging in 5 hours of outdoor sports during mid- day. Sides of the face were randomly assigned; subjects were blinded to SPF strength. Outcome: sunburn Russak JE et al. JAAD 2010; 62:

Results: Table I -- Dermatologist grading of sunburn after an average of 5 hours of skiing/snowboarding (P =.03; Fisher’s exact test) Sun protection factorSunburnedNot sunburned The authors use Fisher’s exact test to compare 1/56 versus 8/56. But this counts individuals twice and ignores the correlations in the data!

McNemar’s test Statistical question: Is SPF 85 more effective than SPF 50 at preventing sunburn? What is the outcome variable? Sunburn on half a face (yes/no) What type of variable is it? Binary Are the observations correlated? Yes, split-face trial Are groups being compared and, if so, how many? Yes, two groups (SPF 85 and SPF 50) Are any of the counts smaller than 5? Yes, smallest is 0  McNemar’s test exact test (if bigger numbers, would use McNemar’s chi-square test)

Correct analysis of data… Table 1. Correct presentation of the data from: Russak JE et al. JAAD 2010; 62: (P =.016; McNemar’s test). SPF-50 side SPF-85 sideSunburnedNot sunburned Sunburned10 Not sunburned748 Only the 7 discordant pairs provide useful information for the analysis!

McNemar’s exact test… There are 7 discordant pairs; under the null hypothesis of no difference between sunscreens, the chance that the sunburn appears on the SPF 85 side is 50%. In other words, we have a binomial distribution with N=7 and p=.5. What’s the probability of getting X=0 from a binomial of N=7, p=.5? Probability = Two-sided probability =

McNemar’s chi-square test Basically the same as McNemar’s exact test but approximates the binomial distribution with a normal distribution (works well as long as sample sizes in each cell >=5)

Binary or categorical outcomes (proportions) Outcome Variable Are the observations correlated?Alternative to the chi- square test if sparse cells: independentcorrelated Binary or categorical (e.g. patency, revision) Chi-square test: compares proportions between more than two groups Relative risks: odds ratios or risk ratios (for 2x2 tables) Logistic regression: multivariate technique used when outcome is binary; gives multivariate-adjusted odds ratios McNemar’s test: compares binary outcome between correlated groups (e.g., before and after) Conditional logistic regression: multivariate regression technique for a binary outcome when groups are correlated (e.g., matched data) GEE modeling: multivariate regression technique for a binary outcome when groups are correlated (e.g., repeated measures) Fisher’s exact test: compares proportions between independent groups when there are sparse data (some cells <5). McNemar’s exact test: compares proportions between correlated groups when there are sparse data (some cells <5).

Recall: Political party and drinking… Drinking by political affiliation

Recall: Political party and alcohol… This association could be analyzed by a ttest or a linear regression or also by logistic regression: Republican (yes/no) becomes the binary outcome. Alcohol (continuous) becomes the predictor.

Logistic regression Statistical question: Does alcohol drinking predict political party? What is the outcome variable? Political party What type of variable is it? Binary Are the observations correlated? No Are groups being compared? No, our independent variable is continuous  logistic regression

The logistic model… ln(p/1- p) =  +  1 *X Logit function =log odds of the outcome

The Logit Model (multivariate) Logit function (log odds) Baseline odds Linear function of risk factors for individual i:  1 x 1 +  2 x 2 +  3 x 3 +  4 x 4 …

Review question 7 If X=.50, what is the logit (=log odds) of X? a..50 b. 0 c. 1.0 d. 2.0 e. -.50

Review question 7 If X=.50, what is the logit (=log odds) of X? a..50 b. 0 c. 1.0 d. 2.0 e. -.50

Example: political party and drinking… Model: Log odds of being a Republican (outcome)= Intercept+ Weekly drinks (predictor) Fit the data in logistic regression using a computer…

Fitted logistic model: “Log Odds” of being a Republican = * (d/wk) Slope for drinking can be directly translated into an odds ratio: Interpretation: every 1 drink more per week decreases your odds of being a Republican by 75% ( 95% CI is to 1.325; p=.10)

To get back to OR’s…

“Adjusted” Odds Ratio Interpretation

Adjusted odds ratio, continuous predictor

Practical Interpretation The odds of disease increase multiplicatively by e ß for for every one-unit increase in the exposure, controlling for other variables in the model.

Multivariate logistic regression Litvick JR et al. Predictors of Olfactory Dysfunction in Patients With Chronic Rhinosinusitis. The Laryngoscope Dec 2008; 118: pp

Logistic regression Statistical question: What factors are associated with anosmia (and hyposmia)? What are the outcome variables? anosmia vs. normal olfaction (and hyosmia vs. normal) What type of variable is it? Binary Are the observations correlated? No Are groups being compared? We want to examine multiple predictors at once, so we need multivariate regression.  multivariate logistic regression

Multivariate logistic regression Litvick JR et al. Predictors of Olfactory Dysfunction in Patients With Chronic Rhinosinusitis. The Laryngoscope Dec 2008; 118: pp Interpretation: being a smoker increases your odds of anosmia by 658% after adjusting for older age, nasal polyposis, asthma, inferior turbinate hypertrophy, and septal deviation.

Logistic regression in cross- sectional and cohort studies… Many cohort and cross-sectional studies report ORs rather than RRs even though the data necessary to calculate RRs are available. Why? If you have a binary outcome and want to adjust for confounders, you have to use logistic regression. Logistic regression gives adjusted odds ratios, not risk ratios. These odds ratios must be interpreted cautiously (as increased odds, not risk) when the outcome is common. When the outcome is common, authors should also report unadjusted risk ratios and/or use a simple formula to convert adjusted odds ratios back to adjusted risk ratios.

Example, wrinkle study… A cross-sectional study on risk factors for wrinkles found that heavy smoking significantly increases the risk of prominent wrinkles. Adjusted OR=3.92 (heavy smokers vs. nonsmokers) calculated from logistic regression. Interpretation: heavy smoking increases risk of prominent wrinkles nearly 4-fold?? The prevalence of prominent wrinkles in non- smokers is roughly 45%. So, it’s not possible to have a 4-fold increase in risk (=180%)! Raduan et al. J Eur Acad Dermatol Venereol Jul 3.

Interpreting ORs when the outcome is common… If the outcome has a 10% prevalence in the unexposed/reference group*, the maximum possible RR=10.0. For 20% prevalence, the maximum possible RR=5.0 For 30% prevalence, the maximum possible RR=3.3. For 40% prevalence, maximum possible RR=2.5. For 50% prevalence, maximum possible RR=2.0. *Authors should report the prevalence/risk of the outcome in the unexposed/reference group, but they often don’t. If this number is not given, you can usually estimate it from other data in the paper (or, if it’s important enough, the authors).

Interpreting ORs when the outcome is common… Formula from: Zhang J. What's the Relative Risk? A Method of Correcting the Odds Ratio in Cohort Studies of Common Outcomes JAMA. 1998;280: Where: OR = odds ratio from logistic regression (e.g., 3.92) P 0 = P(D/~E) = probability/prevalence of the outcome in the unexposed/reference group (e.g. ~45%) If data are from a cross-sectional or cohort study, then you can convert ORs (from logistic regression) back to RRs with a simple formula:

For wrinkle study… Zhang J. What's the Relative Risk? A Method of Correcting the Odds Ratio in Cohort Studies of Common Outcomes JAMA. 1998;280: So, the risk (prevalence) of wrinkles is increased by 69%, not 292%.

Sleep and hypertension study… OR hypertension = 5.12 for chronic insomniacs who sleep ≤ 5 hours per night vs. the reference (good sleep) group. OR hypertension = 3.53 for chronic insomiacs who sleep 5-6 hours per night vs. the reference group. Interpretation: risk of hypertension is increased 500% and 350% in these groups? No, ~25% of reference group has hypertension. Use formula to find corresponding RRs = 2.5, 2.2 Correct interpretation: Hypertension is increased 150% and 120% in these groups. -Sainani KL, Schmajuk G, Liu V. A Caution on Interpreting Odds Ratios. SLEEP, Vol. 32, No. 8, Vgontzas AN, Liao D, Bixler EO, Chrousos GP, Vela-Bueno A. Insomnia with objective short sleep duration is associated with a high risk for hypertension. Sleep 2009;32:491-7.

Review problem 8 In a cross-sectional study of heart disease in middle-aged men and women, 10% of men in the sample had prevalent heart disease compared with only 5% of women. After adjusting for age in multivariate logistic regression, the odds ratio for heart disease comparing males to females was 1.1 (95% confidence interval: 0.80—1.42). What conclusion can you draw? a. Being male increases your risk of heart disease. b. Age is a confounder of the relationship between gender and heart disease. c. There is a statistically significant association between gender and heart disease. d. The study had insufficient power to detect an effect.

Review problem 8 In a cross-sectional study of heart disease in middle-aged men and women, 10% of men in the sample had prevalent heart disease compared with only 5% of women. After adjusting for age in multivariate logistic regression, the odds ratio for heart disease comparing males to females was 1.1 (95% confidence interval: 0.80—1.42). What conclusion can you draw? a. Being male increases your risk of heart disease. b. Age is a confounder of the relationship between gender and heart disease. c. There is a statistically significant association between gender and heart disease. d. The study had insufficient power to detect an effect.

Review topic: Diagnostic Testing and Screening Tests

Characteristics of a diagnostic test Sensitivity= Probability that, if you truly have the disease, the diagnostic test will catch it. Specificity=Probability that, if you truly do not have the disease, the test will register negative.

Calculating sensitivity and specificity from a 2x2 table +- +ab -cd Screening Test Truly have disease Sensitivity Specificity Among those with true disease, how many test positive? Among those without the disease, how many test negative? a+b c+d

Hypothetical Example Mammography Breast cancer ( on biopsy) Sensitivity=9/10= Specificity= 881/990 =.89 1 false negatives out of 10 cases 109 false positives out of 990

Positive predictive value The probability that if you test positive for the disease, you actually have the disease. Depends on the characteristics of the test (sensitivity, specificity) and the prevalence of disease.

Calculating PPV and NPV from a 2x2 table +- +ab -cd Screening Test Truly have disease PPV NPV Among those who test positive, how many truly have the disease? Among those who test negative, how many truly do not have the disease? a+cb+d

Hypothetical Example Mammography Breast cancer ( on biopsy) PPV=9/118=7.6% Prevalence of disease = 10/1000 =1% NPV=881/882=99.9%

What if disease was twice as prevalent in the population? Mammography Breast cancer ( on biopsy) sensitivity=18/20= specificity=872/980=.89 Sensitivity and specificity are characteristics of the test, so they don’t change!

What if disease was more prevalent? PPV=18/126=14.3% Prevalence of disease = 20/1000 =2% NPV=872/874=99.8% Mammography Breast cancer ( on biopsy)

Conclusions Positive predictive value increases with increasing prevalence of disease Or if you change the diagnostic tests to improve their accuracy.

Review question 9 In a group of patients presenting to the hospital casualty department with abdominal pain, 30% of patients have acute appendicitis. 70% of patients with appendicitis have a temperature greater than 37.5ºC; 40% of patients without appendicitis have a temperature greater than 37.5ºC. a. The sensitivity of temperature greater than 37.5ºC as a marker for appendicitis is 21/49. b. The specificity of temperature greater than 37.5ºC as a marker for appendicitis is 42/70. c. The positive predictive value of temperature greater than 37.5ºC as a marker for appendicitis is 21/30. d. The predictive value of the test will be the same in a different population. e. The specificity of the test will depend upon the prevalence of appendicitis in the population to which it is applied.

Review question 9 In a group of patients presenting to the hospital casualty department with abdominal pain, 30% of patients have acute appendicitis. 70% of patients with appendicitis have a temperature greater than 37.5ºC; 40% of patients without appendicitis have a temperature greater than 37.5ºC. a. The sensitivity of temperature greater than 37.5ºC as a marker for appendicitis is 21/49. b. The specificity of temperature greater than 37.5ºC as a marker for appendicitis is 42/70. c. The positive predictive value of temperature greater than 37.5ºC as a marker for appendicitis is 21/30. d. The predictive value of the test will be the same in a different population. e. The specificity of the test will depend upon the prevalence of appendicitis in the population to which it is applied.