LECTURE 8 HYPOTHESIS TESTING AND STATISTICAL SIGNIFICANCE MSc (Addictions) Addictions Department

Questions we are trying to answer: 1. What does the difference between the group means in our sample tell us about the difference between the group means in the population? A confidence interval provides a range of likely values for the difference. 2. Do the data provide evidence that the exposure affects the outcome or could the observed difference be due to chance? A p-value from a significance (or hypothesis) test provides the probability of observing the difference due to chance alone.

Tools to answer these questions Confidence interval 95% C.I. = estimate – (1.96 × S.E. ) to estimate + (1.96 × S.E. ) Test statistic  p-value Test statistic = estimate / S.E. estimate = mean, difference between means, any other measures of exposure effect S.E. = standard error of the estimate; inversely related to sample size so larger sample = smaller confidence interval

Hypothesis testing A hypothesis is a statement that we want to explore about our population. It is a statement that can be tested. “Everyone who lives to age 90 or more is a non-smoker” To prove the hypothesis: Find everyone aged ≥90 and check they are all non-smokers To disprove the hypothesis: Find just one person aged ≥90 who is a smoker Generally easier to find evidence against a hypothesis than to prove that it is correct

The null versus the alternative Null hypothesis = statement of no effect/association No difference between groups; no association between variables P-values quantify strength of evidence against null hypothesis (smaller p-value = stronger evidence) Example: 12-month abstinence rates are the same in individuals treated with nicotine replacement therapy compared to cognitive behavioural therapy Alternative hypothesis = statement of effect/association The effect/association we aim to identify Example: 12-month abstinence rates are different in those treated with NRT compared to CBT

One-tailed versus two-tailed tests2.5%2.5%±1.965% -1.64

Example of one- versus two-tailed p-values Relationship between smoking and lung function Investigate whether smoking affects lung function, as measured by forced vital capacity (FVC) in 100 men Mean difference = SE of difference in mean FVC = √( ) = Test statistic for mean difference = -0.22/0.125 = GroupNumberMean FVCStandard deviation SE of mean FVC Smokers /√36 = Non-smokers /√64 = 0.075

Example of one- versus two-tailed p-values2.5%2.5% 5% -1.76

Should I use a one-tailed or two-tailed test? Wrong answer: “Use the one that gives you the most significant result!” Right answer: “Use the one that reflects your alternative hypothesis.” Probability that result due to chance usually based on distance from the null hypothesis not direction From our previous example: FVC could have been greater in smokers than non-smokers Using a one-tailed test requires careful specification of the alternative hypothesis

Interpretation of p-values Weak evidence against the null hypothesis Increasing evidence against the null hypothesis with decreasing P-value Strong evidence against the null hypothesis Weak evidence against the null hypothesis Increasing evidence against the null hypothesis with decreasing P-value Strong evidence against the null hypothesis P-value

Interpretation of p-values The smaller the p-value, the lower the chance of a difference as large as that observed if the null hypothesis is true The “0.05 threshold” is arbitrary Three common (and serious) mistakes in interpretation: 1.Potentially important results from small studies ignored because p > All findings with p < 0.05 assumed real By definition, 1 in 20 tests in which null hypothesis is true will produce p < All findings with p-value < 0.05 assumed relevant, even if due to large sample size

Making inferences with p-values and CIs Reducing alcohol consumption Investigate different interventions for reducing alcohol consumption, as measured by units per month Results from five controlled trials of three treatments Assume mean reduction of 40 units/month substantially improves health outcomes; reduction of 20 units/month results in moderate improvements TrialTreatmentCostNumber/groupMean units/month (treatment) Mean units/month (control) Reduction (units/month) 1ACheap ACheap BCheap BCheap CExpensive

Making inferences with p-values and CIs TrialTreatmentCostNumber/groupDifference in units/month SE of difference 95% CI for difference P-value 1ACheap to ACheap to -32.2< BCheap to BCheap to CExpensive to Questions… 1.Do you think your treatment has an impact on alcohol consumption? 2.What is the range of possible values for the effect of your treatment? 3.Do you think this is a treatment that should be implemented in clinical practice?

ONLINE RESOURCES NORMAL DISTRIBUTION inferential/normal_distribution/v/introduction-to-the-normal-distribution SAMPLING DISTRIBUTION inferential/sampling_distribution/v/central-limit-theorem CONFIDENCE INTERVALS inferential/confidence-intervals/v/confidence-interval-1 HYPOTHESIS TESTING inferential/hypothesis-testing/v/hypothesis-testing-and-p-values inferential/hypothesis-testing-two-samples/v/variance-of-differences-of- random-variables