1. Hypothesis Testing

2. Testing your beliefs Ultimately in most research, the aim is to investigate whether the data supports a particular hypothesis, or whether there is evidence to reject this hypothesis. For example: How do you know that male students are more likely to weigh more than female students? How do we know that smoking is associated with increased risk of lung cancer? How do we know that possessing a particular genetic profile means someone is more likely to be obese than others with a different genetic profile?

3. 1.Data checking, identifying problems and characteristics 2.Understanding chance and uncertainty 3.How will the data for one attribute behave, in a theoretical framework? 4.Theoretical framework assumes complete information, need to address uncertainties in real data 5.Testing your beliefs, do the data support what you think is true? Data exploration and Statistical analysis

4. Data Data exploration, categorical / numerical outcomes Estimation of parameters, quantifying uncertainty Hypothesis testing Parametric tests (t-tests, ANOVA, test of proportions) Model each outcome with a theoretical distribution

5. Null hypothesis A statement of status quo, or of no changes Alternative hypothesis Hypothesis which the researcher wishes to investigate Commonly, the alternative hypothesis is first formulated, and the null hypothesis is the negation of the alternative hypothesis. Hypothesis Testing

6. A woman buys a pregnancy test kit, and is interested to find out whether she is pregnant. The null hypothesis in this case (status quo), is that she is not pregnant. The alternative hypothesis (hypothesis of interest), is that she is pregnant. Test kit may show: +ve: indicating there is evidence to suggest pregnancy –ve: indicating lack of evidence to suggest pregnancy Pregnancy Test Kit

7. The test kit may either be accurate, or inaccurate. Actually pregnant Actually not pregnant Test kit shows +ve Test kit shows –ve Correct +ve diagnosis Incorrect –ve diagnosis Incorrect +ve diagnosis Correct –ve diagnosis Pregnancy Test Kit

8. The test kit may either be accurate, or inaccurate. Pregnancy Test Kit

9. Type 1 Error (p-value) False +ve conclusion (+ve when woman is in fact not pregnant) Type 2 Error False –ve conclusion (–ve when woman is in fact pregnant) Power (Sensitivity) True +ve conclusion (+ve when woman is in fact pregnant) Specificity True –ve conclusion ( – ve when woman is in fact not pregnant) Types of Errors

10. Probability of observing a false positive result, also known as the significance of the test. If the p-value is small, we are more confident that the null hypothesis can be rejected. On average, expect 1 false positive result out of 20 results obtained. So if we perform a large study with 1 million variables, on average we expect about 50,000 variables to display p- values of < 0.05! P-values

11. Statistical tests for comparing means One of the most common statistical tests in biomedical sciences is the comparisons of averages. Let’s revisit Example 1 from the previous lecture. Example 1: The Science Faculty is interested to compare between the weights of male and female students in NUS. Recall we: - Randomly sample 200 male students and 200 female students and measure their weight. - Calculate the mean weight of these 200 male students, and use this quantity to estimate the mean weight of all the male students in NUS. - Similarly calculate the mean weight of these 200 female students and use this to estimate the mean weight of all the female students in NUS. So here we have the mean weights for male and female students. Can we compare these values, after accounting for the uncertainties in the estimation, and quantify the statistical evidence for observing a difference?

12. Test statistic A numerical quantification of the amount of evidence against the null hypothesis. Usually takes the form of: (Observed summary value – Hypothesized summary value) --- divided by --- Standard error of observed summary value Or

13. Degrees of freedom Number of independent observations that are allowed to take any values. For example: In an assessment of whether the mean weight of 120 male students in the Science faculty exceeds 70kg, we actually have 120 independent observations. However, we need to estimate the mean weight of these 120 students, thus the degrees of freedom remaining = 120 – 1 = 119. Another way to think about it: Upon collecting the weight of 120 students, IF I know the average weight, then only the weights of 119 students are allowed to ‘vary’ as the weight of the last person must be the specific value to yield the average weight that I know of. Degrees of freedom (df) Number of independent observations (usually the sample size) Number of estimated parameters =–

14. t-tests 1-Sample t-test Useful if we are keen to compare a collection of values against a hypothesized mean value. For example: Previous surveys from the 1980s found that the mean weight of male students in the Science Faculty was 63kg. The lecturer of ST1232 believes that this figure is likely to be an under-estimate for male students in 2010. Design an experiment to test this hypothesis. - Randomly sample 200 male students from the Science Faculty, and measure their weights. - Calculate the mean weight of these 200 students and the corresponding standard error of the mean, and see whether this is significantly higher than 63kg.

15. Identifying the hypotheses In hypothesis testing, it is extremely important to identify the hypotheses that are being tested, since this affects the calculation of the statistical evidence. Null hypothesis, H 0 : Mean weight,  = 63 Alternative hypothesis, H 1 : Mean weight,  > 63 But there are two other alternative hypotheses, each giving different outcome in the estimation of statistical evidence against the null hypothesis. To see whether the weight of current students differ from 63kg: H 1 : Mean weight,   63 To see whether the weight of current students is lighter than 63kg: H 1 : Mean weight,   63

16. One-sided versus two-sided tests Two-sided alternative hypothesis: Unbiased test, without assuming prior knowledge or expectation of the direction of the effect size. So the difference could be greater or smaller than the test value. Remember the test statistic  Value under the null hypothesis Value estimated from the data Numerator measures how different is the data from the null hypothesis Denominator measures the uncertainty of the estimation

17. One-sided versus two-sided tests Two-sided alternative hypothesis: Unbiased test, without assuming prior knowledge or expectation of the direction of the effect size. So the difference could be greater or smaller than the test value. Example of an alternative hypothesis: there is a difference between the height of men and women. One-sided alternative hypothesis: Biased test, where the direction of the effect, if genuine, is known. Examples: (i) men are taller than women; (ii) the weight loss pill successfully reduces weight.

18. One-sided versus two-sided tests Interpreting statistical evidence: P-values are assessed by the probability found in the shaded areas: the shaded areas represent the probability of obtaining a test statistic at least as extreme as from the observed data, under the null hypothesis. SPSS calculates the two-tailed p-value by default, need to work harder to get the one-tail p-value. Two-sided tests One-sided tests

19. One-sided versus two-sided tests Converting two-tailed p-value to one-tail p-value: - If observed effect (or test statistic) is in the same direction as the alternative hypothesis, half the p-value. - If observed effect (or test statistic) is in the opposite direction as the alternative hypothesis, one-tail p-value = 1 – 0.5  (two-tailed p-value) Two-sided tests One-sided tests

20. Example 1: A pharmaceutical company is interested in testing a new weight-loss treatment, and recruited 120 volunteers to take part in a research trial. The weight of the participants are measured before and after taking the weight- loss treatment for the prescribed duration. - Null hypothesis here is that the weight-loss treatment has no effect (or average difference in weight = 0) - Alternative hypothesis here is that the weight-loss treatment is effective (or average difference in weight < 0) Suppose the evidence obtained yields a two-sided p-value of 0.03, and the average difference (after – before) is -3.5kg, what is the correct evidence for the trial? One-tail p-value = 0.03 / 2 = 0.015 (since direction of observed effect is in the same direction as the alternative hypothesis) If however, the average difference (after – before) is 3.5kg, the one-tail p- value will instead be 1 – 0.03 / 2 = 0.985. This intuitively makes sense!

21. Back to t-tests 1-Sample t-test Useful if we are keen to compare a collection of values against a hypothesized mean value. Null hypothesis: Mean = hypothesized value ASSUMPTIONS - Data is normally distributed (theoretical assumption) - Data is symmetrically distributed (practical application) - Observations made are all independent

22. Two independent samples t-test For comparing the means between two groups. Null hypothesis: Mean of group 1 = Mean of group 2 Or effective: Difference in means = 0 Example: Comparing the weights between male and female students in Science faculty. ASSUMPTIONS - Data is normally (symmetrically) distributed within each group - Observations made within each group are all independent - Observations are also independent across the groups

23. Paired-sample t-test For comparing the difference within each pair of observations Null hypothesis: No difference between the observations in each pairing Or effective: Difference within each pairing = 0 Example: Comparing the efficacy of a diet treatment, thus comparing the weight of an individual before and after the treatment. ASSUMPTIONS - Difference within each pairing follows a Normal (symmetric) distribution - Independence between pairs of observations

24. Two independent samples vs paired-sample t-test Is there any difference between these two: Mathematically they seemed similar? Main difference in the calculations of the denominators: takes into account of the uncertainty of two sets of outcomes. first calculates the difference within each pairing, then calculate the standard error of the string of differences.

25.  2 groups Suppose we are interested in comparing the means between three groups, what can we do? (A)Perform 3 sets of two-independent samples t-tests (between groups 1 and 2; groups 2 and 3; groups 1 and 3) (B)Perform a ‘global’ test, checking whether the means are all the same. - A statistical test for comparing the means of multiple independent groups - Test the null hypothesis that the means of ALL the groups are identical Assumptions - Data within each group is normally (symmetrically) distributed - Independent observations within each group - Independent observations between the groups Analysis of Variance (ANOVA)

26. ANOVA Analysis of variance (ANOVA) – interesting name that we are effectively analysing the variance to decide whether there is any differences in the means! Null hypothesis: Means of all the groups are identical Alternative hypothesis: At least one of the groups has a different mean So, observing a significant p-value in this instance means that at least one group is different, but we don’t actually know which group that is!

27. Practical Example Previous studies suggest restriction caloric intake can increase life expectancy. Perform an experiment with mice, each randomly assigned to one of six diet treatment. Measure the time of death for each mouse (in months). Experimental Design

28. Visualising the Data

29. Practical Example Research Questions Is there any difference in life expectancy across the different diet treatments? If there is, which diet treatment contribute to this difference? Which diet treatment significantly increases life expectancy?

30. Practical Example

31. ANOVA Significant differences! But which treatment? Q: Is there any difference in life expectancy across the different diet treatments? Consider the following null hypothesis: All the mean life expectancy of the six groups are identical Alternative hypothesis: At least one group has a different mean life expectancy

32. Multiple Comparisons Can compare every possible pair of treatments. DANGER! More number of tests  more chances of making a false judgement. Remember p-value threshold of 0.05  1 out of 20 judgement may be false. There are 15 possible pairings for the 6 treatment groups  very likely to make a false judgement!

33. Bonferroni Correction Make it harder to define a result as significant. By lowering the p-value threshold. But to lower by how much? Solution Divide the threshold by the total number of tests performed. Thus instead of a critical threshold of 0.05, we now use a critical threshold of

34. Bonferroni Correction A much preferred approach is to calculate the “Bonferroni corrected p-value” instead Bonferroni-corrected p-value Multiple the obtained p-values by the number of tests performed. - Different p-value thresholds may be used, and difficult to decide what the Bonferroni-corrected thresholds are. - By calculating the Bonferroni-corrected p-value, it can be up to the researcher / author / editor / reviewer to decide what the global p-value threshold should be.

35. Post-Hoc Analysis

36. Questions So why don’t we perform the post-hoc analyses all the time then? p-values, is there a difference between 0.049 and 0.051? p-values and effect sizes, which is better? P-values or confidence intervals? Power (sensitivity) and specificity, can we attempt to maximise both?

37. t-tests in SPSS

38. Consider the mathematics.xls dataset again. 1. The average marks of the mathematics exam before starting the omega 3 trial is 70. Is there any evidence that the marks after the omega 3 trial is higher than 70? 2. It is traditionally believed that male students tend to outperform female students in mathematics. Based on the marks before the start of the trial, is there any evidence in support of this hypothesis. 3. Is there any evidence that consuming omega 3 improves the performance in the mathematics exam? 4. Is there any difference in the marks before the trial between the three schools? If there is, which school exhibited the best performance? 5. Is there any difference in the omega 3 consumption between male and female students?

39. Average marks after = 70? This should be relatively straightforward that we need to perform a one- sample test of the mean value. However, before we can decide on the use of a one-sample t-test, we need to check whether the data is indeed symmetrically distributed (and to go through the usual data exploratory procedures). Recall there are at least two ways of doing this in SPSS: 1.Qualitative assessment using a histogram. 2.Quantitative assessment with a Shapiro-Wilk’s Test.

40. Shapiro-Wilk’s test whether the data is normally distributed. H 0 : Data is normally distributed H 1 : Data is not normally distributed

42. H0: Mean marks = 70 H1: Mean marks > 70 (1-tailed test) Actual 2-tailed p-value = 3.28  10 -5 1-tailed p-value = 1.64  10 -5 Conclusion: There exists significant evidence that the marks after consuming omega 3 is greater than 70 (p-value = 1.64  10 -5 ).

43. Males better than females? It should be immediately clear that there are 2 groups (or “populations”) here: one for male students, and one for female students. Thus we can use the 2-independent samples t-test to compare the mean marks for the two groups. However, as before, we need to assess whether there is any violation of the normality or symmetrical distribution assumption. Important: As there are two groups, we need to assess whether both groups satisfy this assumption!

44. Assessing assumptions for 2-independent samples t-test

46. H0: Mean marks for males = Mean marks for females H1: Mean marks for males > Mean marks for females (1-tailed test) Which row to interpret? Levene’s test for equality of variances: H0: Variance for group 1 = Variance for group 2 H1: Variance for group 1  Variance for group 2 However, regardless of what you see here, always go ahead and interpret the second row not assuming equal variances.. 2-tailed p-value, need to convert to 1-tailed p-value. As mean difference = -0.935, which is in opposite direction to H1, therefore: 1-tailed p-value = 1 – 0.5  0.460 = 0.77 Conclusion: There is no evidence (p- value = 0.77) that males perform better than females in the mathematics exam before the start of the omega 3 trial.

47. Evidence that omega3 improves performance? Most appropriate analysis is to compare the marks after against the marks before for the same individual. Here the appropriate test is the paired-sample t-test. H0: There is no difference between the marks before and after within each individual (or Difference = 0) H1: The mark after taking omega3 is higher than the mark before (or Difference > 0)

51. Actual 2-tailed p-value = 4.06  10 -14 1-tailed p-value = 2.03  10 -14 Since mean difference is in the same direction as the alternative hypothesis. Conclusion: There is overwhelming evidence (p-value = 2.03  10 -14 ) that the exam marks after the omega 3 trial are higher than the marks before the trial.

52. Is there any difference in the performance across the 3 schools? There are three groups that we want to compare the average marks. The use of an ANOVA should immediately come to mind for this purpose. H0: There is no difference in the mean marks between the three schools. H1: At least one school has a different mean mark when compared to the rest of the schools.

55. Conclusion: There is at no significant evidence (p-value = 0.063) of a difference in marks between the three schools. (or at best, marginal evidence of a difference, between schools 1 and 2).

56. Is there any difference in omega 3 consumption between males and females? Feeling confident, we decided we can jump straight to performing the definitive analysis without exploring the data. So the appropriate test here is a 2-independent sample t-test. Is this correct?

57. With histograms like these, there really isn’t a need to perform the Shapiro-Wilk tests!

58. Parametric tests It’s clear that any tests that assume the data to be symmetrically distributed will not be applicable in comparing omega 3 consumption, since this outcome is extremely right skewed. There is thus a need for statistical tests that do not explicitly require such assumptions on the distribution of the variable – also known as non- parametric tests. So far, the tests we have seen are considered parametric tests – which require assumptions on the distribution of the data to be satisfied before they can be correctly used. REMEMBER! The computer does not recognise when a test is inappropriate, you have to know!

59. Relationship between p-values and confidence intervals Let’s have a quick recap of all the valid analyses: Look at the p-values and the confidence intervals. Notice a trend? Significant at 0.05 threshold and 0 not in 95% CI Not Significant at 0.05 threshold and 0 is in 95% CI Significant at 0.05 threshold and 0 is not in 95% CI

60. Actually, even at the numerical EDA stage, the 95% confidence intervals about the means are already very informative for indicating whether there will be any differences between the two means in a formal comparison.

61. Relationship between p-values and confidence intervals Thus, if the test value under the null hypothesis falls within the 95% CI of the estimated quantity  p-value from the hypothesis test will be > 0.05. If the test value under the null hypothesis does not fall within the 95% CI  p-value from the hypothesis test will be < 0.05. Conversely, If the p-value from the hypothesis test is < 0.05  the 95% CI will not contain the test value under H0. If the p-value from the hypothesis test is > 0.05  the 95% CI is certain to contain the test value under H0.

62. understand the concept of the null and alternative hypotheses know what a test statistic is, and understand that it is always calculated assuming the null hypothesis is true understand what is meant by power, sensitivity, specificity and type I and type II errors understand and interpret a p-value from a hypothesis test know which statistical tests should be used know the assumptions for these statistical tests understand the relationship between a p-value and CI perform the appropriate analyses in SPSS and RExcel Students should be able to