Statistical Significance or Hypothesis Testing

Significance testing Learning objectives of this lecture are to Understand Hypothesis: definition & types Know What does Statistical Significance really mean ? Understand Why should we test significance? Know How to test statistical significance? Discuss Type 1 error Type 2 error Know How to interpret P value?

Significance testing Learning outcomes At the end of this lecture the students should be able to Define Hypothesis: Explain the meaning of Statistical Significance. Describe the need for testing of significance. Steps of testing statistical significance Explain Type 1 error Type 2 error Interpret P value?

Statistics InferentialDescriptive Hypothesis Testing

Types of Data Interval Small Medium Large Ordinal Height Ratio Females Nominal

Descriptives RANGERANGE Average Standard Deviation

Variabilty RANGERANGE

Hypothesis Testing Research Question: Is there a difference in the weights of AIMST male and female medical students?

Weight (kilos) of AIMST medical students Hypothesis Testing is all about answering the question, “Could These Observations [Difference In Weight] Have Occurred By CHANCE Or Is It REAL?” MaleFemale Number3230 Mean74.6 kg55.4 kg Median72.5 kg54.5 kg SD

Explanation for the difference Due to Chance Due to Bias – 2 groups are dissimilar (in what they eat, health behavior patterns, male-female ratio, different ethnic mix, etc) True difference (confirms hypothesis that there is a significant difference in the weights of AIMST male and female medical students)

Role of chance To rule out chance as being the explanation for the difference, statistical tests are carried out on the data. Two approaches:  1. Significance test - P-values  2. Estimation - confidence intervals. Performance of hypothesis testing is by using tests of significance such as  t test  χ ² (chi-squared) test

Why should we test significance? Is the difference obtained TRUE or SPURIOUS? Will another set of samples be also different? What are the chances that the difference obtained is spurious? The above questions can be answered by STAT TEST

Data Type and Statistical Test Variable 1Variable 2SummaryStat Test Nominal Ordinal Counts Proportion Chi Sqr Proportions Nominal Ordinal RatioMeansT-test Z-test Ratio Scatter PlotCorelation

Heart rate (bpm) Mean ± SEM n In men ± In women ± The difference between means ( ) = 7.11 We do not need a stat test of significance, if only : a. the data from all subjects in a group are IDENTICAL b. we can collect data from all subjects in a population Why should we test significance?

Heart rate (bpm) Mean ± SEM n In men ± In women ± The difference between means ( ) = 7.11 t test t = t test t = Xt – Xc = t is which is more than 1.96 [2 SE] so P less 0.05 The difference is SIGNIFICANT, so REJECT Null Hypothesis There is a SIGNIFICANT, REAL difference in the heart rate of Males & Females Testing Differences SE(X T - X C )

Normal Distribution of mean - SE Av. Sd - Sd+ Standard deviation of the mean – standard Error

Statistical Analysis control group mean treatment group mean Is there a difference?

How to find statistical difference?

What does difference mean? medium variability high variability low variability The mean difference is the same for all three cases

What does difference mean? medium variability high variability low variability Which one shows the greatest difference?

What does difference mean? a statistical difference is a function of the difference between means relative to the variability a small difference between means with large variability could be due to chance low variability Which one shows the greatest difference?

Hypothesis Hypothesis is a statement/ research question Null hypothesis [Ho] (statistical hypothesis) states that there is no difference between groups / variables compared. Alternative hypothesis [Ha] ( research hypothesis) states that there is a difference between groups. example. New drug ‘X’ is an analgesic - (Research hypothesis) New drug ‘X’ is not an analgesic – (Null hypothesis) New drug ‘X’ is different from drug ‘Y’- (Alt. hypothesis)

How to test statistical significance? State Null hypothesis Set alpha (level of significance) Identify the variables to be analysed Identify the groups to be compared Choose a test Calculate the test statistic Find out the P value Interpret the P value Calculate the CI of the difference Calculate Power if required

The Judges Dilemma Guilty/Dis eased Not Guilty/ No Diseas e Type II Error Type I error Diseased Not Diseased

Type 1 Type 2 Errors

Alpha / type 1 error  The level of significance is to be set  It is generally set at 0.05 (5%) and not above. [This means that the investigator is willing to accept a 5% risk of committing type 1 error.]  If the P value is less than this limit then null hypothesis is rejected i.e. the difference between groups is not due to chance.

Beta / type II error A level of 0.20 for Beta error is considered adequate The Statistical Power or the ability of a study to detect a true difference between groups is [1-β] Statistical Power for a study with a Beta level of 0.20 would be = 0.80 or 80%

Which Error is More Dangerous Most will vote for Type 2 Type 1 error is generally the more dangerous. Except in Quality Control

Group Work In your group identify a dependant variable that could be compared between two groups (dependant variable) What is the data type of dependant and independent variable? Stae a research question, a null hypothesis and alternate hypothesis.

Calculating test statistic difference between group means variability of groups X T - X C SE(X T - X C ) e.g. t test t = e.g. t test t = Determining P Determining P Find out the degrees of freedom (df) – (n-1) Use t and df to find out P using a formula or ‘critical values table’

Table 8.1 Distribution by socioeconomic class of patients admitted to self poisoning (sample A) and gastroenterological (sample B) units Socioeconomic class Sampl es TotalProportion in group A AB ab n = a + b p = a/n I II III IV V Total

Figure 8.3 The critical region for an alpha of.05 Copyright © 2002 Wadsworth Group. Wadsworth is an imprint of the Wadsworth Group, a division of Thomson Learning

Figure 8.4 The critical region boundaries for three levels of significance Copyright © 2002 Wadsworth Group. Wadsworth is an imprint of the Wadsworth Group, a division of Thomson Learning

Levels of significance Critical values of zLevel of Significance

Heart rate (bpm) Mean ± SEM n In men ± In women ± The difference between means ( ) = 7.11 t test t = t test t = Xt – Xc = t is which is more than 1.96 [2 SE] so P less 0.05 The difference is SIGNIFICANT, so REJECT Null Hypothesis There is a SIGNIFICANT, REAL difference in the heart rate of Males & Females Why should we test significance? SE(X T - X C )

Significant, real & important The difference between high and low tide has no significant effect on the depth of the ocean.

Significant, real & important Statistically significant difference of a study - may be real but not necessarily important e.g. a 1 mm increase in BP is statistically significant but not clinically significant. Not statistically significant – could be real. “n.s.” may be due to small sample size. The difference may still be important. “n.s.” does not imply no effect, but only failed to show the existence of one.

Thank you