SUMMARY Hypothesis testing

Self-engagement assesment

Null hypothesis no song song Null hypothesis: I assume that populations without and with song are same. At the beginning of our calculations, we assume the null hypothesis is true.

Hypothesis testing song Because of such a low probability, we interpret 8.2 as a significant increase over 7.8 caused by undeniable pedagogical qualities of the 'Hypothesis testing song'.

Four steps of hypothesis testing 1. Formulate the null and the alternative (this includes one- or two-directional test) hypothesis. 2. Select the significance level α – a criterion upon which we decide that the claim being tested is true or not. --- COLLECT DATA Compute the p-value. The p-value is the probability that the data would be at least as extreme as those observed, if the null hypothesis were true. 4. Compare the p-value to the α-level. If p ≤ α, the observed effect is statistically significant, the null is rejected, and the alternative hypothesis is valid. Za předpokladu platnosti nulové hypotézy je p-hodnota pravděpodobnost, že data jsou nejméně tak extrémní jako ta pozorovaná.

One-tailed and two-tailed one-tailed (directional) test two-tailed (non-directional) test Z-critical value, what is it?

NEW STUFF

Decision errors Hypothesis testing is prone to misinterpretations. It's possible that students selected for the musical lesson were already more engaged. And we wrongly attributed high engagement score to the song. Of course, it's unlikely to just simply select a sample with the mean engagement of 8.2. The probability of doing so is , pretty low. Thus we concluded it is unlikely. But it's still possible to have randomly obtained a sample with such a mean.

Four possible things can happen Decision Reject H 0 Retain H 0 State of the world H 0 true 13 H 0 false 24 In which cases we made a wrong decision?

Four possible things can happen Decision Reject H 0 Retain H 0 State of the world H 0 true 1 H 0 false 4 In which cases we made a wrong decision?

Four possible things can happen Decision Reject H 0 Retain H 0 State of the world H 0 true Type I error H 0 false Type II error

Type I error When there really is no difference between the populations, random sampling can lead to a difference large enough to be statistically significant. You reject the null, but you shouldn't. False positive – the person doesn't have the disease, but the test says it does

Type II error When there really is a difference between the populations, random sampling can lead to a difference small enough to be not statistically significant. You do not reject the null, but you should. False negative - the person has the disease but the test doesn't pick it up Type I and II errors are theoretical concepts. When you analyze your data, you don't know if the populations are identical. You only know data in your particular samples. You will never know whether you made one of these errors.

The trade-off If you set α level to a very low value, you will make few Type I/Type II errors. But by reducing α level you also increase the chance of Type II error. Your decision whether to allow more false positives (Type I error) or more false negatives (Type II error) is based on practical consequences of these errors. See next example

Clinical trial for a novel drug Drug that should treat a disease for which there exists no therapy. If the result is statistically significant, drug will me marketed. If the result is not statistically significant, work on the drug will cease. Type I error: treat future patients with ineffective drug Type II error: cancel the development of a functional drug for a condition that is currently not treatable. Which error is worse? I would say Type II error. To reduce its risk, it makes sense to set α = 0.10 or even higher. Harvey Motulsky, Intuitive Biostatistics

Clinical trial for a me-too drug Similar to the previous example, but now our prospective drug should treat a disease for which there already exists another therapy. Type I error: treat future patients with ineffective drug Type II error: cancel the development of a functional drug for a condition that can be treated adequately with existing drugs. Thinking scientifically (not commercially) I would minimize the risk of Type I error (set α to a very low value). Harvey Motulsky, Intuitive Biostatistics

population of students that did not attend the musical lesson population of students that did attend the musical lesson parameters are known sample statistic is known

Test statistic test statistic Z-test

New situation An average engagement score in the population of 100 students is 7.5. A sample of 50 students was exposed to the musical lesson. Their engagement score became 7.72 with the s.d. of 0.6. DECISION: Does a musical performance lead to the change in the students' engagement? Answer YES/NO. Setup a hypothesis test, please.

Hypothesis test

Formulate the test statistic but this is unknown! population of students that did attend the musical lesson sample population of students that did not attend the musical lesson known unknown

t-statistic one sample t-test jednovýběrový t-test

One-sample t-test

Quiz

Z-test vs. t-test

Typical example of an one-sample t-test

Two-sample t-test So far, we have been working with just one sample. Now, we want to compare the sample of students without song with the sample of students with the song. population of students that did not attend the musical lesson population of students that did attend the musical lesson unknown sample statistic is known sample statistic is known dvouvýběrový t-test

Two-sample t-test

Dependent t-test for paired samples Two samples are dependent when the same subject takes the test twice. For example, give one person two different conditions to see how he/she reacts. paired t-test (párový t-test) Examples: Each subject is assigned to two different conditions Give a person two types of treatment. Growth over time. Compare the effect of the treatment (drug) after 12 hours and after 24 hours.

Engagement example

Test statistic

Paired t-test

Dependent samples Advantages we can use fewer subjects cost-effective less time-consuming Disadvantages carry-over effects order may influence results