Tuesday, September 10, 2013 Introduction to hypothesis testing

Last time:

Probability & the Distribution of Sample Means We can use the Central Limit Theorem to calculate z-scores associated with individual sample means (the z-scores are based on the distribution of all possible sample means). Each z-score describes the exact location of its respective sample mean, relative to the distribution of sample means. Since the distribution of sample means is normal, we can then use the unit normal table to determine the likelihood of obtaining a sample mean greater/less than a specific sample mean.

Probability & the Distribution of Sample Means When using z scores to represent sample means, the correct formula to use is:

Probability & the Distribution of Sample Means EXAMPLE: What is the probability of obtaining a sample mean greater than M = 60 for a random sample of n = 16 scores selected from a normal population with a mean of μ = 65 and a standard deviation of σ = 20? M = 60; μ = 65; σ = 20; n = 16

Last topic before the exam: Hypothesis testing (pulls together everything we’ve learned so far and applies it to testing hypotheses about about sample means). Before we move on, questions about CLT, distributions of samples, standard error of the mean and how to calculate it?

Hypothesis testing Example: Testing the effectiveness of a new memory treatment for patients with memory problems –Our pharmaceutical company develops a new drug treatment that is designed to help patients with impaired memories. –Before we market the drug we want to see if it works. –The drug is designed to work on all memory patients, but we can’t test them all (the population). –So we decide to use a sample and conduct the following experiment. –Based on the results from the sample we will make conclusions about the population.

Hypothesis testing Example: Testing the effectiveness of a new memory treatment for patients with memory problems Memory treatment No Memory treatment Memory patients Memory Test Memory Test 55 errors 60 errors 5 error diff Is the 5 error difference: –A “real” difference due to the effect of the treatment –Or is it just sampling error?

Testing Hypotheses Hypothesis testing – Procedure for deciding whether the outcome of a study (results for a sample) support a particular theory (which is thought to apply to a population) – Core logic of hypothesis testing Considers the probability that the result of a study could have come about by chance if the experimental procedure had no effect If this probability is low, scenario of no effect is rejected and the theory behind the experimental procedure is supported

Hypothesis testing Can make predictions about likelihood of outcomes based on this distribution. Distribution of possible outcomes (of a particular sample size, n) In hypothesis testing, we compare our observed samples with the distribution of possible samples (transformed into standardized distributions) This distribution of possible samples is often Normally Distributed (This follows from the Central Limit Theorem).

Inferential statistics Hypothesis testing – Core logic of hypothesis testing Considers the probability that the result of a study could have come about if the experimental procedure had no effect If this probability is low, scenario of no effect is rejected and the theory behind the experimental procedure is supported Step 1: State your hypotheses Step 2: Set your decision criteria Step 3: Collect your data & compute your test statistics Step 4: Make a decision about your null hypothesis –A four step program

–Step 1: State your hypotheses : as a research hypothesis and a null hypothesis about the populations Null hypothesis (H 0 ) Research hypothesis (H A ) Hypothesis testing There are no differences between conditions (no effect of treatment) Generally, not all groups are equal This is the one that you test Hypothesis testing: a four step program –You aren’t out to prove the alternative hypothesis If you reject the null hypothesis, then you’re left with support for the alternative(s) (NOT proof!)

In our memory example experiment: Testing Hypotheses μ Treatment > μ No Treatment μ Treatment < μ No Treatment H0:H0: HA:HA: – Our theory is that the treatment should improve memory (fewer errors). –Step 1: State your hypotheses Hypothesis testing: a four step program One -tailed

In our memory example experiment: Testing Hypotheses μ Treatment > μ No Treatment μ Treatment < μ No Treatment H0:H0: HA:HA: – Our theory is that the treatment should improve memory (fewer errors). –Step 1: State your hypotheses Hypothesis testing: a four step program μ Treatment = μ No Treatment μ Treatment ≠ μ No Treatment H0:H0: HA:HA: – Our theory is that the treatment has an effect on memory. One -tailedTwo -tailed no direction specified direction specified

One-Tailed and Two-Tailed Hypothesis Tests Directional hypotheses – One-tailed test Nondirectional hypotheses – Two-tailed test

Testing Hypotheses –Step 1: State your hypotheses –Step 2: Set your decision criteria Hypothesis testing: a four step program Your alpha ( α ) level will be your guide for when to reject or fail to reject the null hypothesis. –Based on the probability of making a certain type of error

Testing Hypotheses –Step 1: State your hypotheses –Step 2: Set your decision criteria –Step 3: Collect your data & Compute sample statistics Hypothesis testing: a four step program

Testing Hypotheses –Step 1: State your hypotheses –Step 2: Set your decision criteria –Step 3: Collect your data & Compute sample statistics Hypothesis testing: a four step program Descriptive statistics (means, standard deviations, etc.) Inferential statistics (z-test, t-tests, ANOVAs, etc.)

Testing Hypotheses –Step 1: State your hypotheses –Step 2: Set your decision criteria –Step 3: Collect your data & compute sample statistics –Step 4: Make a decision about your null hypothesis Hypothesis testing: a four step program Based on the outcomes of the statistical tests researchers will either: –Reject the null hypothesis –Fail to reject the null hypothesis This could be the correct conclusion or the incorrect conclusion

Error types Type I error (α): concluding that there is a difference between groups (“an effect”) when there really isn’t. – Sometimes called “significance level” or “alpha level” – We try to minimize this (keep it low) Type II error (β): concluding that there isn’t an effect, when there really is. – Related to the Statistical Power of a test (1-β)

Error types Real world (‘truth’) H 0 is correct H 0 is wrong Experimenter’s conclusions Reject H 0 Fail to Reject H 0 There really isn’t an effect There really is an effect

Error types Real world (‘truth’) H 0 is correct H 0 is wrong Experimenter’s conclusions Reject H 0 Fail to Reject H 0 I conclude that there is an effect I can’t detect an effect

Error types Real world (‘truth’) H 0 is correct H 0 is wrong Experimenter’s conclusions Reject H 0 Fail to Reject H 0 Type I error Type II error

Performing your statistical test H 0 : is true (no treatment effect)H 0 : is false (is a treatment effect) Two populations One population What are we doing when we test the hypotheses? Real world (‘truth’) MAMA they aren’t the same as those in the population of memory patients MAMA the memory treatment sample are the same as those in the population of memory patients.

Performing your statistical test What are we doing when we test the hypotheses? – Computing a test statistic: Generic test Could be difference between a sample and a population, or between different samples Based on standard error or an estimate of the standard error

“Generic” statistical test The generic test statistic distribution (think of this as the distribution of sample means) – To reject the H 0, you want a computed test statistic that is large – What’s large enough? The alpha level gives us the decision criterion Distribution of the test statistic α-level determines where these boundaries go

“Generic” statistical test If test statistic is here Reject H 0 If test statistic is here Fail to reject H 0 Distribution of the test statistic The generic test statistic distribution (think of this as the distribution of sample means) – To reject the H 0, you want a computed test statistics that is large – What’s large enough? The alpha level gives us the decision criterion

“Generic” statistical test Reject H 0 Fail to reject H 0 The alpha level gives us the decision criterion One -tailedTwo -tailed Reject H 0 Fail to reject H 0 Reject H 0 Fail to reject H 0 α = split up into the two tails

“Generic” statistical test Reject H 0 Fail to reject H 0 The alpha level gives us the decision criterion One -tailedTwo -tailed Reject H 0 Fail to reject H 0 Reject H 0 Fail to reject H 0 α = all of it in one tail

“Generic” statistical test Reject H 0 Fail to reject H 0 The alpha level gives us the decision criterion One -tailedTwo -tailed Reject H 0 Fail to reject H 0 Reject H 0 Fail to reject H 0 α = all of it in one tail

“Generic” statistical test An example: One sample z-test Memory example experiment: We give a n = 16 memory patients a memory improvement treatment. How do they compare to the general population of memory patients who have a distribution of memory errors that is Normal, μ = 60, σ = 8? After the treatment they have an average score of M = 55 memory errors. Step 1: State the hypotheses H0:H0: The treatment sample is the same as (or worse than) the population of memory patients. HA:HA: The treatment sample does better than the population (fewer errors) μ Treatment ≥ μ pop = 60 μ Treatment < μ pop = 60

“Generic” statistical test An example: One sample z-test Memory example experiment: We give a n = 16 memory patients a memory improvement treatment. How do they compare to the general population of memory patients who have a distribution of memory errors that is Normal, μ = 60, σ = 8? After the treatment they have an average score of M = 55 memory errors. Step 2: Set your decision criteria μ Treatment ≥ μ pop = 60 μ Treatment < μ pop = 60 α = 0.05 One -tailed

“Generic” statistical test An example: One sample z-test Memory example experiment: We give a n = 16 memory patients a memory improvement treatment. How do they compare to the general population of memory patients who have a distribution of memory errors that is Normal, μ = 60, σ = 8? After the treatment they have an average score of M = 55 memory errors. α = 0.05 One -tailed Step 3: Collect your data & μ Treatment ≥ μ pop = 60 μ Treatment < μ pop = 60

“Generic” statistical test An example: One sample z-test Memory example experiment: We give a n = 16 memory patients a memory improvement treatment. How do they compare to the general population of memory patients who have a distribution of memory errors that is Normal, μ = 60, σ = 8? After the treatment they have an average score of M = 55 memory errors. α = 0.05 One -tailed Step 3: Collect your data & compute your test statistics = -2.5 μ Treatment ≥ μ pop = 60 μ Treatment < μ pop = 60

“Generic” statistical test An example: One sample z-test Memory example experiment: We give a n = 16 memory patients a memory improvement treatment. How do they compare to the general population of memory patients who have a distribution of memory errors that is Normal, μ = 60, σ = 8? After the treatment they have an average score of M = 55 memory errors. α = 0.05 One -tailed Step 4: Make a decision about your null hypothesis 5% Reject H 0 μ Treatment ≥ μ pop = 60 μ Treatment < μ pop = 60

“Generic” statistical test An example: One sample z-test Memory example experiment: We give a n = 16 memory patients a memory improvement treatment. How do they compare to the general population of memory patients who have a distribution of memory errors that is Normal, μ = 60, σ = 8? After the treatment they have an average score of μ = 55 memory errors. α = 0.05 One -tailed Step 4: Make a decision about your null hypothesis - Reject H 0 - Support for our H A, the evidence suggests that the treatment decreases the number of memory errors μ Treatment ≥ μ pop = 60 μ Treatment < μ pop = 60