Synthesis and Review 2/20/12 Hypothesis Tests: the big picture Randomization distributions Connecting intervals and tests Review of major topics Open Q+A Section 4.4, 4.5, ES 2 Professor Kari Lock Morgan Duke University

Make one double-sided page of notes for in- class exam WORK PRACTICE PROBLEMS! (old exams and solutions to review questions under Documents on course website) Read sections corresponding to anything you are still confused about Practice using technology to summarize, visualize, and perform inference on data To Do

Today, 3 – 4 pm (Prof Morgan) Today, 4 – 6 pm (Christine) Tuesday, 3 – 6 pm (Prof Morgan) Tuesday, 6 – 8 pm (Yue) Tuesday, 8 – 9 pm (Michael) (My office hours this week have been moved to Monday and Tuesday to answer questions before the exam) Office Hours this Week

Hypothesis Testing 1.Define the parameter(s) of interest 2.State your hypotheses 3.Set significance level,  (usually 0.05 if unspecified) 4.(Collect your data) 5.Plot your data 6.Calculate the observed sample statistic 7.Create a randomization distribution 8.Calculate the p-value 9.Assess the strength of evidence against H 0 10.Make a formal decision based on the significance level 11.Interpret the conclusion in context

Exercise and Gender Among college students, does one gender exercise more than the other?

Exercise and Gender

p-value =  Little evidence against H 0  Do not reject H 0  This study does not provide evidence that there is any association between gender and exercise times among college students Conclusion: Results this extreme would happen about 22% of the time just by random chance if H 0 were true, so this study does not provide adequate evidence against H 0 Think:

A randomization distribution is the distribution of statistics that would be observed, just by random chance, if the null hypothesis were true 1.Simulate randomizations assuming the null hypothesis is true 2.Calculate the statistic for each simulated randomization Randomization Distribution

In a randomized experiment the “randomness” is the random allocation of cases to treatment groups If the null hypothesis is true, it doesn’t make any difference which treatment group you get placed in Simulate randomizations assuming H 0 is true by reallocate units to treatment groups, and keeping the response values the same Randomized Experiments

In observational studies, there is no random allocation to treatment groups In observational studies, what does “by random chance” even mean? What is random??? How could we generate a randomization distribution for observational studies? Observational Studies

When data is collected by random sampling, without random allocation between groups, we can bootstrap to see what would happen by random chance Bootstrapping (resampling with replacement) simulates the distribution of the sample statistic that we would observe when taking many random samples of the population Bootstrapping

For a randomization distribution however, we need to know the distribution of the sample statistic, when the null hypothesis is true How could we bootstrap assuming the null hypothesis is true? Add/subtract values to each unit first to make the null hypothesis true (“shift the distribution”) Bootstrapping

Reallocating versus Resampling What is random?How do we simulate “random chance”? Randomized Experiments Random assignment to treatment groups Reallocate (rerandomize) Observational Studies Random sampling from the population Resample (bootstrap) In both cases, we need to make the null hypothesis true for a randomization distribution

Was the exercise by gender data collected via a randomized experiment? (a)Yes (b) No (c) There is no way to tell Exercise by Gender

The randomness is not who is which gender (as with randomized experiments), but who is selected to be a part of the study Male sample mean: 12.4 hours Female sample mean: 9.4 hours Add 3 hours to all the females, and then resample using bootstrapping Exercise by Gender

Reallocating and resampling usually give similar answers in terms of a p-value For this class, it is fine to just use reallocating for tests, even if it is not actually a randomized experiment The point is to understand the reason for generating a randomization distribution Method of Randomization

Let’s return to the body temperature data Using bootstrapping, we found a 95% confidence interval for the mean body temperature to be (98.05 ,  ) Let’s do a hypothesis test to see how much evidence this data provides against  = 98.6  H 0 :  = 98.6  H a :  ≠ 98.6  Body Temperatures

How would we create a randomization distribution? The sample mean is . Add 0.34  to each unit so we can sample with replacement mimicking sampling from a population with mean 98.6  Take many bootstrap samples to create a randomization distribution Body Temperatures

Randomization Distribution p-value = 0.002

Two Distributions

If a (1 – α)% confidence interval does not contain the value of the null hypothesis, then a two-sided hypothesis test will reject the null hypothesis using significance level α Intervals provide a range of plausible values for the population parameter, tests are designed to assess evidence against a null hypothesis Intervals and Tests

Using bootstrapping, we found a 95% confidence interval for the mean body temperature to be (98.05 ,  ) H 0 :  = 98.6  H a :  ≠ 98.6  At α = 0.05, we would reject H 0 Body Temperatures

REVIEW

Population Sample Sampling Statistical Inference The Big Picture Exploratory Data Analysis

Was the sample randomly selected? Possible to generalize to the population Yes Should not generalize to the population No Was the explanatory variable randomly assigned? Possible to make conclusions about causality Yes Can not make conclusions about causality No Data Collection

Variable(s)VisualizationSummary Statistics Categoricalbar chart, pie chart frequency table, relative frequency table, proportion Quantitativedotplot, histogram, boxplot mean, median, max, min, standard deviation, z-score, range, IQR, five number summary Categorical vs Categorical side-by-side bar chart, segmented bar chart, mosaic plot two-way table, proportions Quantitative vs Categorical side-by-side boxplotsstatistics by group Quantitative vs Quantitative scatterplotcorrelation

Descriptive Statistics Think of a topic or question you would like to use data to help you answer. – What would the cases be? – What would the variables be? (Limit to one or two variables)

Descriptive Statistics How would you visualize and summarize the variable or relationship between variables? a)bar chart/pie chart, proportions, frequency table/relative frequency table b)dotplot/histogram/boxplot, mean/median, sd/range/IQR, five number summary c)side-by-side or segmented bar plots/mosaic plots, difference in proportions, two-way table d)side-by-side boxplot, stats by group e)scatterplot, correlation

Statistic vs Parameter A sample statistic is a number computed from sample data. A population parameter is a number that describes some aspect of a population

Sampling Distribution A sampling distribution is the distribution of statistics computed for different samples of the same size taken from the same population The spread of the sampling distribution helps us to assess the uncertainty in the sample statistic In real life, we rarely get to see the sampling distribution – we usually only have one sample

A bootstrap sample is a random sample taken with replacement from the original sample, of the same size as the original sample A bootstrap statistic is the statistic computed on the bootstrap sample A bootstrap distribution is the distribution of many bootstrap statistics Bootstrap

Original Sample Bootstrap Sample Bootstrap Statistic Sample Statistic Bootstrap Statistic Bootstrap Distribution

Confidence Interval A confidence interval for a parameter is an interval computed from sample data by a method that will capture the parameter for a specified proportion of all samples A 95% confidence interval will contain the true parameter for 95% of all samples

Standard Error The standard error (SE) is the standard deviation of the sample statistic The SE can be estimated by the standard deviation of the bootstrap distribution For symmetric, bell-shaped distributions, a 95% confidence interval is

Percentile Method If the bootstrap distribution is approximately symmetric, a P% confidence interval can be gotten by taking the middle P% of a bootstrap distribution

Bootstrap Distribution

How unusual would it be to get results as extreme (or more extreme) than those observed, if the null hypothesis is true? If it would be very unusual, then the null hypothesis is probably not true! If it would not be very unusual, then there is not evidence against the null hypothesis Hypothesis Testing

The p-value is the probability of getting a statistic as extreme (or more extreme) as that observed, just by random chance, if the null hypothesis is true The p-value measures evidence against the null hypothesis p-value

Hypothesis Testing

A randomization distribution is the distribution of sample statistics we would observe, just by random chance, if the null hypothesis were true The p-value is calculated by finding the proportion of statistics in the randomization distribution that fall beyond the observed statistic Randomization Distribution

Statistical Conclusions Strength of evidence against H 0 : Formal decision of hypothesis test, based on  = 0.05 :

Formal Decisions For a given significance level, , p-value <   Reject H o p-value >   Do not Reject H o

Errors Reject H 0 Do not reject H 0 H 0 true H 0 false TYPE I ERROR TYPE II ERROR Truth Decision

QUESTIONS???