Lecture 3 Outline: Thurs, Sept 11 Chapters Probability model for 2-group randomized experiment Randomization test p-value Probability model for random sampling

Vocabulary of Experiments A study is an experiment when we actually do something to people, animals or objects to observe the response. Experimental units are the things to which treatments are applied, e.g., people, rats, samples of materials or pieces of land. When units are human beings, they are called subjects. A specific experimental condition applied to the units is called a treatment. The “control” refers to a treatment that is considered a baseline for comparing all other treatments. Creativity study: Experimental units? Treatments?

Probability Model for 2-treatment Randomized Experiment Creativity Study –Chance mechanism for randomizing units to treatment groups ensures that every subset of 24 subjects gets the same chance of becoming intrinsic group –For example, 23 red and 24 black cards could be shuffled and dealt, one to each subject and the subjects with black cards would be the intrinsic group. –Tables of random numbers can be used to assign units to groups (assign the units with the 24 highest numbers to group 1).

Additive Treatment Effect Model Potential Outcomes: For each subject, we can imagine what the subject’s outcome would be if placed in the extrinsic group (Y) and what subject’s outcome would be if placed in the intrinsic group (Y*). We only see the outcome for the group to which they were assigned. Additive Treatment Effect Model: For every subject, Y*=Y+ is a parameter – an unknown constant that describes a key feature in model for answering questions of interest.

Test for Treatment Effect Meaning of : – >0: Intrinsic questionnaire improves creativity. – =0 : Intrinsic questionnaire (treatment) makes no difference. – <0: Intrinsic questionnaire makes creativity worse. Hypothesis testing: Questions of interests are translated into questions about parameters in probability models. Null hypothesis (H 0 ): =0 (group status has no effect on outcome) Alternative Hypothesis: (group status has an effect on outcome)

Test Statistic A test statistic is a numerical data summary for testing a hypothesis. We try to find a test statistic that tends to be small when the null hypothesis is true and tends to be large when the alternative hypothesis is true. Test statistic for 2-group randomized exp.: –Let be the sample mean of the outcome for units assigned to group 1. –Let be the sample mean of the outcome for units assigned to group 2. –Test statistic: T=

Testing the Null Hypothesis If there is no treatment effect (H 0 ), subjects would receive same outcome regardless of their assigned group. If there is a treatment effect, subjects will receive a higher (or lower) outcome if they are assigned to one group. Therefore a large value of argues against the null hypothesis. But how large is large? Even if there is no treatment effect, T will not necessarily equal 0 because the random assignment can result in an uneven mix of abilities.

Randomization Test p-value The observed value of the test statistic can be large because –(a) there is an effect of the treatment –(b) the random assignment resulted in an uneven mix A randomization test p-value is the probability associated with explanation (b) The smaller the p-value, the less believable (b) is as an explanation.

Exact Calculation of the p-value The p-value is the probability that T>=4.14 if, in fact, there is no treatment effect (and based on the random assignment of units to groups) Important starting point: If there is no treatment effect, then the creativity score for an individual would have been the same had they been assigned to the other group. Exact Calculation of p-value –Calculate T for every possible grouping of the 47 numbers into groups of size 23 and 24 –The p-value is the proportion of regroupings with T>=4.14.

Exact calculation of p-value If there is no treatment effect, subjects would receive same outcome regardless of their assigned group. Distribution of test statistic T if there is no treatment effect: –For every possible random assignment of units into two groups, calculate T using the observed outcomes. –The T’s associated with each possible random assignment have the same probability. The p-value is the probability that if the null hypothesis were true (no treatment effect), T would be greater than or equal to the observed T 0

Example Suppose the creativity study had just six students. Suppose the three students assigned to the intrinsic group had scores of 12, 20 and 28. The three students assigned to the extrinsic group had scores of 10, 18 and 26. Calculate the p-value for testing if there is a treatment effect.

P-value for Creativity Study For the actual creativity study, using a computer program, the p-value is.011. Conclusion: either –(i) there is no treatment effect and we happened to get an uneven randomization –(ii) there is a treatment effect. The probability associated with (i) is.011. So either there is a treatment effect or we obtained an unusual (one-in-a-hundred) randomization. A p-value of around.01 is considered strong evidence against the null hypothesis, see pg. 47

One-sided vs. Two-sided Tests For some problems, we might know that the treatment effect is >=0 or <=0 and want to use a one-sided alternative hypothesis –(i) H a : or –(ii) H a : The appropriate test statistics for the one-sided alternative hypotheses are for testing (i) and for testing (ii), where it is assumed that group 2 is the “treatment” group and group 1 is the “control” group.

Randomization Distribution and p-value Defn.: The randomization distribution of a test statistic describes its possible values over all the ways the randomization could have turned out. The p-value of the randomization test is the proportion of the randomization distribution that is at least as large as the observed test statistic.

Approximating the p-value For the creativity study, there are 1.6*10 13 different groupings. Approximating the randomization test p-value. –(i) Monte Carlo simulation: Randomly choose many groupings. Approximate the randomization distribution by the histogram of the test statistic for the randomly chosen groupings –(ii) (Chapter 2). The randomization distribution of the “t-statistic” is approximated by the “t”-distribution.

Probability Model for Random Sampling Consider taking random samples from two populations with respective means and. Are the means different? H 0 : H 1 : Test statistic:

Probability Model for Random Sampling The sampling distribution of is represented by a histogram of all values for the statistic from all possible samples that can be drawn from the two populations. The p-value for testing a hypothesis (and confidence intervals) follows from an understanding of the sampling distribution. We will discuss sampling distribution in Ch. 2.