Copyright © Cengage Learning. All rights reserved. 8 Tests of Hypotheses Based on a Single Sample

Copyright © Cengage Learning. All rights reserved. 8.5 Some Comments on Selecting a Test

3 Once the experimenter has decided on the question of interest and the method for gathering data (the design of the experiment), construction of an appropriate test consists of three distinct steps: 1. Specify a test statistic (the function of the observed values that will serve as the decision maker). 2. Decide on the general form of the rejection region (typically reject H 0 for suitably large values of the test statistic, reject for suitably small values, or reject for either small or large values).

4 Some Comments on Selecting a Test 3. Select the specific numerical critical value or values that will separate the rejection region from the acceptance region (by obtaining the distribution of the test statistic when H 0 is true, and then selecting a level of significance). In the examples thus far, both Steps 1 and 2 were carried out in an ad hoc manner through intuition. For example, when the underlying population was assumed normal with mean  and known , we were led from X to the standardized test statistic

5 Some Comments on Selecting a Test For testing H 0 :  =  0 versus H a :  >  0, intuition then suggested rejecting H 0 when z was large. Finally, the critical value was determined by specifying the level of significance  and using the fact that Z has a standard normal distribution when H 0 is true. The reliability of the test in reaching a correct decision can be assessed by studying type II error probabilities.

6 Some Comments on Selecting a Test Issues to be considered in carrying out Steps 1–3 encompass the following questions: 1. What are the practical implications and consequences of choosing a particular level of significance once the other aspects of a test have been determined? 2. Does there exist a general principle, not dependent just on intuition, that can be used to obtain best or good test procedures?

7 Some Comments on Selecting a Test 3. When two or more tests are appropriate in a given situation, how can the tests be compared to decide which should be used? 4. If a test is derived under specific assumptions about the distribution or population being sampled, how will the test perform when the assumptions are violated?

8 Statistical Versus Practical Significance

9 Although the process of reaching a decision by using the methodology of classical hypothesis testing involves selecting a level of significance and then rejecting or not rejecting H 0 at that level , simply reporting the  used and the decision reached conveys little of the information contained in the sample data. Especially when the results of an experiment are to be communicated to a large audience, rejection of H 0 at level.05 will be much more convincing if the observed value of the test statistic greatly exceeds the 5% critical value than if it barely exceeds that value.

10 Statistical Versus Practical Significance This is precisely what led to the notion of P-value as a way of reporting significance without imposing a particular  on others who might wish to draw their own conclusions. Even if a P-value is included in a summary of results, however, there may be difficulty in interpreting this value and in making a decision. This is because a small P-value, which would ordinarily indicate statistical significance in that it would strongly suggest rejection of H 0 in favor of H a, may be the result of a large sample size in combination with a departure from H 0 that has little practical significance.

11 Statistical Versus Practical Significance In many experimental situations, only departures from H 0 of large magnitude would be worthy of detection, whereas a small departure from H 0 would have little practical significance. Consider as an example testing H 0 :  = 100 versus H a :  > 100 where  is the mean of a normal population with  = 10. Suppose a true value of  = 101 would not represent a serious departure from H 0 in the sense that not rejecting H 0 when  = 101 would be a relatively inexpensive error.

12 Statistical Versus Practical Significance For a reasonably large sample size n, this  would lead to an x value near 101, so we would not want this sample evidence to argue strongly for rejection of H 0 when x = 101 is observed. For various sample sizes, Table 8.1 records both the P-value when x = 101 and also the probability of not rejecting H 0 at level.01 when  = 101. An Illustration of the Effect of Sample Size on P-values and  Table 8.1

13 Statistical Versus Practical Significance The second column in Table 8.1 shows that even for moderately large sample sizes, the P-value of x = 101 argues very strongly for rejection of H 0, whereas the observed x itself suggests that in practical terms the true value of  differs little from the null value  0 = 100. The third column points out that even when there is little practical difference between the true  and the null value, for a fixed level of significance a large sample size will almost always lead to rejection of the null hypothesis at that level.

14 Statistical Versus Practical Significance To summarize, one must be especially careful in interpreting evidence when the sample size is large, since any small departure from H 0 will almost surely be detected by a test, yet such a departure may have little practical significance.

15 The Likelihood Ratio Principle

16 The Likelihood Ratio Principle Let x 1, x 2,…, x n be the observations in a random sample of size n from a probability distribution f (x;  ). The joint distribution evaluated at these sample values is the product f (x 1 ;  )  f (x 2 ;  )  · · ·  f (x n ;  ). As in the discussion of maximum likelihood estimation, the likelihood function is this joint distribution, regarded as a function of . Consider testing H 0 :  is in Ω 0 versus H a :  is in Ω a, where Ω 0 and Ω a are disjoint (for example, H 0 :   100 verses H a :   100).

17 The Likelihood Ratio Principle The likelihood ratio principle for test construction proceeds as follows: 1. Find the largest value of the likelihood for any  in Ω 0 (by finding the maximum likelihood estimate within Ω 0 and substituting back into the likelihood function). 2. Find the largest value of the likelihood for any  in Ω a. 3. Form the ratio

18 The Likelihood Ratio Principle The ratio (x 1,…, x n ) is called the likelihood ratio statistic value. The test procedure consists of rejecting H 0 when this ratio is small. That is, a constant k is chosen, and H 0 is rejected if (x 1,…, x n )  k. Thus H 0 is rejected when the denominator of greatly exceeds the numerator, indicating that the data is much more consistent with H a than with H 0. The constant k is selected to yield the desired type I error probability.

19 The Likelihood Ratio Principle Often the inequality  k can be manipulated to yield a simpler equivalent condition. For example, for testing H 0 :    0 versus H a :    0 in the case of normality,  k is equivalent to t  c. Thus, with c = t , n – 1, the likelihood ratio test is the one-sample t test. The likelihood ratio principle can also be applied when the X i ’s have different distributions and even when they are dependent, though the likelihood function can be complicated in such cases.

20 The Likelihood Ratio Principle Many of the test procedures to be presented in subsequent chapters are obtained from the likelihood ratio principle. These tests often turn out to minimize β among all tests that have the desired a, so are truly best tests. A practical limitation on the use of the likelihood ratio principle is that, to construct the likelihood ratio test statistic, the form of the probability distribution from which the sample comes must be specified. To derive the t test from the likelihood ratio principle, the investigator must assume a normal pdf.

21 The Likelihood Ratio Principle If an investigator is willing to assume that the distribution is symmetric but does not want to be specific about its exact form (such as normal, uniform, or Cauchy), then the principle fails because there is no way to write a joint pdf simultaneously valid for all symmetric distributions. we will present several distribution-free test procedures, so called because the probability of a type I error is controlled simultaneously for many different underlying distributions. These procedures are useful when the investigator has limited knowledge of the underlying distribution.