Quantitative Methods – Week 7: Inductive Statistics II: Hypothesis Testing Roman Studer Nuffield College

Introduction In last week’s class on inductive statistics, we learned how to determine and interpret confidence intervals... with 0-1,96+1,96 =1=1 2,5% of cases 95% of cases A standard normal distribution is a normal distribution N(0,1) with mean  =0 and standard deviation  =1  Explain!

Introduction (II)  … where the t-value tells us how many standard errors either side of the mean we have to add to achieve a certain degree of confidence (90%, 95%, 99%)  E.g. 95% confidence level:  With confidence intervals, the implicit hypothesis was that the specific intervals contains a certain value  With hypothesis testing, the hypothesis is now made explicit; so we explicitly want to know whether… Can the result from one sample be considered different from the result of another sample? Example: Is the mean relief payment in Kent different from the one in Sussex? Is there really a connection between one variable and another one? Example: Is the regression coefficient on unemployment statistically significant and can therefore help to explain the voting share of the Nazi party?  Why do we actually need hypothesis testing ??

The 5 Steps of Hypothesis Testing 1.Specify the hypothesis in an appropriate form for statistical testing 2.Set a level of probability on the basis of which the hypothesis should be rejected 3.Select the relevant test statistic 4.Calculate the test statistic and compare this calculated value with the tabulated theoretical probability distribution in order to reach a decision about the hypothesis 5.Interpret the results of the decision

Step 1: Null and Alternative Hypothesis  For testing a hypothesis, the first step is set up a null hypothesis (H 0 ) that can be rejected  The null hypothesis is presumed to be true until the data strongly suggests otherwise (like a defendant on trial)  The alternative hypothesis H 1 specifies the opposite  Examples: H 0 : The defendant is innocent H 1 : The defendant is guilty H 0 : Unemployment rates had no impact on Nazi votes H 1 : Unemployment had an impact on Nazi votes H 0 : bUnemployment=0 H 1 : bUnemployment≠0

Step 2: The Level of Probability Type I and Type II error  In hypothesis testing we can make two kinds of mistakes  Type I error: Rejecting the null hypothesis when it is in fact true  Type II error: Failing to reject the null hypothesis when it is actually false What is the risk we are willing to take of making a Type I error? Statistical decision True state of nature Reject H 0 Do not reject H 0 H 0 is trueType I errorCorrect H 0 is falseCorrectType II error

Step 2: The Level of Probability (II)  The Significance level, a, is the probability of making a Type I error   A small probability of a type I error is preferred  To what extent we are willing to take a risk of making wrong conclusion? Common choices for a are… 10% 5% (most common) 1%  5% level means that we are taking a risk of being wrong five times per 100 trials  Trade-off: If we reduce the probability of a type I error, the probability for a type II error will increase

Step 3: The Test Statistic  What is the probability, that H 0 is true given the observed outcome?  For every sample statistic there is a corresponding sampling distribution  Test statistics and critical values in order to test a null hypothesis against an alternative  For the time being, the test statistic we are using is the t- statistic CAUTION!  Always remember the underlying assumptions when testing: a)The distribution is approximately normal b)Our sample doesn’t suffer from a serious sample bias

Step 4: Calculate and Assess the Test Statistic Test statistic for a regression coefficient  If we could repeat the “social” experiment, we would obtain different values for the error term and consequently for the dependent variable  impact on the slope coefficient  How reliable is it to conclude that there is a relationship? Example: y 1 st sample 2 nd sample x b<b

Step 4: Calculate and Assess the Test Statistic (II)  Is b significantly different from 0?  H 0 : b=0   t-statistics  The nominator b is the regression coefficient derived from the OLS method  The denominator corresponds to the estimated standard deviation of the sampling distribution of the coefficient, i.e. the SE(b)

Step 4: Calculate and Assess the Test Statistic (III) Statistical significance using the t-statistics  The t-statistic indicates how many standard deviations the sample regression coefficient is from 0 (we can also express it as deviation from some certain value of interest if we want)  Central Limit Theorem applies Shape of the sampling distribution becomes normal With increasing sample size the t-distribution approximates a standard normal distribution  If  t-value  >1.96, the estimated regression coefficient is more than 1.96 standard deviations from 0 and the probability for such an outcome is less than 5%, if H0 is true

Step 4: Calculate and Assess the Test Statistic (IV) Sampling distribution and critical value % of cases 95% of cases Reject H 0 Accept H 0 Confidence level  =0.95

Step 4: Calculate and Assess the Test Statistic (V) Significance level and p-level  P-value is the probability that the outcome observed would be present if the null hypothesis is true  Small p-values are evidence against H 0  P-value<a  Reject H 0, accept H 1  Failing to reject the null hypothesis does not necessarily constitute support for H0; it just means that the data or pattern is not sufficiently strong to reject H0

Step 4: Calculate and Assess the Test Statistic (VI) One- and two-tailed tests  Two-tailed test: The parameter value is calculated for both tails of the sampling distribution   The critical region is divided equally between the left- and the right-hand tails  If the hypothesis is about the directions:  One-tailed test: H 0 : μ 1 = μ 2 ; H 1 : μ 1 > μ 2  Critical region is in the left- or the right-hand tail   a one-tailed test increases the critical region at one tail  Strong a priori reasons must exist!

Step 5: Interpreting the Results Statistical versus historical significance  Statistical significance refers to the probability of type I error (rejecting the null hypothesis when it is in fact true)  Statistical significance is influenced by magnitude of the parameter magnitude of the standard error, i.e. sample size (because the standard error decreases with N everything else being equal)  Historical significance: What is the practical significance of rejecting the hypothesis?  The aim is to find statistically significant results that are relevant from a historical perspective.

Repetition & Hypothesis Testing Computer Class:

Exercises Relief dataset Get the “Relief” dataset athttp:// Note: In the column “county”, 1 stands for Kent and 2 stands for Sussex. Look at the dataset: what is the basic unit of measurement here? Is it a time-series, a cross-sectional or a panel dataset? Look at the variables relief and unemployment (unemp) Get a first overview by visualising the data: Are the variables approximately normally distributed? Compute the mean, median, standard deviation, coefficient of variation, kurtosis and skewness for the variables Are the level of relief and the level of employment statistically different in Kent and Sussex? (Hint: create new variables and test for difference of means.) What is the null hypothesis? Are the means different at a 95% significance level? Work again with the whole dataset, not just with Kent and Sussex. Compute the correlation coefficient between unemployment and the relief payments. How do you interpret the result? Is the correlation coefficient significant at a 95% and at a 99% level? State the null hypothesis, and test it Estimate a regression using relief and unemployment. Which one is the dependent, which one the independent variable? Report the results of the regression (a, b, R 2 ). Is the regression coefficient (b) statistically significant at a 95% level? Formulate the null hypothesis and test it. What is the critical t- value shown in the t-statistics table? What is the actual t-value of b? What is the p-value? What does all that mean for the interpretation of the regression results?

Homework  Readings: Feinstein & Thomas, Ch. 8 & 9  Problem Set 5:  Finish the exercises from today’s computer class if you haven’t done so already. Include all the results and answers in the file you send me.  Do exercise 1 from Feinstein & Thomas, p. 181.