Hypothesis Testing

Outline The Null Hypothesis Type I and Type II Error Using Statistics to test the Null Hypothesis The Logic of Data Analysis

Research Questions and Hypotheses Non-directional: No stated expectation about outcome Example: Do men and women differ in terms of conversational memory? Hypothesis: Statement of expected relationship Directionality of relationship Women will have greater conversational memory than men

Grounding Hypotheses in Theory Hypotheses have an underlying rationale: Logical reasoning behind the direction of the hypotheses (theoretical rationale – explanation) Why do we expect women to have better conversational memory? Theoretical rationale based on: 1. Past research 2. Existing theory 3. Logical reasoning

The Null Hypothesis Null Hypothesis - the absence of a relationship E..g., There is no difference between men’s and women’s with regards to conversational memories Compare observed results to Null Hypothesis How different are the results from the null hypothesis? We do not propose a null hypothesis as research hypothesis - need very large sample size / power Used as point of contrast for testing

Hypotheses testing When we test observed results against null: We can make two decisions: 1. Accept the null No significant relationship Observed results similar to the Null Hypothesis 2. Reject the null Significant relationship Observed results different from the Null Hypothesis Whichever decision, we risk making an error

Type I and Type II Error 1. Type I Error 2. Type II Error Reality: No relationship Decision: Reject the null Believe your research hypothesis have received support when in fact you should have disconfirmed it Analogy: Find an innocent man guilty of a crime 2. Type II Error Reality: Relationship Decision: Accept the null Believe your research hypothesis has not received support when in fact you should have rejected the null. Analogy: Find a guilty man innocent of a crime

Potential outcomes of testing Decision Accept Null Reject Null R E No A Relationship L I Relationship T Y 1 2 3 4

Potential outcomes of testing Decision Accept Null Reject Null R E No A Relationship L I Relationship T Y Correct decision 2 3 4

Potential outcomes of testing Decision Accept Null Reject Null R E No A Relationship L I Relationship T Y 1 2 3 Correct decision

Potential outcomes of testing Decision Accept Null Reject Null R E No A Relationship L I Relationship T Y 1 Type I Error 3 4

Potential outcomes of testing Decision Accept Null Reject Null R E No A Relationship L I Relationship T Y 1 2 4 Type II Error

Potential outcomes of testing Decision Accept Null Reject Null R E No A Relationship L I Relationship T Y Correct decision Type I Error Correct decision Type II Error

Function of Statistical Tests Statistical tests determine: Accept or Reject the Null Hypothesis Based on probability of making a Type I error Observed results compared to the results expected by the Null Hypotheses What is the probability of getting observed results if Null Hypothesis were true? If results would occur less than 5% of the time by simple chance then we reject the Null Hypothesis

Start by setting level of risk of making a Type I Error How dangerous is it to make a Type I Error: What risk is acceptable?: 5%? 1%? .1%? Smaller percentages are more conservative in guarding against a Type I Error Level of acceptable risk is called “Significance level” : Usually the cutoff - <.05

Conventional Significance Levels .05 level (5% chance of Type I Error) .01 level (1% chance of Type I Error) .001 level (.1% chance of Type I Error) Rejecting the Null at the .05 level means: Taking a 5% risk of making a Type I Error

Steps in Hypothesis Testing 1. State research hypothesis 2. State null hypothesis 3.Set significance level (e.g., .05 level) 4. Observe results 5. Statistics calculate probability of results if null hypothesis were true 6. If probability of observed results is less than significance level, then reject the null

Guarding against Type I Error Significance level regulates Type I Error Conservative standards reduce Type I Error: .01 instead of .05, especially with large sample Reducing the probability of Type I Error: Increases the probability of Type II Error Sample size regulates Type II Error The larger the sample, the lower the probability of Type II Error occurring in conservative testing

Statistical Power The power to detect significant relationships The larger the sample size, the more power The larger the sample size, the lower the probability of Type II Error Power = 1 – probability of Type II Error

Statistical Analysis Statistical analysis: Examines observed data Calculates the probability that the results could occur by chance (I.e., if Null was true) Choice of statistical test depends on: Level of measurement of the variables in question: Nominal, Ordinal, Interval or Ratio

Logic of data analysis Univariate analysis Bivariate analysis One variable at a time (descriptive) Bivariate analysis Two variables at a time (testing relationships) Multivariate analysis More than two variables at a time (testing relationships and controlling for other variables)

Variables Dependent variable: Independent variables: What we are trying to predict E.g., Candidate preference Independent variables: What we are using as predictors E.g., Gender, Party affiliation

Testing hypothesis for two nominal variables Variables Null hypothesis Procedure Gender Passing is not Chi-square related to gender Pass/Fail

Testing hypothesis for one nominal and one ratio variable Variables Null hypothesis Procedure Gender Score is not T-test related to gender Test score

Testing hypothesis for one nominal and one ratio variable Variable Null hypothesis Procedure Year in school Score is not related to year in ANOVA school Test score Can be used when nominal variable has more than two categories and can include more than one independent variable

Testing hypothesis for two ratio variables Variable Null hypothesis Procedure Hours spent studying Score is not related to hours Correlation spent studying Test score

Testing hypothesis for more than two ratio variables Variable Null hypothesis Procedure Hours spent studying Score is not positively related to hours Classes spent studying and Multiple missed not negatively related regression to classes missed Test score

Commonality across all statistical analysis procedures Set the significance level: E.g., .05 level Means that we are willing to conclude that there is a relationship if: Chance of Type I error is less than 5% Statistical tests tell us whether: The observed relationship has less than a 5% chance of occurring by chance

Summary of Statistical Procedures Variables Procedure Nominal IV, Nominal DV Chi-square Nominal IV, Ratio DV T-test Multiple Nominal IVs, Ratio DV ANOVA Ratio IV, Ratio DV Pearson’s R Multiple Nominal IVs, Ratio DV with ratio covariates ANCOVA Multiple ratio Multiple Regression