1. Hypothesis Testing

2. Outline The Null Hypothesis The Null Hypothesis Type I and Type II Error Type I and Type II Error Using Statistics to test the Null Hypothesis Using Statistics to test the Null Hypothesis The Logic of Data Analysis The Logic of Data Analysis

3. Research Questions and Hypotheses Research question: Research question: Non-directional: Non-directional: No stated expectation about outcome No stated expectation about outcome Example: Example: Do men and women differ in terms of conversational memory? Do men and women differ in terms of conversational memory? Hypothesis: Hypothesis: Statement of expected relationship Statement of expected relationship Directionality of relationship Directionality of relationship Example: Example: Women will have greater conversational memory than men Women will have greater conversational memory than men

4. Grounding Hypotheses in Theory Hypotheses have an underlying rationale: Hypotheses have an underlying rationale: Logical reasoning behind the direction of the hypotheses (theoretical rationale – explanation) Logical reasoning behind the direction of the hypotheses (theoretical rationale – explanation) Why do we expect women to have better conversational memory? Why do we expect women to have better conversational memory? Theoretical rationale based on: Theoretical rationale based on: 1. Past research 1. Past research 2. Existing theory 2. Existing theory 3. Logical reasoning 3. Logical reasoning

5. The Null Hypothesis Null Hypothesis - the absence of a relationship Null Hypothesis - the absence of a relationship E..g., There is no difference between men’s and women’s with regards to conversational memories E..g., There is no difference between men’s and women’s with regards to conversational memories Compare observed results to Null Hypothesis Compare observed results to Null Hypothesis How different are the results from the null hypothesis? How different are the results from the null hypothesis? We do not propose a null hypothesis as research hypothesis - need very large sample size / power We do not propose a null hypothesis as research hypothesis - need very large sample size / power Used as point of contrast for testing Used as point of contrast for testing

6. Hypotheses testing When we test observed results against null: When we test observed results against null: We can make two decisions: We can make two decisions: 1. Accept the null 1. Accept the null No significant relationship No significant relationship Observed results similar to the Null Hypothesis Observed results similar to the Null Hypothesis 2. Reject the null 2. Reject the null Significant relationship Significant relationship Observed results different from the Null Hypothesis Observed results different from the Null Hypothesis Whichever decision, we risk making an error Whichever decision, we risk making an error

7. Type I and Type II Error 1. Type I Error 1. Type I Error Reality: No relationship Reality: No relationship Decision: Reject the null Decision: Reject the null Believe your research hypothesis have received support when in fact you should have disconfirmed it Believe your research hypothesis have received support when in fact you should have disconfirmed it Analogy: Find an innocent man guilty of a crime Analogy: Find an innocent man guilty of a crime 2. Type II Error 2. Type II Error Reality: Relationship Reality: Relationship Decision: Accept the null Decision: Accept the null Believe your research hypothesis has not received support when in fact you should have rejected the null. Believe your research hypothesis has not received support when in fact you should have rejected the null. Analogy: Find a guilty man innocent of a crime Analogy: Find a guilty man innocent of a crime

8. Potential outcomes of testing Decision Accept NullReject Null R ENo ARelationship L IRelationship TY 12 34

9. Potential outcomes of testing Decision Accept NullReject Null R ENo ARelationship L IRelationship TY Correct decision 2 34

10. Potential outcomes of testing Decision Accept NullReject Null R ENo ARelationship L IRelationship TY 12 3 Correct decision

11. Potential outcomes of testing Decision Accept NullReject Null R ENo ARelationship L IRelationship TY 1 Type I Error 34

12. Potential outcomes of testing Decision Accept NullReject Null R ENo ARelationship L IRelationship TY 12 Type II Error 4

13. Potential outcomes of testing Decision Accept NullReject Null R ENo ARelationship L IRelationship TY Type II Error Correct decision Type I Error Correct decision

14. Function of Statistical Tests Statistical tests determine: Statistical tests determine: Accept or Reject the Null Hypothesis Accept or Reject the Null Hypothesis Based on probability of making a Type I error Based on probability of making a Type I error Observed results compared to the results expected by the Null Hypotheses Observed results compared to the results expected by the Null Hypotheses What is the probability of getting observed results if Null Hypothesis were true? What is the probability of getting observed results if Null Hypothesis were true? If results would occur less than 5% of the time by simple chance then we reject the Null Hypothesis If results would occur less than 5% of the time by simple chance then we reject the Null Hypothesis

15. Start by setting level of risk of making a Type I Error How dangerous is it to make a Type I Error: How dangerous is it to make a Type I Error: What risk is acceptable?: What risk is acceptable?: 5%? 5%? 1%? 1%?.1%?.1%? Smaller percentages are more conservative in guarding against a Type I Error Smaller percentages are more conservative in guarding against a Type I Error Level of acceptable risk is called “Significance level” : Level of acceptable risk is called “Significance level” : Usually the cutoff - <.05 Usually the cutoff - <.05

16. Conventional Significance Levels.05 level (5% chance of Type I Error).05 level (5% chance of Type I Error).01 level (1% chance of Type I Error).01 level (1% chance of Type I Error).001 level (.1% chance of Type I Error).001 level (.1% chance of Type I Error) Rejecting the Null at the.05 level means: Rejecting the Null at the.05 level means: Taking a 5% risk of making a Type I Error Taking a 5% risk of making a Type I Error

17. Steps in Hypothesis Testing 1. State research hypothesis 1. State research hypothesis 2. State null hypothesis 2. State null hypothesis 3.Set significance level (e.g.,.05 level) 3.Set significance level (e.g.,.05 level) 4. Observe results 4. Observe results 5. Statistics calculate probability of results if null hypothesis were true 5. Statistics calculate probability of results if null hypothesis were true 6. If probability of observed results is less than significance level, then reject the null 6. If probability of observed results is less than significance level, then reject the null

18. Guarding against Type I Error Significance level regulates Type I Error Significance level regulates Type I Error Conservative standards reduce Type I Error: Conservative standards reduce Type I Error:.01 instead of.05, especially with large sample.01 instead of.05, especially with large sample Reducing the probability of Type I Error: Reducing the probability of Type I Error: Increases the probability of Type II Error Increases the probability of Type II Error Sample size regulates Type II Error Sample size regulates Type II Error The larger the sample, the lower the probability of Type II Error occurring in conservative testing The larger the sample, the lower the probability of Type II Error occurring in conservative testing

19. Statistical Power The power to detect significant relationships The power to detect significant relationships The larger the sample size, the more power The larger the sample size, the more power The larger the sample size, the lower the probability of Type II Error The larger the sample size, the lower the probability of Type II Error Power = 1 – probability of Type II Error Power = 1 – probability of Type II Error

20. Statistical Analysis Statistical analysis: Statistical analysis: Examines observed data Examines observed data Calculates the probability that the results could occur by chance (I.e., if Null was true) Calculates the probability that the results could occur by chance (I.e., if Null was true) Choice of statistical test depends on: Choice of statistical test depends on: Level of measurement of the variables in question: Level of measurement of the variables in question: Nominal, Ordinal, Interval or Ratio Nominal, Ordinal, Interval or Ratio

21. Logic of data analysis Univariate analysis Univariate analysis One variable at a time (descriptive) One variable at a time (descriptive) Bivariate analysis Bivariate analysis Two variables at a time (testing relationships) Two variables at a time (testing relationships) Multivariate analysis Multivariate analysis More than two variables at a time (testing relationships and controlling for other variables) More than two variables at a time (testing relationships and controlling for other variables)

22. Variables Dependent variable: Dependent variable: What we are trying to predict What we are trying to predict E.g., Candidate preference E.g., Candidate preference Independent variables: Independent variables: What we are using as predictors What we are using as predictors E.g., Gender, Party affiliation E.g., Gender, Party affiliation

23. Testing hypothesis for two nominal variables Variables Null hypothesisProcedure Gender Passing is not Chi-square related to gender Pass/Fail

24. Testing hypothesis for one nominal and one ratio variable VariablesNull hypothesisProcedure Gender Score is notT-test related to gender Test score

25. Testing hypothesis for one nominal and one ratio variable VariableNull hypothesisProcedure Year in school Score is not related to year inANOVA school Test score Can be used when nominal variable has more than two categories and can include more than one independent variable Can be used when nominal variable has more than two categories and can include more than one independent variable

26. Testing hypothesis for two ratio variables VariableNull hypothesisProcedure Hours spent studyingScore is not related to hours Correlation related to hours Correlation spent studying spent studying Test score

27. Testing hypothesis for more than two ratio variables VariableNull hypothesisProcedure Hours spent studying Score is positively related to hours related to hours Classes spent studying and Multiple missed negatively related regression to classes missed to classes missed Test score

28. Commonality across all statistical analysis procedures Set the significance level: Set the significance level: E.g.,.05 level E.g.,.05 level Means that we are willing to conclude that there is a relationship if: Means that we are willing to conclude that there is a relationship if: Chance of Type I error is less than 5% Chance of Type I error is less than 5% Statistical tests tell us whether: Statistical tests tell us whether: The observed relationship has less than a 5% chance of occurring by chance The observed relationship has less than a 5% chance of occurring by chance

29. Summary of Statistical Procedures VariablesProcedure Nominal IV, Nominal DVChi-square Nominal IV, Ratio DVT-test Multiple Nominal IVs, Ratio DV ANOVA Ratio IV, Ratio DVPearson’s R Multiple Nominal IVs, Ratio DV with ratio covariates ANCOVA Multiple ratioMultiple Regression