Lecture 14: Thur., Feb. 26 Multiple Comparisons (Sections 6.3-6.4) Next class: Inferences about Linear Combinations of Group Means (Section 6.2).

Slides:



Advertisements
Similar presentations
Lecture 3 Outline: Thurs, Sept 11 Chapters Probability model for 2-group randomized experiment Randomization test p-value Probability model for.
Advertisements

Lecture 6 Outline – Thur. Jan. 29
T-tests Computing a t-test  the t statistic  the t distribution Measures of Effect Size  Confidence Intervals  Cohen’s d.
Chapter Seventeen HYPOTHESIS TESTING
Class 23: Thursday, Dec. 2nd Today: One-way analysis of variance, multiple comparisons. Next week: Two-way analysis of variance. I will the final.
Lecture 13 – Tues, Oct 21 Comparisons Among Several Groups – Introduction (Case Study 5.1.1) Comparing Any Two of the Several Means (Chapter 5.2) The One-Way.
Stat 512 – Lecture 14 Analysis of Variance (Ch. 12)
Behavioural Science II Week 1, Semester 2, 2002
Stat 112: Lecture 22 Notes Chapter 9.1: One-way Analysis of Variance. Chapter 9.3: Two-way Analysis of Variance Homework 6 is due on Friday.
Lecture 15: Tues., Mar. 2 Inferences about Linear Combinations of Group Means (Chapter 6.2) Chi-squared test (Handout/Notes) Thursday: Simple Linear Regression.
Lecture 5 Outline – Tues., Jan. 27 Miscellanea from Lecture 4 Case Study Chapter 2.2 –Probability model for random sampling (see also chapter 1.4.1)
9-1 Hypothesis Testing Statistical Hypotheses Statistical hypothesis testing and confidence interval estimation of parameters are the fundamental.
ANOVA Determining Which Means Differ in Single Factor Models Determining Which Means Differ in Single Factor Models.
Lecture 14 – Thurs, Oct 23 Multiple Comparisons (Sections 6.3, 6.4). Next time: Simple linear regression (Sections )
Lecture 11 Psyc 300A. Null Hypothesis Testing Null hypothesis: the statistical hypothesis that there is no relationship between the variables you are.
Class 22: Tuesday, Nov. 30th Today: One-way analysis of variance I will you tonight or tomorrow morning with comments on your project. Schedule:
Lecture 7 Outline – Thur, Sep 25
Lecture 6 Outline: Tue, Sept 23 Review chapter 2.2 –Confidence Intervals Chapter 2.3 –Case Study –Two sample t-test –Confidence Intervals Testing.
Lecture 19: Tues., Nov. 11th R-squared (8.6.1) Review
Lecture 5 Outline: Thu, Sept 18 Announcement: No office hours on Tuesday, Sept. 23rd after class. Extra office hour: Tuesday, Sept. 23rd from 12-1 p.m.
Lecture 24: Thurs. Dec. 4 Extra sum of squares F-tests (10.3) R-squared statistic (10.4.1) Residual plots (11.2) Influential observations (11.3,
Lecture 9: One Way ANOVA Between Subjects
One-way Between Groups Analysis of Variance
Lecture 12 One-way Analysis of Variance (Chapter 15.2)
Lecture 9 Today: –Log transformation: interpretation for population inference (3.5) –Rank sum test (4.2) –Wilcoxon signed-rank test (4.4.2) Thursday: –Welch’s.
The Analysis of Variance
Chapter 9 Hypothesis Testing.
Today Concepts underlying inferential statistics
Stat 112: Lecture 21 Notes Model Building (Brief Discussion) Chapter 9.1: One way Analysis of Variance. Homework 6 is due Friday, Dec. 1 st. I will be.
5-3 Inference on the Means of Two Populations, Variances Unknown
Lecture 13: Tues., Feb. 24 Comparisons Among Several Groups – Introduction (Case Study 5.1.1) Comparing Any Two of the Several Means (Chapter 5.2) The.
“There are three types of lies: Lies, Damn Lies and Statistics” - Mark Twain.
Chapter 12: Analysis of Variance
Intermediate Applied Statistics STAT 460
Section 10.1 ~ t Distribution for Inferences about a Mean Introduction to Probability and Statistics Ms. Young.
1 Level of Significance α is a predetermined value by convention usually 0.05 α = 0.05 corresponds to the 95% confidence level We are accepting the risk.
+ Chapter 9 Summary. + Section 9.1 Significance Tests: The Basics After this section, you should be able to… STATE correct hypotheses for a significance.
Comparing Two Population Means
One-Way Analysis of Variance Comparing means of more than 2 independent samples 1.
Week 8 Chapter 8 - Hypothesis Testing I: The One-Sample Case.
CHAPTER 18: Inference about a Population Mean
9-1 Hypothesis Testing Statistical Hypotheses Definition Statistical hypothesis testing and confidence interval estimation of parameters are.
Learning Objectives In this chapter you will learn about the t-test and its distribution t-test for related samples t-test for independent samples hypothesis.
January 31 and February 3,  Some formulae are presented in this lecture to provide the general mathematical background to the topic or to demonstrate.
t(ea) for Two: Test between the Means of Different Groups When you want to know if there is a ‘difference’ between the two groups in the mean Use “t-test”.
Statistics 11 Confidence Interval Suppose you have a sample from a population You know the sample mean is an unbiased estimate of population mean Question:
Essential Statistics Chapter 131 Introduction to Inference.
CHAPTER 14 Introduction to Inference BPS - 5TH ED.CHAPTER 14 1.
Section Copyright © 2014, 2012, 2010 Pearson Education, Inc. Lecture Slides Elementary Statistics Twelfth Edition and the Triola Statistics Series.
Chapter 10: Analyzing Experimental Data Inferential statistics are used to determine whether the independent variable had an effect on the dependent variance.
Confidence intervals and hypothesis testing Petter Mostad
Educational Research Chapter 13 Inferential Statistics Gay, Mills, and Airasian 10 th Edition.
Interval Estimation and Hypothesis Testing Prepared by Vera Tabakova, East Carolina University.
ANOVA: Analysis of Variance.
One-way ANOVA: - Comparing the means IPS chapter 12.2 © 2006 W.H. Freeman and Company.
Ch11: Comparing 2 Samples 11.1: INTRO: This chapter deals with analyzing continuous measurements. Later, some experimental design ideas will be introduced.
Ex St 801 Statistical Methods Inference about a Single Population Mean.
Marshall University School of Medicine Department of Biochemistry and Microbiology BMS 617 Lecture 13: One-way ANOVA Marshall University Genomics Core.
Lecture PowerPoint Slides Basic Practice of Statistics 7 th Edition.
Stat 112: Lecture 22 Notes Chapter 9.1: One Way Analysis of Variance Chapter 9.2: Two Way Analysis of Variance.
One-way ANOVA Example Analysis of Variance Hypotheses Model & Assumptions Analysis of Variance Multiple Comparisons Checking Assumptions.
+ Unit 6: Comparing Two Populations or Groups Section 10.2 Comparing Two Means.
Review: Stages in Research Process Formulate Problem Determine Research Design Determine Data Collection Method Design Data Collection Forms Design Sample.
Topic 22: Inference. Outline Review One-way ANOVA Inference for means Differences in cell means Contrasts.
Hypothesis Tests. An Hypothesis is a guess about a situation that can be tested, and the test outcome can be either true or false. –The Null Hypothesis.
Statistical Inference for the Mean Objectives: (Chapter 8&9, DeCoursey) -To understand the terms variance and standard error of a sample mean, Null Hypothesis,
Educational Research Inferential Statistics Chapter th Chapter 12- 8th Gay and Airasian.
Class Seven Turn In: Chapter 18: 32, 34, 36 Chapter 19: 26, 34, 44 Quiz 3 For Class Eight: Chapter 20: 18, 20, 24 Chapter 22: 34, 36 Read Chapters 23 &
Statistical Decision Making. Almost all problems in statistics can be formulated as a problem of making a decision. That is given some data observed from.
More About Tests Notes from
Presentation transcript:

Lecture 14: Thur., Feb. 26 Multiple Comparisons (Sections ) Next class: Inferences about Linear Combinations of Group Means (Section 6.2).

Discrimination against the Handicapped (Case Study 6.1) Study of how physical handicaps affect people’s perception of employment qualifications. Researchers prepared five videotaped job interviews, using same two male actors for each. Tapes differed only in that applicant appeared with a different handicap in each– (i) wheelchair; (ii) on crutches; (iii) hearing impaired; (iv) one leg amputated; (v) no handicap. Each tape shown to 14 students from U.S. university. Students rate qualifications of candidate on 0 to 10 point scale based on tape. Questions of interest: Do subjects systematically evaluate qualifications differently according to candidate’s handicap? If so, which handicaps produce different evaluations?

Multiple Comparisons Simultion In multiplecomp.JMP, 20 groups are compared with sample sizes of ten for each group. The observations for each group are simulated from a standard normal distribution. Thus, in fact, Number of pairs found to have significantly different means using t-test at level Iterat ion # of Pairs 3

Compound Uncertainty Compound uncertainty: When drawing more than one direct inference, there is an increased chance of making at least one mistake. Impact on tests: If using a conventional criteria such as a p-value of 0.05 to reject a null hypothesis, the probability of falsely rejecting at least one null hypothesis will be greater than 0.05 if considering multiple tests. Impact on confidence intervals: If forming multiple 95% confidence intervals, the chance that all of the confidence intervals will contain true parameter is less than 95%.

Simultaneous Inferences When several tests are considered simultaneously, they constitute a family of tests. Individual Type I error rate: Probability for a single test that the null hypothesis will be rejected assuming that the null hypothesis is true. Familywise Type I error rate: Probability for a family of test that at least one null hypothesis will be rejected assuming that all of the null hypotheses are true.

Individual vs. Family Confidence Levels If a family consists of k tests, each with individual type I error rate 0.05, the familywise type I error rate is at least 0.05 and no larger than k*0.05. Actual familywise type I error rate depends on degree of dependence between tests. If the tests are independent, the familywise type I error rate is 1-(.95) k If all the null hypotheses in a family true, the mean number of Type I errors equals k*0.05.

Familywise Type I error rates KUpper BoundFamilywise Error Rate if independent

Multiple Comparison Procedures Multiple comparison procedures are methods of carrying out tests so that the familywise type I error rate is controlled (at 0.05 for example). Key issue: What is the appropriate family to consider?

Planned vs. Unplanned Comparisons Consider one-way layout with 20 groups. Planned Comparisons: researcher is specifically interested in comparing groups 1 and 4 because comparison answers a research question directly. This is a planned comparison. In the mice diets example, the researchers had five planned comparisons. Unplanned Comparisons: researcher examines all possible pairs of groups – 190 groups. As a result, researcher finds that only groups 5 and 8 suggest actual differences. Only this pair is reported as significant. For the handicap example, the comparisons were unplanned.

Families in Planned/Unplanned Planned Comparisons: The family of tests is the family of all planned comparisons (e.g., the family of five planned comparisons in mice diet). For small number of planned comparisons, it is usual practice to just use individual type I error rates. Unplanned Comparisons: The family of tests is the family of all possible comparisons - (k*(k-1)/2) for a k-group one-way layout. It is important to control the familywise type I error rate for unplanned comparisons. The handicap study involves unplanned comparisons.

Multiple Comparisons Procedures Consider testing vs. as part of a family of tests. t-statistic: Test to control individual Type I error rate for at level : Reject if Multiple Comparisons procedure to control familywise type I error rate at level : Higher critical value for rejecting We consider two multiple comparison procedures: (i) Tukey-Kramer; (ii) Bonferroni

Tukey-Kramer Based on computing the distribution of the largest |t| statistic under the null hypothesis that all group means are equal. For testing vs., reject if where q* depends on familywise type I error rate, I and n-I. For handicap study, q*=2.81 for familywise type I error rate=0.05 whereas as cutoff for individual type I error rate of 0.05 is

Tukey-Kramer in JMP To see which groups are significantly different (in sense of statistical significance) at a familywise Type I error rate of 0.05, click Compare Means under Oneway Analysis (after Analyze, Fit Y by X) and click All Pairs, Tukey’s HSD. In table “Comparison of All Pairs Using Tukey’s HSD,” two groups are significantly different if and only if the entry in the table for the pair of groups is positive. The cutoff value q* is listed in the output.

Bonferroni Method General method for doing multiple comparisons for any family of k tests. Denote familywise type I error rate we want by p*. Compute p-values for each individual test -- Reject null hypothesis for ith test if Guarantees that familywise type I error rate is at most p*.

Bonferroni for mice diets Five comparisons were planned. Suppose we want the familywise error rate for the five comparisons to be Bonferroni method: We should consider two groups to be significantly different if the p-value from the two-sided t-test is less than 0.05/5=0.01. Bonferroni in JMP: To do each test at a given level, after Fit Y by X, click red triangle next to Oneway Analysis and click Set Alpha Level. Then click Compare Means and Each Pair, Student’s t. This will show results of tests at chosen alpha level.

Multiple Comparisons and Confidence Intervals When several 95% confidence intervals are considered simultaneously, they constitute a family of confidence intervals Individual Confidence Level: Success rate of a procedure for constructing a single confidence interval. Familywise Confidence Level: Success rate of procedure for constructing a family of confidence intervals, where a “successful” usage is one in which all intervals in the family capture their parameters.

Multiple Comparison Procedures Confidence Interval: Estimate Margin of Error. Margin of Error = (Multiplier)x(Standard Error of Estimate). For individual confidence level of 95%, multiplier is about 2. For familywise confidence level of 95%, the multiplier is greater than 2. Family of confidence intervals for all group mean differences that has 95% familywise confidence level: can be found on Table A.5. For df=n-I, use closest df > n-I on chart.

Multiplicity A news report says, “A 15 year study of more than 45,000 Swedish solidiers revealed that heavy users of marijuana were six times more likely than nonusers to develop schizophrenia.” Were the investigators only looking for difference in schizophrenia among heavy/non-heavy users of marijuana? Key question: What is their family of tests? If they were actually looking for a difference among 100 outcomes (e.g., blood pressure, lung cancer), Bonferroni should be used to control the familywise Type I error rate, i.e., only consider a difference significant if p-value is less than.05/100= The best way to deal with the multiple comparisons problem is to design a study to search specifically for a pattern that was suggested by an exploratory data analysis. In other words we convert an “unplanned” comparison into a “planned” comparison by doing a new experiment.