1. Pairwise comparisons: Confidence intervals Multiple comparisons Marina Bogomolov and Gili Baumer

2. Simulation Model: 6 independent groups of size 16, each sample comes from normal distribution with mean 0 and variance 10. The data is generated 100 times. For each of 100 iterations, 15 confidence intervals (confidence level 95%) for the pairwise comparisons are constructed. Assume we are interested only in one comparison. We look at the 100 confidence intervals in our simulation. What is the expected number of confidence intervals that do not cover 0? – What will happen to the proportion of confidence intervals not covering 0 when we will increase the number of iterations from 100 to 1000?

3. Results for 100 iterations, a single comparison How many intervals do not cover 0? What is the estimated confidence level?

4. Recall that we have 15 comparisons 0.06 0.05 0.04 0.03 0.03 0.06 0.05 0.07 0.06 0.03 0.03 0.05 0.02 0.03 0.02. What are these proportions? What does the oval show?

5. 15 comparisons In this experiment, what do you think is the number of confidence intervals which do not cover 0? What does this number correspond to in terms of hypothesis testing? On the average across all the experiments, what do you think is the number of confidence intervals which do not cover 0? What is its theoretical value?

6. 15 comparisons In this experiment, what do you think is the number of confidence intervals which do not cover 0? 3 What does this number correspond to in terms of hypothesis testing? 3 Type I errors. On the average across all the experiments, what do you think is the number of confidence intervals which do not cover 0? 0.66 What is its theoretical value? 15*0.05=0.75

7. 10000 iterations 0.0471 [2,] 0.0527 [3,] 0.0494 [4,] 0.0472 [5,] 0.0496 [6,] 0.0468 [7,] 0.0495 [8,] 0.0472 [9,] 0.0483 [10,] 0.0478 [11,] 0.0505 [12,] 0.0508 [13,] 0.0497 [14,] 0.0511 [15,] 0.0493 What are these values? For this experiment, the number of confidence intervals not covering 0 is 6  the number of false discoveries is 6. On the average across all the experiments, the number of confidence intervals not covering 0 : 0.737 Theoretical value: 0.75 What is it in terms of hypothesis testing?

8. 10000 iterations 0.0471 [2,] 0.0527 [3,] 0.0494 [4,] 0.0472 [5,] 0.0496 [6,] 0.0468 [7,] 0.0495 [8,] 0.0472 [9,] 0.0483 [10,] 0.0478 [11,] 0.0505 [12,] 0.0508 [13,] 0.0497 [14,] 0.0511 [15,] 0.0493 What are these values? On the average across all the experiments, the number of confidence intervals not covering 0 : 0.737 Theoretical value: 0.75 What is it in terms of hypothesis testing? If we would like to estimate the probability that at least one confidence interval (among the 15) does not cover 0, how do we estimate it using the simulation? What is the name of this probability for the family of pairwise comparison tests? Give a guess for this probability in case we are interested only in 5, 10 specific comparisons. What is your guess for the case all the 15 comparisons are of interest? What do you think happens to this probability when the number of interesting comparisons increases? For all 15 comparisons, will this probability be lower or higher than 0.75?

9. 10000 iterations 0.0471 [2,] 0.0527 [3,] 0.0494 [4,] 0.0472 [5,] 0.0496 [6,] 0.0468 [7,] 0.0495 [8,] 0.0472 [9,] 0.0483 [10,] 0.0478 [11,] 0.0505 [12,] 0.0508 [13,] 0.0497 [14,] 0.0511 [15,] 0.0493 What are these values? On the average across all the experiments, the number of confidence intervals not covering 0 : 0.737 Theoretical value: 0.75 What is it in terms of hypothesis testing? What is the name of this probability for the family of pairwise comparison tests? Give a guess for this probability in case we are interested in 5, 10, 15 specific comparisons. Estimated probabilities: 0.1803, 0.2861, 0.3554 If we would ignore the dependency, and calculate these probabilities under independence: 0.2262, 0.4013, 0.5367.