1. Hypothesis testing and effect sizes Sylvain Chartier Laboratory for Computational Neurodynamics and Cognition Centre for Neural Dynamics

2. Example Preliminary test of a virtual reality (VR) anxiety-provoking tool using a sample of participants with obsessive-compulsive disorder (OCD). In order to be considered OCD, a difference of 1 SD (35) must be observed.

3. Example The results in this study suggest that the anxiety of those with OCD is higher than that of healthy controls in the VR. As shown in Table 1, the OCD group and the control group presented a difference in checking time. OCD characteristics can account for these differences in behavioral and anxiety. In conclusion, the results suggest that VR is valuable method for anxiety provocation.

4. Beliefs in the H0 The data are the result of chance (i.e. that H0 is true); Doing a type I error if H0 is rejected (i.e. rejecting H0 when it should not have been rejected); That an experimental replication produces statistically significant results (by calculating 1-p); That the decision (to reject H0 or not) is correct; To obtain results as extreme as these if H0 is true

5. Probability of what? p(D|H) = p(H|D) The probability of being a female given that I am pregnant Observed  The probability of being pregnant given that I am a female

6. H0=? Hypothesis testing H0 H1 H0:  = 0 H1:  0 H0 has a probability of 1/  of being true -> 0 H1 has a probability of (  -1)/  of being true -> 1 We will always reject H0! Since we cannot prove H1, we will try to refute H0.

7. What to do? Treatment Effect Which hypotheses would you reject? B C D E F For example, B: -Confidence interval does not cross zero. -So the results for that experiment are statistically significant  <0.05. -We have substantial evidence that the difference is not really zero. A B C D E F 0 Confidence interval ?

8. What to do? Treatment Effect Which hypotheses would you reject? C D Which hypotheses would you consider equivalent? A B Which hypotheses would you consider ambiguous? E F A B C D E F 0 Zone of scientific or clinical indifference Confidence interval + equivalence

9. Example Preliminary test of a virtual reality anxiety-provoking tool using a sample of participants with obsessive-compulsive disorder (OCD). In order to be considered OCD, a difference of 1 SD (35) must be observed.

10. Example 0 -1 SD +1SD A confidence interval of 95% will gives the following bounds: [16.82 à 74.99] What about the effect size?

11. Confidence intervals around effect sizes Getting confidence intervals on means, means difference, Z-scores, standard deviation, regression coefficients, etc. is quite simple. However, things are not that simple for standardized effect sizes and bounded effect sizes (coefficient of correlation, R2, proportion, etc.). We have to use numerical methods.

12. What about nonpivotal quantities? Pivotal quantity: is a function of observations and unobservable parameters whose probability distribution does not depend on unknown parameters. Examples: Mean, mean difference, Z-score, standard deviation, bivariate correlation, regression coefficient, etc. Nonpivotal quantity: is a function of observations and unobservable parameters whose probability distribution depend on unknown parameters. Examples: standardized effect sizes (standardized mean differences, standardized regression coefficients, coefficients of variation, etc.) and effect sizes that are bounded (correlation coefficients, squared multiple correlation coefficients, proportions, etc.). Confidence intervals for such effects cannot be obtained by inverting their corresponding test statistic (like in pivotal quantities).

13. Confidence intervals around effect sizes Effect size of the study is 0.81, which is considered as high according to Cohen. However, if we compute the confidence interval we get : [0.279; 1.31] In other words, we have no idea about the real utility of VR to elicit compulsive behaviors.

14. Conclusion If we asked to a runner how much time it takes him to complete 42 km? And its answer would be between 3 to 15 hours. We would be skeptical, then why don’t we use the same skepticism about our statistical analysis?