1. Statistical and Practical Significance
Advanced Statistics
Petr Soukup

2. Outline Reminder of statistical significance
Limits of statistical significance
Misuses of statistical significance
Alternatives to statistical significance
Practical significance
Effect sizes

3. REMINDER OF STATISTICAL SIGNIFICANCE (NHST)

4. Hypotheses and tests Tested hypothesis in experiments (Fisher, 1925)
Null and alternative hypothesis (NHST) (Neyman&Pearson, 1937)
Common tests - t-tests, analysis of variance, analysis of covariance, correlation analysis etc.

5. Definition of statistical significance
Decision
True status H0 H1
OK (P=1- α)
Type I error (P= α)
Type II error (P= β)
OK (P= 1-β) Test Power
Definition: Conditional probability, that our sample can
be drawn from population in which null hypothesis
is valid (α). Statistical significance is P(D/H0) and not P(H0/D)

6. LIMITS OF NHST

7. Assumptions for classical NHST
Big big probability samples from infinite or very big finite populations
Three assumptions:
Big (infinite) population (at least 100times bigger than the sample)
Probability sampling (all units same probability of selection)
Big sample (> units)

8. LIMITS OF NHST 1. data from censuses
2. data from non-probability samples
3. data from small samples
4. data based on sample that are big proportion of the basic population
5. big data samples from merged (internationally or by time) files

9. Beyond the limits of NHST in CSR*
N=32 articles, Czech sociological review (selected 29 issues), own research
*CSR-Czech sociological review

10. MISUSES OF NHST

11. Objections against NHST (Misuses of NHST)
a) Insufficient statement about population,
b) null hypotheses are unreal (nill null),
c) mechanical usage of classical 5% statistical significance (asterisks, stepwise methods, best models etc.),
d) statistical significant doesn’t mean important,
e) publishing only statistical significant results (file drawer problem).

12. Misuses of NHST in CSR* *CSR-Czech sociological review
N=32 articles, Czech sociological review (selected 29 issues), own research
*CSR-Czech sociological review

13. Conference examples (P<0,01)

14. Conference examples (***)

15. Conference examples (*** and stepwise)

16. ALTERNATIVES TO NHST

17. Some alternatives to statistical significance
a) Confidence Intervals (Problems for r, formulas, regression etc.)
b) Test power (quite good in sociology),
c) Estimate of minimum sample size & What if strategy,
d) Comparison of models via information criterias (AIC, BIC)
e) Bayesian approach

18. PRACTICAL SIGNIFICANCE

19. Practical significance - terminology
b) Substantive significance
c) Logical significance
d) Scientific significance
sometimes also:
e) result importance or
f) result meaningfulness

20. How to measure Practical sig.?
History - Absolute and relative approach
Example: Income differencies
Absolute and relative difference

21. How to measure Practical sig.?
Effect sizes – measures of practical significance
Some well known:
Cohen d
Hayes ω
But also R2, r, C, Fisher η2 are effect sizes
Problem: Sometimes published but not interpreted

22. OTHER SIGNIFICANCES

23. Special significances
Economic significance
Clinical significance
Etc.

24. CONCLUSION? Statistical significance is: LIMITED MISUSED BUT NOT BAD
Substantive significance is:
NOT OFTEN USED
BUT NECESSARY