1. Is the probability value enough for appraising the effectiveness of a therapy? The role of effect size.
Zacharias Dimitriadis, MSc,PhD1, Maria Tsekoura, MSc1,2, Alexandros Kastrinis, MSc, CMP1, Eleni Nomikou, MSc1
1. Clinical physiotherapist
2. Laboratory collaborator ΑΤΕΙ West Greece, Aigio
Introduction
Figure 1.
Interpretation of probability value and effect size
Shows how possible is the findings of an experiment to be attributed to random chance (Hicks, 1999)
P value
Nowadays, physiotherapy is one of the most important health sciences. The consideration of physiotherapy as a science prerequisites the application of therapeutic interventions which are based on solid research evidence. Last years, it has been noted a research explosion as medical-related studies are continuously increased. However, the vast amount of studies provides less and less time for critical appraisal resulting in a higher rate of misunderstanding of studies and quite often to their inappropriate application in clinical practice (Reid and Chung, 2004).
Shows the amount of the influence of an independent variable (e.g. a treatment) to a dependent variable (e.g. pain) (Hicks, 1999)
Effect size
A usual misconception
Figure 2.
Potential conclusions about the effectiveness of
a treatment based on probability value
(p value)
and effect size (e.s.)
One of the most important misconceptions is the erroneous consideration of statistical indices such as of P value (Probability value) and effect size (e.s.) (Fig 1). Many Clinicians and researchers erroneously accept the effectiveness and appropriateness of a therapy by only considering the statistical significance of the results (p value). This fact also explains why systematic reviews, whom results are mainly based on statistical significance, can lead to different conclusions from meta-analyses which are mainly based on effect sizes (Borenstein et al, 2009).
P < 0.05
e.s.: low
P > 0.05
e.s.: high
P > 0.05
e.s.: low
P < 0.05
e.s.: high
The treatment is highly effective and this effectiveness has low chance to be random.
Acceptance of the treatment as effective
The treatment is minimally effective and this effectiveness has low chance to be random.
Acceptance of the treatment as minimally effective
The treatment is highly effective and this effectiveness has high chance to be random.
The treatment is rather effective but bigger sample size is needed for the acceptance of these findings
The treatment is minimally effective and this effectiveness has high chance to be random
These findings are discouraging for the collection of more participants (although it is needed in order to have safer conclusions)
Which is the relationship between P value
and effect size?
In reality, the effectiveness/efficacy of the treatments under examination is depicted to indices which express effect sizes. P value should be used as an additional index which shows the probability of each effect size to have been randomly found. The simultaneous consideration of both indices will finally show whether the associated research findings can be statistically and clinically accepted (Fig 2).
Which indices express the effect sizes?
Table 1: Frequently used standardized
effect sizes
The calculation equations and symbols of effect size vary according to the statistical analysis which has been selected and the researchers’ preferences about the effect sizes that may rise from the same statistical analysis. However, it is important to mention the need for standardized indices since effect size may be also considered the difference between two means (e.g. Pre-intervention and post-intervention).
However, the use of such unstadardized indices renders impossible the comparison of the effectiveness of a treatment between two different studies which have examined the same concept (variable) but by using different units. Thus, standardized indices (Table 1) significantly help to the comparison of the effectiveness of treatments for a certain problem even though the same problem/variable has been examined by using different units.
Effect size index
Type
Comments
Standardized effect size
ES= (Mpost – Mpre) / spre
Informs about the magnitude of change in standardized units relative to the pre-test standard deviation
Standardized response mean
d = (Mpost – Mpre ) / schange
Informs about the magnitude of change in standardized units relative to the variance of change
Hedges’ g
g = J x d
It is the effect size of meta-analyses. It is the product of the standardized mean difference through a corrective factor (J) (for bias reduction)
Conclusions
References
Borenstein M., Hedges L.V., Higgins J.P.T., Rothstein H.R. (2009). Introduction to meta-analysis. Great Britain: Wiley
Hicks C. (1999).Research methods for clinical therapists: applied project design and analysis. China: Churchill Livingstone
Portney and Watkins (2009).Foundations of clinical research: applications to practice 3rded. New Jersey: Pearson Education International
Reid W.D., Chung F. (2004). Clinical management notes and case histories in cardiopulmonary physical therapy. China: Slack Incorporated
The estimation of effect size is equally important to the calculation of statistical significance in order to extrapolate safe conclusions regarding the effectiveness of an examined treatment. Thus, these two indices should be simultaneously considered during the interpretation of each research findings.