Effect Sizes

Coding Coding Effect Sizes Coding Studies Usually the first thing I do because you can’t include a study in a meta-analysis if you can’t compute an effect size from it. However, I will present it second here, after Coding Studies Separate coding of studies from coding of effect sizes to reduce biased coding.

Many Kinds of Effect Sizes For Continuous Variables d (the standardized mean difference statistic) r (the correlation coefficient) For Dichotomous Variables o (the odds ratio, for dichotomous outcomes) l (the log odds ratio) The relative risk The rate difference etc. All have the effect of standardizing results into a common metric.

Effect Sizes for Dichotomous Data

The Odds Ratio Most widely used effect size measure for dichotomous outcomes Where A, B, C, and D are cell frequencies OR = 1 is no effect. Lower Bound 0 (Control better than Treatment) upper bound infinity (Treatment < Control)

The Original Simpson & Pearson’s Hospital Staff Incidence Data

A Fourfold Table from Pearson’s Hospital Staff Incidence Data Condition of Interest Group Immune Diseased All Inoculated A = 265 B = 32 M =297 Not Inoculated C = 204 D = 75 M =279 Total N = 469 N = 107 T = 576

OR from Proportions The odds ratio can also be computed from proportions And there are many variations on this formula depending on which proportions you do or do not know

Log Odds Ratio Easier to work with statistically Makes interpretation more intuitive, similar in some respects to d. 0 = no effect Range is +/-. In example, LOR = ln(3.04) = 1.11

Converting LOR to d May wish to do this if most of your effect sizes are in d and just a few in OR, and you want to pool them all: Cox (1970): Sanchez-Meca et al (Psych Methods) showed this approximation works well

The Relative Risk (Risk Ratio) Also used for dichotomous outcomes Where p1+ and p2+ are the marginal proportions of the first row and the second row, respectively. Commonly converted to the Log Risk Ratio

Example Condition of Interest Group Immune Diseased All Inoculated p11 = .4600 p12 = .0556 p1+ = .5156 Not Inoculated p21 = .3541 p22 = .1302 p2+ = .4843 Total p+1 = .8141 p+2 = .1858 The probability of being immune if inoculated is 1.22 times higher than the probability if not inoculated.

Converting RR to Odds Ratio Which in our example is

Number Needed to Treat How many units must be treated to produce a successful outcome Where R1 is the success rate in the treatment group and R0 is the success rate in the control group That is, treat 9 units with T to obtain one more success than would have occurred under C A measure of cost-effectiveness of tmt

Difference Between Proportions (Risk Difference) Intuitive: For Pearson’s Data: 89.2% of those inoculated were immune 73.1% of those not inoculated were immute, so The difference in immunity rates for those inoculated or not is about 16.1%

Analyzing Fourfold Tables by the Correlation Tetrachoric, approximated by Useful if binary measures created by dichotomizing continuous ones Bad to use the phi coefficient (ordinary Pearson correlation applied to binary data) Sensitive to marginal distributions

Preferences? Most tend to prefer OR or LOR Likelihood ratio test of the null But all measures for fourfold tables have potential interpretation problems Reporting success rates, and NNT, advisable. Bonnet (2007) American Psychologist, also Kraemer (2004) Statistics in Medicine are nice summaries of issues in summarizing fourfold tables

Effect Sizes for Continuous Variables r (the Pearson correlation) Not widely used or reported in treatment outcome studies d (the standardized mean difference statistic) By far the most widely used

d: The Standardized Mean Difference Statistic Where s is an estimate of the standard deviation of the numerator, and standardizes the numerator. It is often the pooled standard deviation (a weighted average of the standard deviation of each group)

Estimating d d itself Algebraic equivalents to d Good approximations to d Methods that require intraclass correlation Methods that require ICC and change scores Methods that underestimate effect Note: Italicized methods will be covered.

Sample Data Set I: Two Independent Groups Treatment Comparison 3 2 4 2 4 4 4 5 5 5 6 6 6 7 7 8 7 9 Mean 5.2 5.4 Standard Deviation 1.398 2.319 Sample Size 10 10 Correlation between treatment and outcome is r = -.055

Calculating d

Interpreting d Cohen suggested: 0 = no effect .20 = small effect .50 = medium effect .80 = large effect Lipsey and Wilson found a slightly narrower range empirically (.3, .5, .67) However, remember that what counts as small, medium or large will vary from topic to topic (e.g.,SCDs)

More on Interpreting d Suppose d = .51 (Shadish et al 1993 MFT) Convert to Cohen’s U3 index From the Unit Normal Curve Implies that a therapy client at the mean was better off than 69.5% of control clients; z Below z Above z Between mean and z Ordinate 0.50 0.6915 0.3085 0.1915 0.3521 0.51 0.6950 0.3050 0.1950 0.3503 0.52 0.6985 0.3015 0.1985 0.3485 0.53 0.7019 0.2981 0.2019 0.3467

More on Interpreting d Use U3 to compute an improvement in percentile rank: Improvement = U3 – 50% = 69.5% - 50% = 19.5% The average treatment unit improves 19.5% compared to the average control unit

More on Interpreting d Converts to a correlation coefficient of roughly .25 (Hedges & Olkin, 1985, p. 77) So that treatment accounts for about r2 = .24712 = 6% of outcome variance

More on Interpreting d Rosenthal and Rubin 1982: Translates into a treatment success rate of about 62% in marital and family therapies compared to 38% in control groups

Hedges’ Correction for Small Sample Bias d overestimates effect size in small samples (< 10-15 total) Correction is I always use this correction as it never harms estimation. In SPSS COMPUTE D = ES*(1-(3/((4*(NT+NC))-9))).