Effect Size

The Effect of a Training Program Suppose employees go through a training program. We can measure their progress as function of how much time (x) they spent in training. Skill Y1 Skill Y3 The effect of the training was greater on Skill Y1 than Skill Y3. => r is a measure of effect size.

Effect Size: Difference Between Groups Suppose a group of leaders are randomly assigned to two types of leadership training (blue and red). Two months later they are rated by their employees. Cohen’s d measures effect size of the training:

Cohen’s d d is measured in units of standard deviation (like z) Examples

Cohen’s Classification of Effect Sizes Conversion: d

Introduction to Meta-Analyses

Meta-Analyses: Combining Multiple Studies of a Phenomenon When a phenomena is measured numerous times, we get a variety of effect sizes. Example: the correlation between conscientiousness and cooperation when conflicts occur. r = -.02, n = 142, p = .81 r = .04, n = 92, p = .71 r = .33, n = 322, p < .001 r = .35, n = 115, p < .001 A meta analysis creates a weighted mean effect size ρ = .22, k = 4, N = 671, p < .001

Meta-Analyses: Moderators Meta-analyses can detect moderators Moderator: a condition that changes the strength of the relationship. Example Studies of conscientiousness and cooperation on college students r = -.02, n = 142, p = .81 r = .04, n = 92, p = .71 Studies of conscientiousness and cooperation on working adults r = .33, n = 322, p < .001 r = .35, n = 115, p < .001 Age is a moderator of the relationship between conscientiousness and cooperation.

Meta-Analysis: Example A meta-analysis of the relationship between organizational citizenship behavior and counterproductive work behavior. (Dalal, 2005)

Multiple Regression Significance Testing

Testing the significance of the Coefficients (B) in Multiple Regression. Predicting salaries at a lumber company from Education Age Sex Are all the predictors significant? Significant means we can be reasonably sure whether a predictor is negative or positive.

Testing the significance of the Coefficients (B) in Multiple Regression. Example 2: Can we predict who will like Justin Bieber? File: Chapter Supplemental Class Data.xlsx

Reading Regression Analyses. Rather than report the coefficient B, the standardized coefficient β is often used. It is the coefficient of the regression equation if we used all z-scores rather than raw scores. It has the same intuitive meaning as r. It is an effect size. For each model (set of variables used to predict the dependent variable), we look at R2 to see if it is significant. When we add new variables to the model we check the significance of 1) the new coefficient and 2) the change of R2 (= ΔR2)

Ex: Hierarchical multiple regression analyses (standardized beta coefficient) with insomnia as dependent variable (N=1,254) Notes: *P < .10, **P < .05, ***P < .01, ****P < . 001

Power

β error (Type II error) and Power β is the fraction of the time we’re going to make a Type II error: retaining the null hypothesis when we should reject it.

Definition of Power The probability that a study will detect a difference or relationship between two variables if such a difference or relationship is real. The value of power can go from 0 (0% chance of detecting the relationship) to 1.00 (100% chance) Tests with power of more than .80 are rare in behavioral sciences. Often around .50.

β Error (Type II Error) and Power If we mistakenly retain the null hypothesis 10% of the time when we should reject it, the power of the test = 100% - 10% = 90% The higher the power, the better (we want to make as few mistakes as possible).

Power and Effect Size Power is a measure of the sensitivity of a test to detect a real effect. The greater the effect size (r or d families) the more power we have to detect the effect. Example: A correlation of r = .30 between a personality trait and productivity is easier to detect than a correlation of r = .12.

Ways to Increase Power BEAN: If you want to change the B, you change the others. B = β, Power = 1 – β. Low β error means high power. E = Effect size. The greater the real effect, the greater the power to detect it. A = α. The probability of falsely rejecting H0. The greater it is, the greater the power. N = sample size. Increasing N increases the power. Sample Size Calculator: http://www.sample-size.net/correlation-sample-size/

ANOVA Analysis of Variance

t-Test vs. ANOVA (F-Test) A t-test looks for a difference between 2 groups that differ in one specific factor. Example: In the company being studied, does organizational satisfaction differ by gender? Dependent Variable (DV) = Organizational Satisfaction (ratio data) Independent Variable (IV, the factor that interests us) = gender (Male or Female, nominal data))

t-Test vs. ANOVA (F-Test) An ANOVA looks for a difference between 2 or more groups that differ in 1 or more specific factors. Example: In the company being studied, does organizational satisfaction differ by gender, seniority, and position? DV = Organizational Satisfaction (ratio data) IVs, the various factors (all nominal): Gender (Male or Female) Seniority (Less than 5 years, More than 5 years) Position (Hourly, Salaried, Upper management) An ANOVA can also test for interactions between factors: Do gender differences in Org Satisfaction differ with seniority?