1. Analyzing observed composite differences across groups: Is partial measurement invariance enough? Holger Steinmetz Faculty of Economics and Business Administration Department of Human Resource Management, Small Business Enterprises, and Entrepreneurship University of Giessen / Germany

2. Introduction Importance of analyses of mean differences For instance: - gender differences on wellbeing, self-esteem, abilities, behavior - differences between leaders and non-leaders on intelligence and personality traits - differences between cultural populations on psychological competencies, values, wellbeing Usual procedure: t-test or ANOVA with observed composite scores Latent means vs. observed means Partial invariance as legitimation for the composite difference test Research question: Effects of unequal intercepts and/or factor loadings across groups on composite differences

3. Relationship between latent and observed means x1x1 x4x4 x2x2 x3x3         

4. xixi  i ii x1x1 x4x4 x2x2 x3x3         

5. xixi  i ii x1x1 x4x4 x2x2 x3x3         

6. x1x1 x4x4 x2x2 x3x3  xixi  i ii  E(x i )        

7. Group differences in intercepts and factor loadings xixi   E(x i ) x1x1 x4x4 x2x2 x3x3  x1x1 x4x4 x2x2 x3x3  Group AGroup B

8. Group differences in intercepts and factor loadings xixi   E(x i ) x1x1 x4x4 x2x2 x3x3  x1x1 x4x4 x2x2 x3x3  Group AGroup B

9. Group differences in intercepts and factor loadings xixi   E(x i ) x1x1 x4x4 x2x2 x3x3  x1x1 x4x4 x2x2 x3x3  Group AGroup B

10. The study Partial invariance: Some loadings / intercepts are allowed to differ Research question: Is partial invariance enough for composite mean difference testing? - Pseudo-differences - Compensation effects Procedure (Mplus): - Step 1: a) Specification of two-group population models with varying differences in latent mean, intercepts and loadings b) 1000 replications, raw data saved - Step 2: Creation of a composite score - Step 3: Analysis of composite differences - Step 4: Aggregation (-> sampling distribution)

11. The study Population model: - Two groups - One latent variable Conditions: - 4 vs. 6 indicators - Latent mean difference: 0 vs..30 - Intercepts: equal vs. one vs. two intercepts unequal in varying directions (-.30 vs. +.30) - Loadings: equal ( ‘s =.80) vs. one vs. two loadings =.60 - Sample size: 2x100 vs. 2x300 Dependent variables - Average composite mean difference - Percent of significant composite differences Group A Group B x1x1 x4x4 x2x2 x3x3  x1x1 x4x4 x2x2 x3x3  x5x5 x6x6 x5x5 x6x6  =.00  =-.30 =.80 =.60  =.00  =.30

12. Pseudo-Differences Effects on the average composite difference 4 Ind.6 Ind. N = 2 x 300 4 Ind.6 Ind. 0.00 0.05 0.10 0.15 0.20 0.25 0.30 1 intercept unequal 2 intercepts unequal N = 2 x 100

13. Pseudo-Differences Effects on the probability of significant differences (Type I error) 0.00 0.10 0.20 0.30 0.40 0.50 0.60 4 Ind.6 Ind. 1 intercept unequal 2 intercepts unequal All intercepts equal N = 2 x 100

14. Pseudo-Differences Effects on the probability of significant differences (Type I error) 0.00 0.10 0.20 0.30 0.40 0.50 0.60 4 Ind.6 Ind.4 Ind.6 Ind. 1 intercept unequal 2 intercepts unequal All intercepts equal N = 2 x 300N = 2 x 100

15. Compensation effects Effects on the average composite differences 1 intercept unequal 2 intercepts unequal All intercepts equal Loadings equal 1 Loading unequal 4 Indicators 2 Loadings unequal 0.00 0.05 0.10 0.15 0.20 0.25 0.30 Effect of unequal loadings Effect of unequal intercepts

16. Compensation effects Effects on the average composite differences 1 intercept unequal 2 intercepts unequal All intercepts equal Loadings equal 1 Loading unequal 4 Indicators 2 Loadings unequal Loadings equal 1 Loading unequal 6 Indicators 2 Loadings unequal 0.00 0.05 0.10 0.15 0.20 0.25 0.30

17. Compensation effects Effects on the probability of significant differences (Power) Loadings equal 1 Loading unequal N = 2x300 / 6 Indicators 2 Loadings unequal 1 intercept unequal 2 intercepts unequal All intercepts equal Loadings equal 1 Loading unequal N = 2x100 / 4 Indicators 2 Loadings unequal 0.00 0.10 0.20 0.30 0.40 0.60 0.90 0.50 0.70 0.80

18. Summary Pseudo-differences - Even one unequal intercept increases the risk to find composite differences - High sample size increases risk (up to 60% with two unequal intercepts) - Unequal factor loadings have only a low influence - Number of indicators reduces the risk – but not substantially Compensation effects - Just one unequal intercept reduces the size of the composite difference to 50% - With a “small” sample size little chance to find a significant composite difference (power =.25 -.40) - Two unequal intercepts drastically reduce the composite difference: The power in the „best“ condition (2x300, 6 Ind.) is only.50

19. Conclusons Most comparisons of means rely on traditional composite difference analysis Researcher must not use supported partial invariance as a legitimation for using all items of the scale as a composite Recommendations - Use SEM: a)Testing latent mean differences under partial invariance possible b)Greater power even in small samples - use only those items that were invariant in tests of invariance - Increasing number of items (will, however, probably violate the factor model)

20. Thank you very much! Contact: Holger.Steinmetz@web.de