Testing factorial invariance in multilevel data: A Monte Carlo study Eun Sook Kim Oi-man Kwok Myeongsun Yoon

The purpose of the present study  Investigate the influence of data dependency on the test of factorial invariance in multilevel data.  Comparisons between multilevel SEM (the model- based or the designed-based multiple-group multilevel CFA) and multiple-group CFA

Measurement invariance studies in multilevel data  Multilevel SEM  The groups of interest are treated as random samples.  Good for measurement invariance testing in individual and organizational levels.  Multiple-group CFA  The number of groups is small or relatively finite (e.g., <20 or so).  Can be utilized to test measurement invariance over groups at the organizational level.

Multilevel measurement model  A conventional measurement model  A multilevel measurement model

In a multilevel measurement model  The variance of the factor  The variance of residual  The variance of observed scores

Examine measurement invariance in multilevel data  The general multilevel CFA model

Examine measurement invariance in multilevel data  Test at the between level  Test at the within level Study 1 Study 2

Data generation

Design  Noninvariance in the organizational (study 1) and the individual level (study 2).  Manipulated factors  The number of clusters (CN):30, 50, 80 and 160  Cluster size (CS): 10 and 20  The size of ICC:.09,.20, and.33  The total sample size was between 300 and 3,200, and 1,000 replications were in a condition.

 Data dependency on group membership:  Intraclass correlation (ICC)

Analysis  Tested a configural and a metric models.  The LRT was used to test factorial invariance in multiple-group CFA.  The Satorra-Bentler scaled chi-squared test was used in multiple-group multilevel CFA.  Examined power and type I error.  Conducted ANOVA to examine the influence of 3 manipulated factors on power and type I error.

Results (Study 1) Low admissible rates.

Analysis in study 2  Examined factorial invariance at the individual level using the design-based multilevel CFA, instead of the model-based one, due to the limitation of the SEM software.

Results (study 2)

Limitations and conclusions  The explanations of the discrepant findings between study 1 and study 2 might be confounded by the type of analysis.  It is necessary to take account of the data dependency in multilevel data.  A substantially large number of clusters is required in model-based multiple-group multilevel modeling.

Comments  Information of the accuracy of estimated parameters is not available.  MLR provides accurate factor loadings but underestimates the residual variances and standard errors at the between level. Having more clusters or higher ICC does not effectively compensate for this.  Groups of interest might occur at different levels simultaneously, and how to model them?

Comments (Cont.)  It is not clear how the model was constrained. This study seemed to constrain the variance of the latent factor to be equal across groups, which is not practical and reasonable in real data.  The large effect size in the factorial invariance with a large sample size yielded very high power, and it is recommended reducing the effect size to medium.

Comments (Cont.)  Contrast to the findings of Jones-Farmer (2010), the inflated type I error occurred in CFA even though data dependency was considered small. Further examinations are needed to understand the controversial findings.

Future studies  Further examinations in  Scalar and error variance models  Cross-level invariance  The issue of sample sizes using other estimation methods  In MLR, the min. sample size of the between level is 100. How about in FIML?  The assessment of model fit in ML SEM.

Future studies (Cont.)  Examine more than 1 items with different factor loadings.  Implement purification in this study.  Conduct equal means for the latent factor or equal intercepts of indicators to examine factorial invariance.  Examine scalar invariance to see if the findings are similar to the findings of French and Finch (2010).

Future studies (Cont.)  The possibility to develop a new indicator in SEM, such as adding person fit (as IRT approaches do) to SEM approaches, instead of model fit only.  Conduct the model-based multilevel CFA using other software to overcome the limitation in current SEM software.