1. Multilevel Multitrait Multimethod model. Lluís Coromina (Universitat de Girona) Barcelona, 06/06/2005

2. Measurement data quality in social networks analysis. Assess reliability and validity in egocentered social networks. Complete networks Egocentered networks Background

3. Index Reliability and Validity MTMM Model Data Multilevel analysis Results and interpretation

4. Reliability and Validity MTMM Model Confirmatory Factor Analysis (CFA) specification of the MTMM model. Y ij = m ij M j + t ij T i + e ij (1) () where: Y ij : response or measured variable “i” measured by method “j”. T i : unobserved variable of interest (trait). Related to validity. M j : variation in scores due to the method. Related to invalidity. m ij and t ij : factor loadings on the method and trait factors. e ij : random error, which is related to lack of reliability. Reliability and Validity

5. Figure 1 : Path diagram for the MTMM model for trait (T i ) and method (M j ). MTMM model

6. Traits T1T1 Frequency of contact T2T2 Feeling of closeness T3T3 Feeling of importance T4T4 Frequency of the alter upsetting to ego Methods M1M1 Face-to-face interviewing M2M2 Telephone interviewing MTMM model

7. Y 11 Y 21 Y 31 Y 41 Y 22 Y 12 Y 32 Y 42 M1M1 M2M2 T1T1 T2T2 T3T3 T4T4 e 11 e 21 e 31 e 41 e 12 e 22 e 32 e 42 Figure 2: Path diagram of a CFA MTMM model for two methods and four traits. MTMM model

8. Var (Y ij ) = m ij 2 Var (M j ) + t ij 2 Var (T i ) + Var (e ij ) (2)() Validity and Reliability for CFA MTMM model: Reliability coefficient = (3) Validity coefficient = (4)

9. Kogovšek, et al., 2002: Estimating the reliability and validity of personal support measures: full information ML estimation with planned incomplete data. Social Networks, 24, 1-20. T 1 Frequency of contact T 3 Feeling of importance T 2 Feeling of closeness T 4 Frequency of the alter upsetting to ego GNFirst interviewSecond interview 3141371M 1 Face-to-faceM 2 Telephone Table 1: The design of the study Data Representative sample of inhabitants of Ljubljana

10. Multilevel analysis Two-level MTMM model. The highest level: group level = egos = g The lowest level: individual level = alters = k Multilevel MTMM model

11. The mean centred individual scores for group “g” and individual “k” can be decomposed into: Within group component (5) Between group component (6) where: is the total average over all alters and egos. is the average of all alters of the g th ego. Y gk is the score on the name interpreter (questions) of the k th alter chosen by the g th ego. G is the total number of egos. n is the number of alters within each ego. N=nG is the total number of alters. Multilevel MTMM model

12. Sample covariance matrices: Multilevel MTMM model SW=SW=SB=SB= S T = S B + S W = (7)(8) (9) Population covariance matrices:  T =  B +  W (10) Y ij = m Bij M Bj + t Bij T Bi + e Bij + m wij M wj + t wij T wi + e wij (11)() Y Bij Y Wij

13. Härnqvist Method Separate analysis for S B and S W Group measure S w is the ML estimator of Σ W S B is the ML estimator of Σ W +cΣ B (12) Multilevel MTMM model Model estimated by Maximum Likelihood (ML).

14. Figure 3: Multilevel CFA MTMM Model. Multilevel MTMM model

15. Interpretation: We can obtain 2 reliabilities and 2 validities for each trait-method combination. To analyse each component separately: Y ij = m Bij M Bj + t Bij T Bi + e Bij + m wij M wj + t wij T wi + e wij (11)() Y Bij Y Wij Decompose the variance: Var (Y ij ) =m ij 2 w Var (M jW ) + m ij 2 B Var (M jB ) + t ij 2 w Var (T iW ) + t ij 2 B Var (T iB ) + (13) () Var (e ijw ) + Var (e ijB ) Multilevel MTMM model

16. Analysis: Multilevel MTMM model  Analysis 1: traditional analysis on S T. ML estimation.  Analysis 2: traditional analysis on S W. ML estimation.  Analysis 3: traditional analysis on S B, which is a biased estimate of Σ B. ML estimation. Analyses 2 and 3 together constitute the recommendation of Härnqvist (1978).  Analysis 4: multilevel analysis, to fit Σ W and Σ B simultaneously. ML estimation.

17. Table 1: Goodness of fit statistics. Results and interpretation

18. Table 2: Decomposition into 6 variance components. Analysis 4. Table 2 trait variance withinT1T1 T2T2 T3T3 T4T4 M1M1 0.790.560.670.47 M2M2 0.760.550.640.45 method variance within* M1M1 0.03 M2M2 0.00 error variance within M1M1 0.16 0.180.22 M2M2 0.140.13 0.17 trait variance between M1M1 0.180.070.110.13 M2M2 0.180.070.110.13 method variance between* M1M1 0.00 M2M2 0.01 error variance between* M1M1 0.020.03 0.00 M2M2 0.040.02 0.06 * Boldfaced for small non- significant variances constrained to zero. Results and interpretation

19. Table 3Table 3: Decomposition into 6 variance components* trait variance withinT1T1 T2T2 T3T3 T4T4 M1M1 67,2%66,0%66,3%55,2% M2M2 67,5%70,2%70,6%55,0% method variance within* M1M1 2,6%3,6%3,1%3,7% M2M2 0,0% error variance within M1M1 13,3%19,2%17,4%26,0% M2M2 12,6%17,4%14,7%20,6% trait variance between M1M1 15,3% 8,1%10,5%15,1% M2M2 16,0% 9,0%11,6%15,6% method variance between* M1M1 0,0% M2M2 0,8%1,2%1,0%1,1% error variance between* M1M1 1,5%3,1%2,7%0,0% M2M2 3,2%2,2%2,1%7,7% * Boldfaced for small non-significant variances constrained to zero. Results and interpretation

20. Within levelBetween levelOverall level T1T1 T2T2 T3T3 T4T4 T1T1 T2T2 T3T3 T4T4 T1T1 T2T2 T3T3 T4T4 Reliability coef M1M1 0,920,890,900,840,950,850,891,000,920,880,890,86 M2M2 0,920,900,910,850,920,910,930,830,920,900,910,85 Validity coef M1M1 0,980,970,980,971,00 0,98 M2M2 1,00 0,980,940,960,971,000,99 Table 4Table 4: Multilevel reliabilities and validities* Results and interpretation * Boldfaced for small non-significant variances constrained to zero.

21. Table 5Table 5: Within part. Comparison of analyses 2 (S W ) and 4 (multilevel). * Boldfaced for small non-significant variances constrained to zero. Results and interpretation

22. Table 6Table 6: Between part. Comparison of analyses 3 (S B ) and 4 (multilevel). * Boldfaced for small non-significant variances constrained to zero. Results and interpretation

23. Table 7Table 7: Overall analysis. Comparison of analyses 1 (S T ) and 4 (multilevel). * Boldfaced for small non-significant variances constrained to zero. Results and interpretation

24. T1T1 T2T2 T3T3 T4T4 t ij 2 w Var(T iw )/ [t ij 2 w Var(T iw ) + t ij 2 B Var(T iB )]0.830.890.870.85 0.800.880.850.76 Table 8Table 8: Percentages of variance at within level form M1 and M2 Results and interpretation T1T1 T2T2 T3T3 T4T4 Var(eijw)/ Var(Yij)0.130.190.170.26 0.130.170.150.21

25. Contribution: To consider egocentered networks as hierarchical data. To specify a multilevel MTMM. Interpretation from measurement theory of different % of variance. Results and interpretation

26. For further information and contact: http://www.udg.es/fcee/professors/llcoromina