Multilevel Multitrait Multimethod model. Lluís Coromina (Universitat de Girona) Barcelona, 06/06/2005

Measurement data quality in social networks analysis. Assess reliability and validity in egocentered social networks. Complete networks Egocentered networks Background

Index Reliability and Validity MTMM Model Data Multilevel analysis Results and interpretation

Reliability and Validity MTMM Model Confirmatory Factor Analysis (CFA) specification of the MTMM model. Y ij = m ij M j + t ij T i + e ij (1) () where: Y ij : response or measured variable “i” measured by method “j”. T i : unobserved variable of interest (trait). Related to validity. M j : variation in scores due to the method. Related to invalidity. m ij and t ij : factor loadings on the method and trait factors. e ij : random error, which is related to lack of reliability. Reliability and Validity

Figure 1 : Path diagram for the MTMM model for trait (T i ) and method (M j ). MTMM model

Traits T1T1 Frequency of contact T2T2 Feeling of closeness T3T3 Feeling of importance T4T4 Frequency of the alter upsetting to ego Methods M1M1 Face-to-face interviewing M2M2 Telephone interviewing MTMM model

Y 11 Y 21 Y 31 Y 41 Y 22 Y 12 Y 32 Y 42 M1M1 M2M2 T1T1 T2T2 T3T3 T4T4 e 11 e 21 e 31 e 41 e 12 e 22 e 32 e 42 Figure 2: Path diagram of a CFA MTMM model for two methods and four traits. MTMM model

Var (Y ij ) = m ij 2 Var (M j ) + t ij 2 Var (T i ) + Var (e ij ) (2)() Validity and Reliability for CFA MTMM model: Reliability coefficient = (3) Validity coefficient = (4)

Kogovšek, et al., 2002: Estimating the reliability and validity of personal support measures: full information ML estimation with planned incomplete data. Social Networks, 24, T 1 Frequency of contact T 3 Feeling of importance T 2 Feeling of closeness T 4 Frequency of the alter upsetting to ego GNFirst interviewSecond interview M 1 Face-to-faceM 2 Telephone Table 1: The design of the study Data Representative sample of inhabitants of Ljubljana

Multilevel analysis Two-level MTMM model. The highest level: group level = egos = g The lowest level: individual level = alters = k Multilevel MTMM model

The mean centred individual scores for group “g” and individual “k” can be decomposed into: Within group component (5) Between group component (6) where: is the total average over all alters and egos. is the average of all alters of the g th ego. Y gk is the score on the name interpreter (questions) of the k th alter chosen by the g th ego. G is the total number of egos. n is the number of alters within each ego. N=nG is the total number of alters. Multilevel MTMM model

Sample covariance matrices: Multilevel MTMM model SW=SW=SB=SB= S T = S B + S W = (7)(8) (9) Population covariance matrices:  T =  B +  W (10) Y ij = m Bij M Bj + t Bij T Bi + e Bij + m wij M wj + t wij T wi + e wij (11)() Y Bij Y Wij

Härnqvist Method Separate analysis for S B and S W Group measure S w is the ML estimator of Σ W S B is the ML estimator of Σ W +cΣ B (12) Multilevel MTMM model Model estimated by Maximum Likelihood (ML).

Figure 3: Multilevel CFA MTMM Model. Multilevel MTMM model

Interpretation: We can obtain 2 reliabilities and 2 validities for each trait-method combination. To analyse each component separately: Y ij = m Bij M Bj + t Bij T Bi + e Bij + m wij M wj + t wij T wi + e wij (11)() Y Bij Y Wij Decompose the variance: Var (Y ij ) =m ij 2 w Var (M jW ) + m ij 2 B Var (M jB ) + t ij 2 w Var (T iW ) + t ij 2 B Var (T iB ) + (13) () Var (e ijw ) + Var (e ijB ) Multilevel MTMM model

Analysis: Multilevel MTMM model  Analysis 1: traditional analysis on S T. ML estimation.  Analysis 2: traditional analysis on S W. ML estimation.  Analysis 3: traditional analysis on S B, which is a biased estimate of Σ B. ML estimation. Analyses 2 and 3 together constitute the recommendation of Härnqvist (1978).  Analysis 4: multilevel analysis, to fit Σ W and Σ B simultaneously. ML estimation.

Table 1: Goodness of fit statistics. Results and interpretation

Table 2: Decomposition into 6 variance components. Analysis 4. Table 2 trait variance withinT1T1 T2T2 T3T3 T4T4 M1M M2M method variance within* M1M M2M error variance within M1M M2M trait variance between M1M M2M method variance between* M1M M2M error variance between* M1M M2M * Boldfaced for small non- significant variances constrained to zero. Results and interpretation

Table 3Table 3: Decomposition into 6 variance components* trait variance withinT1T1 T2T2 T3T3 T4T4 M1M1 67,2%66,0%66,3%55,2% M2M2 67,5%70,2%70,6%55,0% method variance within* M1M1 2,6%3,6%3,1%3,7% M2M2 0,0% error variance within M1M1 13,3%19,2%17,4%26,0% M2M2 12,6%17,4%14,7%20,6% trait variance between M1M1 15,3% 8,1%10,5%15,1% M2M2 16,0% 9,0%11,6%15,6% method variance between* M1M1 0,0% M2M2 0,8%1,2%1,0%1,1% error variance between* M1M1 1,5%3,1%2,7%0,0% M2M2 3,2%2,2%2,1%7,7% * Boldfaced for small non-significant variances constrained to zero. Results and interpretation

Within levelBetween levelOverall level T1T1 T2T2 T3T3 T4T4 T1T1 T2T2 T3T3 T4T4 T1T1 T2T2 T3T3 T4T4 Reliability coef M1M1 0,920,890,900,840,950,850,891,000,920,880,890,86 M2M2 0,920,900,910,850,920,910,930,830,920,900,910,85 Validity coef M1M1 0,980,970,980,971,00 0,98 M2M2 1,00 0,980,940,960,971,000,99 Table 4Table 4: Multilevel reliabilities and validities* Results and interpretation * Boldfaced for small non-significant variances constrained to zero.

Table 5Table 5: Within part. Comparison of analyses 2 (S W ) and 4 (multilevel). * Boldfaced for small non-significant variances constrained to zero. Results and interpretation

Table 6Table 6: Between part. Comparison of analyses 3 (S B ) and 4 (multilevel). * Boldfaced for small non-significant variances constrained to zero. Results and interpretation

Table 7Table 7: Overall analysis. Comparison of analyses 1 (S T ) and 4 (multilevel). * Boldfaced for small non-significant variances constrained to zero. Results and interpretation

T1T1 T2T2 T3T3 T4T4 t ij 2 w Var(T iw )/ [t ij 2 w Var(T iw ) + t ij 2 B Var(T iB )] Table 8Table 8: Percentages of variance at within level form M1 and M2 Results and interpretation T1T1 T2T2 T3T3 T4T4 Var(eijw)/ Var(Yij)

Contribution: To consider egocentered networks as hierarchical data. To specify a multilevel MTMM. Interpretation from measurement theory of different % of variance. Results and interpretation

For further information and contact: