How to evaluate the cross-cultural equivalence of single items Melanie Revilla, Willem Saris RECSM, UPF Zurich – 15/16 July

Cross-cultural equivalence Usually –Discussed in the frame of cross-national research –Idea: in different countries people can express themselves in different ways –Different cultures can also be defined on other criteria (e.g. language) –Procedure: can be applied in similar way to all kinds of different groups “Equivalence”  measurement equivalence –2 persons with the same opinion will give the same answer (whatever their group) Important –because observed differences may result from non equivalent measures and not be real differences If measurement equivalence does not hold –cannot make comparison across groups!!

Important distinction (Northrop, 1947) Concept by Intuition (CI) –Simple concepts that can be measured directly –Single item –Ex: trust in the parliament Concept by Postulation (CP) –Complex concepts that cannot be measured directly –Also called “construct” –Need several CI to measure them –Ex: political trust: trust in parliament + legal system + police … Classic procedure to test for equivalence for CP but not for CI  start with a reminder of the procedure for CP

When we have multiple indicators CP / complex concepts

Basic C onfirmatory F actor A nalysis model CP 1 Y1Y1 Y2Y2 Y3Y3 λ 11 λ 21 λ 31 e1e1 e2e2 e3e3 τ1τ1 τ3τ3 τ2τ2 Y i = τ i + λ ij CP 1 + e i i = 1,2,3 Political trust Answer Trust in the parliament Answer Trust in the legal system Answer Trust in the police interceptsslopeserror terms ≈ regression equation [ Dependent variable Independent variable

Multiple Group CFA approach Group 1Group 2 Multiple group: –possible to test for equality of the parameters in the different groups –constraints across groups Can be extended to more groups

Different levels of invariance (Meredith, 1993) Configural –Same model holds in all groups Metric –Configural + Slopes (λ ij ) the same in all groups –Sufficient for comparison of relationships Scalar –Metric + Intercepts (τ i ) the same in all groups –Sufficient for comparison of means More: error terms, etc… Group 1 Group 2

In practice Analyses can be done with standard SEM softwares –LISREL/Mplus –based on covariance matrices & means –recommended sample size: >200 in each group –3-step procedure: configural, metric, scalar –syntax quite easy to get estimates More tricky but crucial step: testing

Testing the model Assessing global fit –Chi 2 test / Fit indices: RMSEA (.9), etc… –Limits: Depends on sample size / Sensitive to deviations from normality Assessing local fit –Saris & Satorra  should test at the parameter level + take into account type II errors (H0 not rejected despite being false) –JRule software (van der Veld, Saris, Satorra) + Jrule for Mplus (Oberski) See next presentation! Always check if estimates are really different –Difference may be statistically significant but not substantially meaningful Partial invariance –What if some indicators are equivalent but not all? –Consistent estimates of the means of the latent variables if at least 2 indicators are scalar invariant (Byrne, Shavelson, Muthén, 1989)

If we have single indicator CI / simple concepts

Single items Y1Y1 e1e1 τ1τ1 CI 1 λ 11 Testing equivalence single items Testing equivalence for CI Y i = τ i + λ ij CI i + e i Y1Y1 e1e1 τ1τ1 CI 1 λ 11 Group 1Group 2

Single  multiple indicators? Problem: model just presented not identified “Single indicator” = single trait in fact But possible to use multiple methods So for CI: –Only one trait, but we can always have more than one method –Several indicators = same trait asked using different methods

Can apply again MGCFA Group1Group 2 Similar at the previous model (for CP) but now different methods instead of different traits measuring a same concept Trust in the parliament 11 points 6 points 4 points

Same procedure Different levels of invariance as for CP –Configural –Metric –Scalar –etc Same procedure to get the estimates and test the model –Multiple group analyses –Test of the model: global / local fit –Partial equivalence

Problem for the CI Fix the scale? –As before, necessary to fix the scale of the LV –Usually, fix the first loading to 1 –Can be done here too –Other loadings are relative to the first one –But need to be done in all groups –If there are differences for the method whose loading is fixed to 1 across groups, may be problematic –Should try to use methods that have been shown to be the most similar across groups: e.g. fixed reference points

General model CP 1 Y 31 Y 22 Y 13 c1c1 c2c2 c3c3 e 11 e 12 e 33 τ 11 τ 33 τ 12 CI 1 CI 2 CI 3 v 21 v 22 v 23 u1u1 u2u2 u3u3 α1α1 α2α2 α3α3 Even when working on CP: better to use different methods Y 21 Y 12 Y 23 Y 33 Y 11 Y 32 e 21 τ 21 e 31 τ 31 e 23 τ 23 e 13 τ 13 e 22 τ 22 e 32 τ 32 1 v 31 v 32 v

Final remarks / CCL

Equivalence single items Need to repeat the same item with different methods –3 or more repetitions –Multi Methods (MM)? Same persons get the question several times using different methods Limit: 20 minutes at least to avoid memory effects (Van Meurs & Saris, 1990) –Mix with Split-Ballot (SB) design? Random assignment of respondents to different versions of the questionnaire “SB-MM” (CI) or SB-MTMM (CP)?

Conclusion Measurement equivalence can be assessed both for CP and CI using M ultiple G roup C onfirmatory F actor A nalysis –For CP, process already well-known and used a lot –For CI, possible do similar analyses  But necessary to repeat questions!!  specific data So testing equivalence of simple item can be done using a (SB)-(MT)MM approach –Similar to what exists in the ESS for CP: main + supplementary questionnaires (different versions) –With extension for concepts by intuition

In summary To test single item equivalence –Use multiple methods –Do everything as for multiple items equivalence

Thank you for your attention! Questions?