RCAL1 Validity Threats, Biases & Systematic Error in International Research Roger Calantone 2004 Not Complete Without Commentary

RCAL2 Overarching Empirical Goal Is Cross-cultural Equivalence In: Measures & their properties Instruments & their administration Researcher / experimenter conduct Research settings

RCAL3 We Must Be Able to Set up Controls to Positively Insure Equivalence. Failing This We Must Have Specific Measures of These Threats to Validity, So We Might Control for Them Statistically.

RCAL4 Bias Cross-cultural bias is the asymmetry of measurement across cultures. Bias can be a function of the instrument based constructs interacting differentially with the cultures addressed, the instrument itself or both. A lack of cross-cultural bias permits measurement equivalence, but cannot guarantee it.

RCAL5 Equivalence Construct equivalence (also referred to as structural equivalence). Measurement unit equivalence. Scalar equivalence (also referred to as full score comparability). The above levels of equivalence are listed in increasing order of difficulty.

RCAL6 Construct Equivalence The most basic level of equivalence. The same construct occurs in each culture, although it may be measured in a different manner (items or methods). (Of course you may not be able to compare scores across cultures unless the constructs are operationalized the same way.).

RCAL7 Measurement Unit Equivalence Constructs are sufficiently similar and well used enough across the cultures so offset mechanism is known or discoverable. Simple example is Celsius versus Kelvin scales of temperature.(Simple transform universally). Complex example is use of a Swedish IQ test on Turkish youngsters.(A constant disadvantage is known but varies by sample and/or by study).

RCAL8 Scalar Equivalence Hardest to obtain; Rarely demonstrated. Measurement instrument is on the same ratio scale in each cultural group. (eg:measure of body length & weight). Achievable when scores on an instrument have same interval scale across cultural groups. (Refers to useful characteristic of interval scales => differences on an interval scale are measured at a ratio level).

RCAL9 Equivalence Must Be Established It is not an intrinsic property of a scale or instrument or method. There is no evidence that equivalence in one set of cultural comparisons is present across other cultural comparisons. Equivalence is a function of the properties of an instrument and the characteristics of the cultural groups involved in the study.

RCAL10 Claims of Equivalence Can Be Controversial ! Scalar equivalence is often claimed when only construct equivalence has been established. Many times this is based on finding similar item loadings of similar or even identical items, using an EFA on the several cultural groups. This is very open to suspicion.

RCAL11 2 Big Problems Using EFA to Establish Scalar Equivalence Measurement units may not be the same across groups. If the scores in one group are doubled, but the correlation pattern remains the same, the cross-cultural differences are affected but the loadings will be unaffected ! Measurement unit equiv.. May obtain, but a bias that affects all of an instrument’s stimuli in a systematic way cannot be detected by EFA.

RCAL12 EFA Problems Continued Both arguments against the use of EFA to establish scalar equivalence derive from the use of the correlation matrix as input to an EFA. Linear transformations, e.g.: (y=ax+b with a>0) impact raw scores & hence x-cultural differences but do not impact correlations. Techniques insensitive to such linear transformations are ill-suited to substantiate scalar equivalence.

RCAL13 Avoiding EFA The simple fix is to design for scalar equivalence from the beginning and to only attempt scalar testing using CFA, which uses the covariance matrix and permits statistical equivalence tests of loadings across cultural groups. Unit measure equivalence testing should also employ Optimal Scaling.

RCAL14 Biases & Their Detection Bias is the basic name for all the factors that challenge the validity of cross- cultural comparisons. Three types of Bias are commonly recognized: Construct, Method, & Item. Bias is always present, but with care, can almost always be mitigated to non- threatening levels through good design.

RCAL15 TABLE 1 Overview of Types of Bias and Their Most Common Causes (after van de Vijver & Poortinga, 1997, p. 26) Type of BiasSource Construct incomplete overlap of definitions of the construct across cultures differential appropriateness of (sub)test content (e.g., skills do not belong to the repertoire of one of the cultural groups) poor sampling of all relevant behaviors (e.g., short instruments) incomplete coverage of the construct (e.g., not all relevant domains are sampled) Method differential social desirability differential response styles such as extremity scoring and acquiescence differential stimulus familiarity lack of comparability of samples (e.g.differ in educational background, age, or gender composition) differences in physical conditions of administration differential familiarity with response procedures tester/interviewer effects communication problems between respondent and interviewer in either cultural group Item poor item translation inadequate item formulation (e.g., complex wording) item(s) may invoke additional traits or abilities incidental differences in appropriateness of the item content (e.g., topic of item of educational test not in curriculum in one cultural group)

RCAL16 TABLE 2 Is Level of Equivalence Affected by Bias? Level of Equivalence Construct Measurement Unit a Scalar a, b Type of Bias Construct biasno no no Method bias: uniformyes yes no nonuniformyes no no Item bias: uniformyes yes no nonuniformyes no no a. The same measurement unit is assumed in each cultural group. b. The same origin is assumed in each cultural group.

RCAL17 Construct Bias Occurs when construct measured is not identical across groups. (appears frequently in comparisons of western & non-western cultures). Can be induced by lack of overlap in behaviors associated with the cultures studied. (E.g.. Machiavellianism). Also, Construct underrepresentation.

RCAL18 Construct Bias Fixes Emic approaches must precede. Decentered approach in conceptualization of the study. Convergence approach in conceptualization of the study. This is considerably more difficult if indigenous materials must be developed.

RCAL19 Method Bias Bias arises from characteristics of instrument and/or its administration. Differential response styles across cultures. (use of scales/extremes). Differential familiarity with stimuli used. Differential familiarity with response tool: E.g. draw a picture versus make a wire model.

RCAL20 Method Bias Avoidance “Culture-Free” tests. “Culture-Fair” tests. Interviewer repetition & rotation. Systematic variation of stimuli across cultures & examination of score changes as well as differential learning effects. Use of un-timed exercises.

RCAL21 Item Bias Differential appropriateness of item content (asymmetric item sets). Inadequate item functioning (e.G. Complex wording). Inadequate translation concordance. All of these can be mitigated by rigorous pretesting, with post-inquiry debriefing.

RCAL22 TABLE 3 Four Common Types of Cross-Cultural Studies Orientation More on Consideration of Contextual Factors Hypothesis TestingExploration No Generalizability Psychological differences Yes Theory-driven External validation SOURCE: van de Vijver & Leung,

RCAL23 Discussion Questions Can we really design culture-equivalent instruments? Can we reduce the bias in our comparative studies? Are instrument fixes better than statistical fixes? Where does equivalence and bias fit in the EMIC/ETIC debate?

RCAL24