DIF and cross-cultural measurement of cognitive functioning Paul K. Crane, MD MPH Laura B. Gibbons, PhD

DIF and cross-cultural measurement of cognitive functioning (and a few baby pictures) Paul K. Crane, MD MPH Laura B. Gibbons, PhD (Isaac M. Crane, baby) (Heidi M. Crane, MD MPH, mom)

Cognitive functioning itself (as opposed to its measurement) is intertwined with language –Difficult (impossible?) to separate how we think from our use of language (Saussere, Derrida, etc.) No reason to think cognitive functioning as measured by cognitive tests is the same in two different languages –Translation / back translation may result in test items with the same literal meanings, but relative importance of some tasks may differ in different linguistic contexts (i.e., in different languages) Cross cultural measurement

Examples Obviously wrong: WORLD vs. MUNDO spelled backwards –WORLD backwards is much harder than MUNDO backwards 10/66 group (next few slides, borrowed from their website)

Theory and applications for assessing success of cross-cultural measurement strategies are not well developed Yet we hope to compare cognitive functioning across languages in many specific contexts –CSHA, SALSA, 10/66, Ni-Hon-Sea … –Alzheimer’s Association area of focus –NIA health disparities / diverse elderly focus –R01 application for 2/1/06 many of us will be involved in DIF approaches may help

Lurking in the background of all DIF detection methods is the assumption that some θ = some other θ –Not a conceptual problem for gender, ethnicity, education, or other characteristics within a given language; the other items in the test provide an anchor; iterative techniques (or other techniques) to determine a set of relatively DIF-free items –This is potentially a conceptual problem for assessing DIF related to language of test administration But it’s even more complicated than that!

Educational attainment (quality) interferes with the measurement of cognitive functioning within a single language – (Manly, Teresi, Jones, Crane, Mungas…) Educational attainment differs markedly across language groups –Spanish speakers on average fewer years of school than English speakers in the US –French speakers on average fewer years of school than English speakers in Canada –Japanese speakers on average fewer years of school than English speakers in the Kame study That really throws a wrench into the some θ = some other θ problem

Review: Our general approach to DIF detection Establish θ=θ with a DIF free core: DIF free core Items with DIF Group APresent Missing Group BPresentMissingPresent

Several strategies of attack Goal is to develop a strategy to assess the success or failure of cross-language measurement of cognitive functioning –If DIF is found it would be nice to adjust for it Would like a strategy that can incorporate what we know about DIF related to education as well We will consider 4+ strategies

Strategy 1 Treat the θs as ships passing in the night, ignoring educational DIF: Items 1-nItems (n+1)-2n Language APresentMissing Language BMissingPresent

Explanation of previous slide There are n items in this test There are people in two language groups (A and B) This table illustrates a data structure for PARSCALE –Each person has a response for each item –Parameters for each item are attained separately for the two populations by virtue of the missing data strategy (a “not presented” item for PARSCALE) –Scores for the two populations are defined by a θ determined by 2n items Add education DIF to this structure:

Lang A DIF free items Lang A ed DIF items Lang B DIF free items Lang B ed DIF items Lang A, high ed Present Missing Lang A, low ed PresentMissingPresentMissing Lang B, high ed Missing Present Missing Lang B, low ed Missing PresentMissingPresent

Strategy 1 with education DIF Each language group has two education levels For each language group some of the items will be found to have DIF related to education Within each language: –Items without educational DIF have parameters estimated from whole sample –Items with educational DIF have parameters estimated from the two educational groups separately based on the missing data structure

Comments on strategy 1 θ language A = θ language B = θ Nothing at all in common between the two languages except the underlying metric, which is defined equally by the two language groups Deal with educational DIF separately in the two different languages Feasible with PARSCALE Never actually test for DIF related to language for any particular item (θ is used) Theoretical basis? (None I know of)

Strategy 1 vs. Strategy 0 Strategy 0 is the null strategy – treat the items as coming from two different samples and two different tests –Do some work with the parameters from the two samples See Rasch papers later on in talk

DIF assessment related to education within two language groups Identify a core set of items that do not seem to have education-related DIF in either language Use those items as the initial anchor items for the assessment of language DIF Strategy 2

Ed DIF free core Ed DIF Lang APresent Missing Lang BPresentMissingPresent Items without educational DIF used as initial DIF free core for purposes of calibration Items with educational DIF treated separately in different language groups (i.e., have parameters estimated separately in each language group) Two educational groups per language group on next slide:

No Ed DIF Ed DIF (Lang A) Ed DIF (Lang B) Lang A, High Ed Present Missing Lang A, Low Ed PresentMissingPresentMissing Lang B, High Ed PresentMissing PresentMissing Lang B, Low Ed PresentMissing Present This is the same data structure as the previous slide This slide makes the division of items with educational DIF in the two language groups explicit Also need to consider different findings of educational DIF in the two different languages (next slide):

No Ed DIF Ed DIF Lang B but not Lang A Ed DIF Lang A unique plus both Ed DIF Lang A but not Lang B Ed DIF Lang B unique plus both Lang A, High Ed Present Missing Lang A, Low Ed Present MissingPresentMissing Lang B, High Ed PresentMissing Present Missing Lang B, Low Ed PresentMissing PresentMissingPresent

Here we have added consideration of items that have DIF related to education in one language but not the other, otherwise no difference –Could have separated out columns for 1. DIF related to education in both languages (“both”) 2. DIF related to education only in language A (“lang A unique”), and 3. DIF related to education only in language B (“lang B unique”) Next consider language DIF for the core items

Doub le DIF free core Lang DIF but no ed DIF Ed DIF only in Lang B * Ed DIF in Lang A Ed DIF only in Lang A Ed DIF in Lang B * Lang A Present MissingPresent Missing Lang A Present MissingPresentMissingPresentMissing Lang B PresentMissingPresentMissing Present Missing Lang B PresentMissingPresentMissing PresentMissingPresent

Comments on strategy 2 With this strategy we do test for language DIF, at least for items that do not have education DIF What do we think about the treatment of items with education DIF in the two languages? –Never assess whether they also have DIF related to language (treated as if they do) Practical problem: may have too few anchor items; double DIF free core may be too small for anchoring

Strategy 3 Similar to strategy 2 but we allow items with DIF related to education to be considered as potential anchor items for assessment of language DIF –Items with DIF related to education in one language but not the other de facto have DIF related to language –Items with DIF related to education in both languages may not have DIF related to language

Double DIF free core Lang DIF but no ed DIF Ed DIF only in Lang B * Ed DIF only in Lang A † Ed DIF only in Lang B * Ed DIF in both Lang APresent MissingPresent Missing PresentMissing Lang APresent MissingPresentMissingPresentMissing Present Lang BPresentMissingPresentMissing Present MissingPresentMissing Lang BPresentMissingPresentMissing PresentMissingPresentMissingPresent

This strategy treats items that were found with educational DIF in both languages as other potential core items for the purposes of calibrating related to language of test administration Otherwise no difference from strategy 2

Comments on strategy 3 My favorite of these strategies Accounting problem trying to keep track of item categories Maximizes potential anchor items for the language analysis, increasing the validity of the common θ metric

Strategy 4 A priori (theory-driven) identification of potential anchor items for language evaluation –May make most sense from the perspective of SENAS, where Dan prospectively designed the metrics in Spanish and English with particular anchor items –Uncertain whether it makes sense in other contexts

Core itemsOther items Lang APresent Lang BPresent

Core items Other items without language DIF Other items with language DIF Lang APresent Missing Lang BPresent MissingPresent

Comments on strategy 4 DIF related to education is ignored –But see next slide Whether core items are really DIF free is never assessed Can there be post-hoc determinations of anchor items for tests like the 3MS or the CASI?

Core items, no Ed DIF Core items with Ed DIF Other items without Lang or Ed DIF Other items without Lang DIF but with Ed DIF Other items with lang DIF but no Ed DIF * Other items with both lang DIF and Ed DIF † Other items with lang DIF but no Ed DIF * Other items with both lang DIF and Ed DIF † Lang A, High Ed Present MissingPresent MissingPresent Missing Lang A, low Ed PresentMissingPresent MissingPresent MissingPresentMissing Lang B, High Ed Present MissingPresent Missing Present Missing Lang B, Low Ed PresentMissingPresent MissingPresentMissing PresentMissingPresent

Explanation of prior slide DIF related to education is taken care of –Core items with ed DIF treated distinctly in ed groups but anchor across languages –Similarly for non-core items found not to have language DIF but found to have ed DIF –Items with language DIF but no ed DIF treated separately in different languages –Items with both language and ed DIF treated as 4 different items, specific for educational level and language

DIF strategies from the literature Bjorner (1998) – Danish vs. English SF-36. Contingency table approach incorporating “partial version of the gamma coefficient” conditioning on total score; they left out items found with DIF and re-ran as 2 nd step Kutlay (2003): UK vs. Turkish versions of HRQL scale; no comment on what was used as anchor Martin (2004): 6-item HA scale in 6 languages. LR approach, scale score used as anchor

DIF strategies from lit – 2 Roorda (2004) – Dutch vs. Canadian Arthritis index; Rasch approach; calibrations separately in English and Dutch; difficulties centered and plotted against each other; scores calculated using missing data approach Hahn (2005): FACT-B in Austria and the US. Rasch approach. Difficulty calculated separately in each language; mean location for the 2 languages; Spearman rank correlations; calculated test statistics and confidence intervals; missing data for items with DIF (as here); repeated once with this anchor

Strategy 5 Rasch-like: we could do some sort of equating of difficulties and let the slopes vary (?) –Graphical display and thinking behind centered difficulties within a language is appealing ???

Overall comments on strategies Range from ignoring educational DIF to ignoring language DIF to theoretically uncertain to not worked out yet Rasch-like approach Each θ (language A) = θ (language B) is problematic –Relationships of items with the latent trait assumed to be the same across languages –Is this ever an acceptable assumption? –Possible role for SEM approaches?

Conclusions It is not a simple thing to test for language DIF, especially in the face of large differences in educational background in the language groups We have suggested 4+ potential strategies but would love guidance Thanks for your attention