Reflections and illustrations on DIF Paul De Boeck K.U.Leuven 25th IRT workshop Twente, October 2009

Reflections and illustrations on DIF Paul De Boeck University of Amsterdam 25th IRT workshop Twente, October 2009

Is DIF a dead topic? A non-explanatory approach Paul De Boeck University of Amsterdam 25th IRT workshop Twente, October 2009

Is there life after death for DIF? A non-explanatory approach Paul De Boeck K.U.Leuven 25th IRT workshop Twente, October 2009

The three DIF generations Zumbo, Language Assessment Quarterly, st generation: from “item bias” to “differential item functioning” 2nd generation: modeling item responses, IRT, multidimensional models 3rd generation: explanation of DIF The end of history “.. the pronouncements I hear from some quarters that psychometric and statistical research on DIF is dead or near dying..”

Outline Issues Reflections and more Possible answers

Issues Anchoring Statistic Indeterminacies

I apologize,.. There are already so many methods yes The best among the existing methods are very good methods yes They are standard and good practice yes Do we really need more? no, therefore no real issues And still

1. Anchoring Blind, iterative Purification - all other in step 1 - nonrejected items in following steps A priori set, test They work based on pragmatism and a heuristic, on prior theory, what can one want more?

2. Statistic and its distribution Based on difference per item or set of items MH statistic ST-p-DIF B u from SIBTEST LR test statistic Raju distance Other Parameter estimates They work, what can one want more?

3. Indeterminacies with an IRT modeling approach Basic model is 1PL or Rasch model for uniform DIF 2PL for uniform and non-uniform DIF type 1 2PL multidimensional for uniform and non-uniform DIF type 2

Difficulties – uniform DIF Additive or translational indeterminacy β fi = β ri + δ βi β* fi = β ri + δ* βi δ* βi = δ βi + c β γ* = γ – c β β fi, β ri focal group and reference group difficulties δ βi DIF effect γ group effect * transformed values

Invariance of DIF explanation δ βi = Σ k=0 ω k X ik (+ ε i ) X ik : value of item i on item covariate k ω k : weight of covariate k in explaining DIF k=0 for intercept ω k>0 are translation invariant, and only these covariates have explanatory value

Degrees of discrimination non-uniform DIF type 1 Multiplicative indeterminacy α fi = α ri x δ αi α* fi = α ri x δ* αi δ* αi = δ αi x c α σ θf * = σ θf / c α additive formulation discrimination DIF

Loadings for multidimensional models The indeterminacies look a little embarrassing, because the results depend on one’s choice.

Reflections Random item effects Item mixture models Robust statistics

Intro: Beliefs DIF is gradual why not a random item effect? DIF or no DIF why not a latent class of DIF items? DIF items are a minority why not identify outliers?

Where is the DIF?

Intro: ANOVA approach η gpi = ln(Pr(Y gpi =1)/Pr(Y gpi =0)) η gpi = μ overall mean + λ gp = αθ gp person effect, ability θ gp ~ N (0,1) + λ i = β i item effect, overall item difficulty + λ g = γ g group effect + λ gp interaction p x g does not exist + λ gpi = α’ i θ gp interaction pwg x i + λ gi = β’ gi interaction i x g uniform DIF + λ gpi = α’’ gi θ gp interaction pwg x i x g non-uniform DIF type 1 2PL version

+ λ gpi = α’ i θ gp + λ gpi = α’’ gi θ gp interaction pwg x i x g is non-uniform DIF Type 1 + λ gpi = α’ i θ gp1 + λ gpi = α’’ gi θ gp2 interaction pwg x i x g is non-uniform DIF Type 2

Secondary dimension DIF η gpi = (α i + gδ αi )θ gp + (β i + gδ βi ) + λ g = α i θ gp + gδ αi θ gp + (β i + gδ βi ) + λ g Secondary-dimension DIF η gpi = α i θ gp1 + gδ αi θ gp2 + (β i + gδ βi ) + λ g Cho, De Boeck & Wilson, NCME 2009 g = 0 reference group g = 1 focal group

can explain uniform DIF η gpi = α i θ gp1 + gδ αi θ gp2 + (β i + gδ βi ) + λ g gδ αi μ θg2 + gδ αi θ’ gp2 = gδ βi Cho, De Boeck & Wilson, NCME 2009

Different from the MIMIC model Secondary dimension DIF η gpi θ gp1 θ gp2 G η gpi θ gp1 gθ gp2 G

1. Random item effects Within group random item effects (β ri, β fi ) ~ N(μ βr, 0, σ 2 βr, σ 2 βf, ρ βrβf ) (β i, β f-gi ) ~ N(μ βr, 0, σ 2 β, σ 2 βf-g, ρ ββf-g ) small number of parameters² Idea based on Longford et al in Holland and Wainer (1993) for the MH there is evidence that the true DIF parameters are distributed continuously Van den Noortgate & De Boeck, JEBS, 2005 Gonzalez, De Boeck & Tuerlinckx, Psychological Methods, 2008 De Boeck, Psychometrika, 2008

2. Latent class of DIF items Asymmetric DIF is exported to other items Is avoided when DIF items are removed, appropriate removing eliminates interaction Basis of purification process Let us make a latent class for items to be removed, and identify the DIF items on the basis of their posterior probability

Item mixture model η gpi |c i =0 = θ gp + β i non-DIF class η gpi |c i =1 = θ gp + β gi DIF class θ rp + β i θ rp + β 0i θ fp + β i θ fp + β 1i non-DIFDIF reference focal Frederickx, Tuerlinckx, De Boeck & Magis, resubmitted 2009

further model specifications: - item effects are random - normal for the non-DIF items - bivariate normal for the DIF item difficulties - group specific normals for abilities

Simulation study 1PL P=500, I = 20, 50 x 2 #DIF = 0, 5 (1.5, 1, 0.5, -1, -1.5) x 2 μ θ1 = 0, μ θ2 = 0, 0.5, x 2 = 16 μ β = μ β0 = μ β1, σ 2 β = σ 2 β0 = σ 2 β1 = 1, ρ β0β1 = 0 five replications MCMC WinBUGS prior β variance: Inv Gamma, Half normal, Uniform distributional parameters are estimated posterior prob determines whether flagged as DIF

Results simulation study average #errors LRT 1.64 MH 1.39 ST-p-DIF 0.65 mixture inverse gamma0.30 mixture normal0.36 mixture uniform0.40 item mixture does better or equally good then every other traditional method in all 16 cells

More results - results of mixture model are not affected by DIF being asymetrical - neither by true distribution of item difficulties (normal vs uniform)

3. DIF items are outliers Outlying with respect to the item difficulty difference between reference and focal group Types of difference: - simple difference - standardized – divided by standard error - Raju distance – first equal mean difficulty linking, then standardize τ i = I/(I-1) 2 x (d i - d.) 2 /s 2 d is beta (0.5, (I-2)/2) distributed if d i is normally distributed

Go robust: d. is replaced by the median s d is replaced by mean absolute deviation Taking advantage of the fact that interitem variation is an approximation of se if robustly estimated De Boeck, Psychometrika 2008 Magis & De Boeck, 2009, rejected

20 items, nrs 19 and 20 are the true DIF items

Simple difference

Standardized difference

Raju

Simulation study 1PL P=500, I = 20, 40 x 2 %DIF = 0%, 10%, 20% x 3 size of DIF = 0.2, 0.4, 0.6, 0.8, 1.0 x 5 μ θ1 = 0, μ θ2 = 0, 1 x 2 = replications

Results 0% DIF MH SIBTEST Logistic Raju classic Raju robust Type 1 errors ≈ 5%

Results DIF size = 1, P=1000, I=40, equal μ θ 10% DIF20%DIF Type 1 PowerType 1 Power MH SIBTEST Logistic Raju classic Raju robust

Results are similar for unequal mean abilities Results are similar but less pronounced for smaller P and smaller DIF size

Possible answers Anchoring? Anchor set memberschip is binary latent item variable, or, the clean set of items Statistic? Robust statistic works also for nonparametric approaches Indeterminacy? (go explanatory) no issue for random item model, look at the cov equal means in item mixture approach equal means for Raju distance

Item mixtures and robust statistics do in one step what purification does in several steps, item by item, and through different purification steps – purification is approximate: They both give a rationale for the solving the indeterminacy issue Random item effect approach is not sensitive to indetermincay

Si no è utile è ben ispirazione Good for other purposes or a broader concept than DIF, for qualitative differences between groups Random item models Item mixture models Robust statistics IRT

Thank you, and stay alive

Dimensionality uniform DIF and non-uniform DIF type