1. Part 2 DIF detection in STATA

2. Dif Detect - Stata Developed by Paul Crane et al, Washington University based on Ordinal logistic regression (Zumbo, 1999) Ordinal or continuous covariates (i.e. not restricted to binary). Model incorporates latent trait scores rather than sum scores - advantage over parametric methods http://www.alz.washington.edu/DIFDETECT/welcome.html DIFwithPAR but need Parscale software for this Publications: Crane, Belle and Larson (2004) Test bias in a cognitive test: differential item functioning in the CASI; Statistics in Medicine, 23, 241-256.

3. DifD website

4. DifD Model specification f (item response) = cut + ß ∗ 1θ (1) f (item response) = cut + ß ∗ 1θ + ß ∗ 2 group (2) f (item response) = cut + ß ∗ 1θ + ß ∗ 2 group+ß ∗ 3θ ∗ group (3) Gibbons et al (2009) International Psychogeriatics, 21:1 The program examines 3 ordinal logistic regression models for each item Cut represents the cutpoint(s) for each level in the proportional odds ologit model θ (theta) is the IRT estimate of ability (e.g derived from Mplus) Group is the indicator for the covariate In model 3, β3 is the coefficient for the ability-group interaction term. Tests for Uniform and Non-uniform DIF ologit itemresponse ability group ability*group

5. Detecting Non-Uniform & Uniform DIF Non-Uniform DIF (e.g. demographic interference between ability and item responses differs at varying levels of the trait) Log likelihoods of models 2 & 3 are compared to test the significance of the interaction term Uniform DIF Fits models with and without group assignment (i.e. models 1 & 2) Compares the relative difference between parameters associated with θ If the relative difference was >10% then uniform DIF is present. Option to also be compare -2 log likelihoods). Default alpha 0.20 (Maldonado & Greenland, 1993) Gibbons et al (2009) International Psychogeriatics, 21:1

6. How to run Dif Detect - Stata In stata type: findit difd 1 package found (Stata Journal and STB listed first) difd from http://fmwww.bc.edu/RePEc/bocode/dhttp://fmwww.bc.edu/RePEc/bocode/d 'DIFD': module to evaluate test items for differential item functioning (DIF) / DIF detection is a first step in assessing bias in test items. / difd detects DIF in test items between groups, conditional on the trait that the test is measuring, using logistic regression. Gives installation file: (click here to install) difd.ado difd.hlp

7. How to run DIFd in Stata Code for detection of differential item function (DIF) from difd.hlp. difd varlist, ID(var) ABility(varlist) GRoups(varlist) CATegorical(varlist) RUnname(str) NUL(#) NUP(#) NUPValue(#) UBeta(#) UBP(#) ULPV(#) UP(#) UPPValue(#) ITemsub(#) where: varlist is the list of variables (items) to be tested for DIF id (nb leave this out). ability is the ability variable(s) (derive from Mplus or similar). groups is the list of grouping variables. (can use binary, ordinal or Continuous ‘grouping’ variables) Options categorical is the list of any group variables that are categorical and have more than 2 levels. Note can omit dichotomous variables from this list. Default is none (all continuous or dichotomous). runname names the log file DIFdRUnname.log. Default is DIFd.log.`

8. DIFd in Stata Code for detection of differential item function (DIF) from difd.hlp. difd varlist, ID(var) ABility(varlist) GRoups(varlist) CATegorical(varlist) RUnname(str) NUL(#) NUP(#) NUPValue(#) UBeta(#) UBP(#) UL(#) ULPV(#) UP(#) UPPValue(#) ITemsub(#) Options cont…. ul indicates whether the log-likelihood test will be used as a criterion for uniform DIF. Default is no (0). UL(1) will include this criterion ulpvalue is the p-value for testing uniform DIF with the log-likelhood method. Default is 0.05. Note: DIF results for categorical grouping variables will be in terms of the ordered values of group. For example, if ethnic has 3 levels, 3 sets of DIF results will be reported: ethnic12, ethnic13, ethnic23, where ethnic12 compares the 2 lowest values of ethnic, ethnic13 the lowest and highest, etc.

9. DIFd Stata – back to Mplus Run basic CFA model and save factor scores (ability) from Mplus to a data file 1) Add syntax to specify your ID in VARIABLE command: idvar is caseno; SAVEDATA: SAVE=FSCORES; FILE=C:\DATA\bext16.DAT; 3) Add SAVEDATA following OUTPUT statement and specify file name and location 2) Add AUXILIARY in VARIABLE command to ensure covariates to be used to test for DIF are included in saved data file (as these will not in your basic CFA model or use variable statement ) auxiliary is sex;

10. Mplus to save F scores (ability) USEVARIABLES are rut03 rut04 rut10 rut14 rut18; CATEGORICAL are rut03 rut04 rut10 rut14 rut18; idvar is caseno; AUXILIARY = sex; missing are all ( 88 999 ); ANALYSIS: ! TYPE is missing H1; ESTIMATOR IS wlsmv; ITERATIONS = 1000; CONVERGENCE = 0.00005; MODEL: Conduct by rut03 rut04 rut10 rut14 rut18; OUTPUT: SAMPSTAT STANDARDIZED RES MOD(10) ; SAVEDATA: SAVE=FSCORES; FILE=C:\DATA\bext16.DAT;

11. Mplus output (save data) SAVEDATA INFORMATION Order and format of variables Save file C:\DATA\bext16.DAT Save file format 5F10.3 I6 F10.3 Save file record length 5000 Item responses Individual factor scores / ability scores RUT03 F10.3 RUT04 F10.3 RUT10 F10.3 RUT14 F10.3 RUT18 F10.3 CASENO I6 SEX F10.3 CONDUCT F10.3

12. Import.dat file to spss Open.dat file in spss using text import wizard nb. Step 2 - select fixed width Step 4 - make sure column breaks are right aligned because of missing data Step 6 – check if numeric Right align col Save as stata file!

13. DIFd in Stata difd rut03 rut04 rut10 rut14 rut18, ru(difd16ext) ab(conduct) gr(sex) cat(sex) ul(1) log: C:\data\DIFdfin16ext.log (0 observations deleted) There are 8773 observations. The 5 items of interest: rut03 rut04 rut10 rut14 rut18. The 1 group of interest: sex. The 1 ability of interest: conduct. _______________________________________________________________ Non-Uniform Differential Item Functioning ------------------------------------------------------------------------------------- -> group = sex -------------------------------------------- ability | and item | P(Dif.(LL)) Non-Uniform DIF ----------+--------------------------------- conduct | rut03 |.8930773 no rut04 |.0532258 no rut10 |.0515209 no rut14 |.0001041 yes rut18 |.245261 no Non-Uniform DIF if P(Dif.(LL)) <.05

14. DIFd in Stata (Uniform DIF output) Uniform Differential Item Functioning -> group = sex ---------------------------------------------------------- ability | and item | Change in Est. P(Dif.(LL)) Uniform DIF ----------+----------------------------------------------- conduct | rut03 |.0042654 3.46e-16 yes rut04 |.0135444 5.35e-06 yes rut10 | -.0003362.8060181 no rut14 |.0011879.159341 no rut18 |.0006344.6337193 no ---------------------------------------------------------- Uniform DIF if Change in Est. >.1 or P(Dif.(LL)) <.05 This output was produced using DIFd version 1.0 by Paul Crane, Laura Gibbons, Lance Jolley, and Gerald van Belle University of Washington Copyright 2005

15. DIFd in Stata (output file) Saves parameters estimates an output data set, DIFd.dta, which includes model results, with Brant test p-values for ordinal items and Hosmer-Lemeshow p-values for binary items as data file (difd.dta) Example extract of difd.dta file typeitemgroupabilityllbabsebabbgpsebgpbisebintxpHLpBrantmodeldifll1 1 rut03 sexb16ext-1107.54.1150.414-0.9820.313-0.0390.2870.18310.004-1107.5 1 rut03 sexb16ext-1107.54.0630.143-1.0210.1300.12520.004-1107.5 1 rut03 sexb16ext-1140.84.0460.1400.08030.004-1107.5 2 rut04 sexb16ext-2315.22.8300.2530.1600.1350.3130.1620.04510.014-2315.24 2 rut04 sexb16ext-2317.13.2970.0850.3690.0810.00520.014-2315.24 2 rut04 sexb16ext-2327.53.2530.0840.03830.014-2315.24

16. DIFd in Stata Advantages: Uniform and Non-Uniform Dif Continuous, binary and polytomous items Use ability scores rather than total scores Disadvantages: One subgroup at a time Designed for unidimensional IRT (not multidimensional scales) Based on ordinal logistic regression model so assumes proportional slopes To assess impact of DIF IRT scores can be compared between participants when accounting for DIF and not accounting for Dif

17. References Camilli, G. and Shepard, L. A. (1994). Methods for Identifying Biased Test Items. Thousand Oaks, CA: Sage. Crane, P. K., van Belle, G. and Larson, E. B. (2004). Test bias in a cognitive test: differential item functioning in the CASI. Statistics in Medicine, 23, 241–256. Crane, P. K., Gibbons, L. E., Jolley, L. and van Belle, G. (2006). Differential item functioning analysis with ordinal logistic regression techniques: DIFdetect and difwithpar. Medical Care, 44, S115–S123. Gibbons LE, McCurry S, et al (2009) Japanese–English language equivalence of the Cognitive Abilities Screening Instrument among Japanese-Americans International Psychogeriatrics (2009), 21:1, 129–137 Jones, R. N. (2006). Identification of measurement differences between English and Spanish language versions of the Mini-mental State Examination: detecting differential item functioning using MIMIC modeling. Medical Care, 44, S124–133. Mellenburgh, G. (1989). Item Bias and Item Response Theory. International Journal of Educational Research, 13, 127 – 143. Reise, S.P. Widaman, K.F. and. Pugh RH (1993) Confirmatory Factor Analysis and Item Response Theory: Two Approaches for Exploring Measurement Invariance Psychological Bulletin Vol. 114, No. 3, 552-566 Teresi, J (2006) Different approaches to Differential Item Functioning in Health Applications Advantages, Disadvantages and some neglected topics. Medical Care, 44, 11, S152–170. Zumbo, B. D. (1999). A handbook on the theory and methods of differential item functioning (DIF): Logistic regression modeling as a unitary framework for binary and Likert-type (ordinal) item scores. Ottawa, Canada: Directorate of Human Resources Research and Evaluation, Department of National Defense.