1. Fit of Ideal-point and Dominance IRT Models to Simulated Data Chenwei Liao and Alan D Mead Illinois Institute of Technology

2. Outline  Background and Objective  Hypotheses and Methods  Results  Discussions

3. Background  Personality Used in personnel selection - Incremental validity to predict job performance beyond cognitive ability (Barrick & Mount, 1991; Ones et al, 1993) - Less adverse impact (Feingold, 1994; Hough, 1996; Ones et al, 1993).  Model-data-fit - Need to calibrate personality traits - Use IRT models - Degree of fit depends on data structure

4. Background (cont.)  Item response processes – thinking of data structure  IRT models and item response processes: 1) Traditional dominance IRT models: - high trait - high probability of endorsing 2) Ideal-point IRT models - similar item & trait – high probability of endorsing

5. Background (cont.) Dominance Model IRF: - x: Theta (trait level) - y: Probability of endorsing Ideal-point Model IRF: - x: distance between person trait and item extremity - y: Probability of endorsing

6. Background (cont.)  Chernyshenko et al, (2001) - Traditional dominance IRT models have failed. Suggest to look at item response processes and Ideal-point IRT models  Stark et al. (2006) - Ideal-point IRT models: as good or better fit to personality items than do dominance IRT models  Chernyshenko et al. (2007) - Ideal-point IRT method: more advantageous than dominance IRT and CTT in scale development in terms of model-data-fit

7. Limitation of previous studies and objective of current study  Limitation of previous studies - Unknown item response processes!  Objective of current study 1) Investigate model-data-fit by utilizing simulation with known item response processes 2) Test the assumption that the best fit model represents data underlying structure of response processes

8. Current Study

9. Models  Dominance: - Samejima ’ s Graded Response Model (SGRM);  Ideal Point: - General Graded Unfolding Model (GGUM).  Larger sample and longer test were said to be related to a better fit (Hulin et al, 1982; De la Torre et al, 2006).

10. Hypotheses Generating models  H1: Data generated by an ideal point model will be best fit by an ideal-point model and data generated by a dominance model will be best fit by a dominance model.  H2: The ideal point model will fit the dominance data better than the dominance model will fit the ideal-point data.  H3: The ideal-point model will fit the mixture data better than the dominance model.

11. Hypotheses (cont.) Sample Sizes  H4: All models will fit better in larger samples.  H5: The GGUM model will fit relatively worse in smaller samples, as compared to simpler, dominance models. Test Lengths  H6: The GGUM model will fit relatively worse for very short tests, as compared to longer tests.

12. Datasets  Self-Control Scale from the 16PF  Procedure: 1) Calibrate 16PF data to get item parameters - SGRM: PARSCALE4.1; GGUM: GGUM2004. 2) Generate simulated data: - models: ideal point/dominance/mixed; - sample size: 300, 2000; - test length: 10, 37; - 50 replications;

13. Model-Data-Fit  Cross validation ratio: each item in each condition  Only singles – simulation study assures unidimensionality assumption  Smaller value – better fit  Frequencies of ratios were tallied into 6 groups: very small ( =5).

14. Results overview ConditionBest fitting model Dominance data generationGGUM Ideal point data generationGGUM Mixed data generationGGUM Small Sample (N=300)GGUM Large Sample (N=2000)GGUM Short Test (n=10)GGUM Long Test (n=37)GGUM

15. Results

16. Discussion (1)  “ GGUM fits better ” - Confirm previous findings. - However, because regardless of the underlying response process, GGUM fits better than SGRM, it does not demonstrate that the response process or IRF/ORF is non-monotone. The previous assumption does not hold true. - Possible reason: Software (PARSCALE & GGUM) manifest models differently  Better fit in small samples, especially for SGRM - Explanation: chi-square is sensitive to sample size

17. Discussion (2)  Examine similarities of the theta metrics - Negative correlation between theta estimates from GGUM and those from SGRM TRUESGRMGGUM TRUE1.000 SGRM0.9281.000 GGUM-0.923-0.9951.000

18. Discussion (3)  Scaling issue GGUM: - Reverse the estimate - Add a constant in scaling

19. Thanks!