© UCLES 2013 Assessing the Fit of IRT Models in Language Testing Muhammad Naveed Khalid Ardeshir Geranpayeh

© UCLES 2013 Outline Item Response Theory (IRT) Importance of Model Fit within IRT Fit Procedures Issues and Limitations Lagrange Multiplier (LM) Test An empirical study using LM Fit statistics Sharing Results Conclusions

© UCLES 2013 Item Response Theory (IRT)  A family of mathematical models that provide a common framework for describing people and items  Examinee performance can be predicted in terms of the underlying trait  Provides a means for estimating abilities of people and characteristics of items

© UCLES 2013 IRT Models Dichotomous or Discrete 1 Parameter Logistic Model / Rasch (1PL) 2 Parameter Logistic Model (2PL) 3 Parameter Logistic Model (3PL) Polytomous or Scalar Partial Credit Model (PCM) Generalized Partial Credit Model (GPCM) Graded Response Model (GRM)

© UCLES 2013 Shape of Item Response Function

© UCLES 2013 Model for Item with 5 response categories Probability Response Category

© UCLES 2013 IRT Applications IRT applications in language testing are mainly used in  Test development  Item banking  Differential item functioning (DIF)  Computerized adaptive testing (CAT)  Test equating, linking and scaling  Standard setting The utility of the IRT model is dependent upon the extent to which the model accurately reflects the data

© UCLES 2013 Model Fit from Item Perspective Measurement Invariance (MI): Item responses can be described by the same parameters in all sub- populations. Item Characteristic Curve (ICC): Describes the relation between the latent variable and the observable responses to items. Local Independence (LI): Responses to different items are independent given the latent trait variable value. Uni-dimensionalty Speededness Global

© UCLES 2013 Consequences of Misfit Yen (2000) and Wainer & Thissen (2003) have shown the inadequacy of model-data fit Some of the adverse consequences are:  Biased ability estimates  Unfair ranks  Wrongly equated scores  Student misclassifications  Score precision  Validity

© UCLES 2013 Existing Item Fit Procedures Chi – Square Statistics Tests of the discrepancy between the observed and expected frequencies. Pearson-Type Item-Fit Indices (Yen, 1984; Bock, 1972). Likelihood Ratio Based Item-Fit Indices (McKinley & Mills, 1985).

© UCLES 2013 Issues in Existing Fit Procedures  The standard theory for chi-square statistics does not hold.  Failure to take into account the stochastic nature of the item parameter estimates.  Forming of subgroups for the test are based on model- dependent trait estimates.  There is an issue of the number of degrees of freedom.  It is sensitive to test length and sample size.

© UCLES 2013 Lagrange Multiplier (LM) Test Glas(1999) proposed the LM test to the evaluation of model fit. The LM tests are used for testing a restricted model against a more general alternative one. Consider a null hypothesis about a model with parameters This model is a special case of a general model with parameters

© UCLES 2013 LM Item Fit Statistics Null Model Alternative Model MI / DIF LI ICC Null ModelAlternative Model Null Model Alternative Model

© UCLES 2013 Empirical Example Data from Cambridge English First (FCE) –Reading 3 parts/30 questions –Listening 4 parts/30 questions Sample size over The approach can be applied to any other language exam

© UCLES 2013

Conclusions  LM statistics overcome existing FIT issues  Less computational intensive  Size of residuals in the form of Abs.Dif is highly valuable  Fit of IRT model holds reasonably (FCE)  Items violated - MI (4); ICC (3); LI (7)  Magnitude of violation is not severe

© UCLES 2013 Thank you! & Questions