1. IRT Applications of Kullback- Leibler Divergence and Analysis of its Distribution Dmitry Belov Law School Admission Council Ronald Armstrong Rutgers University

2. Plan 1. Detecting answer copying 2. Other applications 3. General framework 4. What is asymptotic distribution of Kullback-Leibler divergence? 5. Summary

3. Given a set of response vectors, identify pairs of examinees involved in answer copying. Problem statement SourceSubject CABDAEECDBA…

4. Response vector of an examinee 1111101111101110111111111 10110100000010110010 ABCDEABCDEABCDEABCDEABCDE ABCDEABCDEABCDEABCDE operational part (scored) has items identical for all examinees variable part (unscored) has items different for adjacent examinees

5. General method Stage 1: Identify potential subject (Kullback-Leibler Divergence). Stage 2: Given potential subject identify possible source (K-Index or M-Index).

6. Identify potential subject 1111101111101110111111111 10110100000010110010 H(  ) G(  ) Kullback-Leibler divergence:

7. One pair found in real data Operational Variable …1010111111001111101110… 0000000001000000001010100 …5445244253413211313455… 1534134455141423114323454 …1010111111001111101110… 0111111010100111111101101 …5445244253413211313455… 1534134455141423114323454 Subject was seated in front and to the left of the Source.

8. Other pairs found in real data …101111111010111011… 100110000100000000001000000 …531355321223155543… 255254322454242423122254142 …531355321223155543… 445254322454242423125254142 …11100110101… 010000000100000000000100000 …25312243312… 545322243141243332244534423 …25312243312… 54332224312124333224453442 …00911010110100… 000100000000000000001000000 …12445111122215… 323531215351541334523214245 …12445111122215… 232532215114421252132314245 …11011011110110… 000000000010000001000000100 …35122413433333… 121335232115132142254145441 …35122413433333… 121332523211513214254145441

9. Other applications of comparing posteriors method Detecting aberrant responding (operational items vs. variable items, hard items vs. easy items, unexposed items vs. exposed items, uncompromised items vs. compromised items, analyzing test repeaters) Checking unidimensionality of a test (items of one type vs. items of another type) Detecting aberrant speed of responding

10. General framework 1111101111101110111111111 10110100000010110010 H(  ) G(  ) Kullback-Leibler divergence:

11. What is asymptotic distribution of Kullback-Leibler divergence? What about posteriors? Chang, H. H., & Stout, W. (1993). The asymptotic posterior normality of the latent trait in an IRT model. Psychometrika, 58, 37-52. Gelman, A., Carlin, J. B., Stern, H. S., & Rubin, D. B. (2004). Bayesian data analysis. Boca Raton: Chapman & Hall.

12. KL divergence between normal densities

13. True for any population, allows random Consider two parallel tests Tg and Th with smooth information functions administered to an examinee with ability  Tests Tg and Th are different

14. Simulated data 10000 examinees from N(0,1) 70 items test (a=1, b~N(0,1), c=0.1) KSS=0.015

15. Real data In LSAT variable part is about 4 times smaller than the operational part. OperationalVariable + odd responses from + even responses from

16. Real data KSS=0.020

17. Alternative distributions of KL 1. Scaled chi-square with one degree of freedom 2. Scaled noncentral chi-square with one degree of freedom (to check for unidimensionality of a test ) 3. Scaled F 4. Scaled doubly noncentral F

18. Symmetric KL divergence between normal densities

19. Summary Comparing posteriors has many applications. For the comparison one can use KL, phi, or (h, phi) divergences. For normal posteriors in unidimensional IRT the asymptotic distribution of KL is analyzed. LSAC uses corresponding software to detect aberrant responding.