1. Neural Test Theory: A nonparametric test theory using the mechanism of a self-organizing map SHOJIMA Kojiro The National Center for University Entrance Examinations, Japan shojima@rd.dnc.ac.jp 1

2. Neural Test Theory (NTT) Shojima (2008) IMPS2007 CV, in press. –Test theory using the mechanism of a self-organizing map (SOM; Kohonen, 1995) Scaling –Latent scale is ordinal. –Latent rank –Number of latent ranks is about [3, 20] –Item Reference Profile –Test Reference Profile –Rank Membership Profile Equating –Concurrent calibration 2

3. Why an Ordinal Scale? Two main reasons: –Methodological –Sociological 3

4. Methodological Reason Psychological variables are continuous –Reasoning, reading comprehension, ability… –Anxiety, depression, inferiority complex… Tools do not have high resolution for measuring them on a continuous scale –Tests –Psychological questionnaires –Social investigation 4

5. Weight and Weighing Machine Phenomenon (continuous) Measure (high reliability) Weight 12 3 4 5

6. Ability and Test Phenomenon (continuous?) Measure (low reliability) Ability 6 1234

7. Resolution Power to detect difference(s) Weighing machines –can detect the difference between two persons of almost the same weight. –can almost correctly array people according to their weights on the kilogram scale. Tests –cannot discriminate the difference between two persons of nearly equal ability. –cannot correctly array people according to their abilities. The most that tests can do is to grade examinees into several ranks. 7

8. Sociological Reason Negative aspects of continuous scale –Students are motivated to get the highest possible scores. –They should not be pushed back and forth by unstable continuous scores. Positive aspects of ordinal scale –Ordinal evaluation is more robust than continuous scores. –Sustained endeavor is necessary to go up to the next rank. 8

9. NTT Latent Rank Theory SOMGTM Binary Shojima (in press) RN07-12 Polytomous (ordinal) RN07-03 In preparation Polytomous (nominal) RN07-21 In preparation Continuous In preparation ML (RN07-04) Fitness (RN07-05) Missing (RN07-06) Equating (RN07-9) Bayes (RN07-15) 9

10. Statistical Learning of the NTT ・ For (t=1; t ≤ T; t = t + 1) ・ U (t) ←Randomly sort row vectors of U ・ For (h=1; h ≤ N; h = h + 1) ・ Obtain z h (t) from u h (t) ・ Select winner rank for u h (t) ・ Obtain V (t,h) by updating V (t,h−1) ・ V (t,N) ←V (t+1,0) Point 1 Point 2 10

11. Mechanism of Neural Test Theory 0 0 0 1 0 0 0 1 0 0 0 1 0 1 1 1 1 0 1 0 1 0 0 1 Latent rank scale Number of i tems Response Point 1Point 2 Point 1Point 2 11

12. Point 1: Winner Rank Selection The least squares method is also available. Bayes ML Likelihood 12

13. Point 2: Reference Matrix Update The nodes of the ranks nearer to the winner are updated to become closer to the input data h: tension α: size of tension σ: region size of learning propagation 13

14. Analysis Example Geography test N5000 n35 Median17 Max35 Min2 Range33 Mean16.911 Sd4.976 Skew0.313 Kurt-0.074 Alpha0.704 14

15. IRP of Item 25 IRP of Item 14 Item Reference Profile (IRP) 15

16. IRPs of Items 1–15 (ML, Q=10) The monotonic increasing constraint can be imposed on the IRPs in the learning process. 16

17. IRP of Items 16–35 (ML, Q=10) 17

18. IRP index (1) Item Difficulty Beta –Rank stepping over 0.5 B –Its value Kumagai (2007) 18

19. IRP index (2) Item Discriminancy Alpha –Smaller rank of the neighboring pair with the biggest change A –Its value 19

20. IRP index (3) Item Monotonicity Gamma –Proportion of neighboring pairs with negative changes. C –Their sum 20

21. ITEMR1R2R3 ・・ ・ R8R9R10AαBβCγ 1 0.26 2 0.2570.255 ・・ ・ 0.4160.4600.4970.04480.49710-0.007 0.22 2 2 0.27 1 0.2550.240 ・・ ・ 0.3190.3200.3170.02550.31710-0.033 0.33 3 3 0.59 7 0.6240.669 ・・ ・ 0.8560.8670.8800.05740.59710.000 4 0.21 0 0.2040.202 ・・ ・ 0.4600.5390.5920.08470.5399-0.009 0.22 2 5 0.22 7 0.2190.214 ・・ ・ 0.3190.3900.4450.07180.44510-0.013 0.22 2 6 0.74 7 0.7840.836 ・・ ・ 0.9140.9210.9280.05220.74710.000 0.11 1 7 0.35 2 0.3260.296 ・・ ・ 0.4390.4400.4360.05150.43610-0.066 0.44 4 8 0.22 9 0.2340.238 ・・ ・ 0.4900.5930.6670.10480.59390.000 9 0.44 4 0.4910.562 ・・ ・ 0.7780.8020.8160.07120.56230.000 10 0.28 7 0.2540.210 ・・ ・ 0.5480.6480.7190.11260.5488-0.094 0.33 3 32 0.18 9 0.1700.157 ・・ ・ 0.3020.3320.3600.04250.36010-0.032 0.22 2 33 0.16 8 0.1880.221 ・・ ・ 0.3330.3760.4140.04480.414100.000 34 0.40 7 0.4130.424 ・・ ・ 0.5660.5850.5930.03660.53570.000 35 0.48 1 0.5220.569 ・・ ・ 0.7190.7650.7940.05170.52220.000 Item Reference Profile EstimateIRP indices 21

22. Can-Do Table (example) IRP estimates IRP indices Ability category and item content 22

23. Test Reference Profile (TRP) Weakly ordinal alignment condition –Satisfied when the TRP is monotonic, but not every IRP is monotonic. Strongly ordinal alignment condition –Satisfied when all the IRPs are monotonic.  TRP is monotonic. The scale is not ordinal unless at least the weak condition is satisfied. Weighted sum of the IRPs Expected score of each latent rank 23

24. Model-Fit Indices ML, Q=10ML, Q=5 Fit indices are helpful in determining the number of latent ranks. 24

25. Bayes ML Likelihood Latent Rank Estimation Identical to the winner rank selection 25

26. Latent Rank Distribution (LRD) LRD is not always flat Examinees are classified according to the similarity of their response patterns. 26

27. Stratified Latent Rank Distribution LRD stratified by sexLRD stratified by establishment 27

28. Relationship between Latent Ranks and Scores R-S scatter plot –Spearman’s R=0.929 R-Q scatter plot –Spearman’s R=0.925 Validity of the NTT scale 28

29. Rank Membership Profile (RMP) Posterior distribution of latent rank to which each examinee belongs RMP 29

30. RMPs of Examinees 1 – 15 (Q=10) 30

31. Extended Models Graded Neural Test Model (RN07-03) –NTT model for ordinal polytomous data Nominal Neural Test Model (RN07-21) –NTT model for nominal polytomous data Batch-type NTT Model (RN08-03) Continuous Neural Test Model Multidimensional Neural Test Model 31

32. Graded Neural Test Model Boundary Category Reference Profiles of Items 1 – 9 Dashed lines are observation ratio profiles (ORP) 32

33. Graded Neural Test Model Boundary Category Reference Profiles of Items 1 – 9 Dashed lines are observation ratio profiles (ORP) 33

34. Nominal Neural Test Model Item Category Reference Profiles of Items 1 – 16 * correct choice, x merged category of choices with selection ratios of less than 10% 34

35. Discussion Test standardization theory –Self-Organizing Map –Latent scale is ordinal –IRPs are flexible and nonlinear Test editing CBT and CAT Test equating –Concurrent calibration Application –Japan’s National Achievement Test for 6 th and 9 th graders 35

36. Website http://www.rd.dnc.ac.jp/~shojima/ntt/index.htm Software –Neutet Developed by Professor Hashimoto (NCUEE) Available in Japanese and English versions –EasyNTT Developed by Professor Kumagai (Niigata Univ.) Japanese version only 36