Estimation of Ability Using Globally Optimal Scoring Weights Shin-ichi Mayekawa Graduate School of Decision Science and Technology Tokyo Institute of Technology

2 Outline Review of existing methods Globally Optimal Weight: a set of weights that maximizes the Expected Test Information Intrinsic Category Weights Examples Conclusions

3 Background Estimation of IRT ability  on the basis of simple and weighted summed score X. Conditional distribution of X given  as the distribution of the weighted sum of the Scored Multinomial Distribution. Posterior Distribution of  given X. h(  x)  f(x|  ) h(  )  Posterior Mean(EAP) of  given X.  Posterior Standard Deiation(PSD)

4 Item Score We must choose w to calculate X. IRF

5 Item Score We must choose w and v to calculate X. ICRF

6 Conditional distribution of X given  Binary items Conditional distribution of summed score X.  Simple sum: Walsh(1955), Lord(1969)  Weighted sum: Mayekawa(2003) Polytomous items Conditional distribution of summed score X.  Simple sum: Hanson(1994), Thissen et.al.(1995)  With Item weight and Category weight: Mayekawa & Arai(2007)

7 Example Eight Graded Response Model items 3 categories for each item.

8 Example (choosing weight) Example: Mayekawa and Arai (2008) small posterior variance  good weight. Large Test Information (TI)  good weight

9 Test Information Function Test Information Function is proportional to the slope of the conditional expectation of X given  (TCC), and inversely proportional the squared width of the confidence interval (CI) of  given X. Width of CI Inversely proportional to the conditional standard deviation of X given .

10 Confidence interval (CI) of  given X

11 Test Information Function for Polytomous Items ICRF

12 Maximization of the Test Information when the category weights are known. Category weighted Item Score and the Item Response Function

13 Maximization of the Test Information when the category weights are known.

14 Maximization of the Test Information when the category weights are known. Test Information

15 Maximization of the Test Information when the category weights are known. First Derivative

16 Maximization of the Test Information when the category weights are known.

17 Globally Optimal Weight A set of weights that maximize the Expected Test Information with some reference distribution of . It does NOT depend on .

18 Example NABCT A B1 B2 GO GOINT A AINT Q Q Q Q Q Q Q Q LOx LO GO GOINT A AINT CONST

19 Maximization of the Test Information with respect to the category weights. Absorb the item weight in category weights.

20 Maximization of the Test Information with respect to the category weights. Test Information Linear transformation of the category weights does NOT affect the information.

21 Maximization of the Test Information with respect to the category weights. First Derivative

22 Maximization of the Test Information with respect to the category weights. Locally Optimal Weight

23 Globally Optimal Weight Weights that maximize the Expected Test Information with some reference distribution of .

24 Intrinsic category weight A set of weights which maximizes: Since the category weights can be linearly transformed, we set v0=0, ….. vmax=maximum item score.

25 Example of Intrinsic Weights

26 Example of Intrinsic Weights h(  )=N(-0.5, 1): v0=0, v1=*, v2=2

27 Example of Intrinsic Weights h(  )=N(0.5, 1): v0=0, v1=*, v2=2

28 Example of Intrinsic Weights h(  )=N(1, 1 ): v0=0, v1=*, v2=2

29 Summary of Intrinsic Weight It does NOT depend on , but depends on the reference distribution of  : h(  ) as follows. For the 3 category GRM, we found that For those items with high discrimination parameter, the intrinsic weights tend to become equally spaced: v0=0, v1=1, v2=2 The Globally Optimal Weight is not identical to the Intrinsic Weights.

30 Summary of Intrinsic Weight For the 3 category GRM, we found that The mid-category weight v1 increases according to the location of the peak of ICRF. That is: The more easy the category is, the higher the weight. v1 is affected by the relative location of other two category ICRFs.

31 Summary of Intrinsic Weight For the 3 category GRM, we found that The mid-category weight v1 decreases according to the location of the reference distribution of  h(  )  If the location of h(  ) is high, the most difficult category gets relatively high weight, and vice versa. When the peak of the 2nd category matches the mean of h(  ), we have eqaully spaced category weights: v0=0, v1=1, v2=2

32 Globally Optimal w given v

33 Test Information LOx LO GO GOINT CONST

34 Test Information

35 Bayesian Estimation of  from X

36 Bayesian Estimation of  from X

37 Bayesian Estimation of  from X (1/0.18)^2 =

38 Conclusions Polytomous item has the Intrinsic Weight. By maximizing the Expected Test Information with respect to either Item or Category weights, we can calculate the Globally Optimal Weights which do not depend on . Use of the Globally Optimal Weights when evaluating the EAP of  given X reduces the posterior variance.

39 References

40 ご静聴 ありがとう ございました 。 Thank you.

41

42

43