1. A Multi-Dimensional PSER Stopping Rule
Richard Neapolitan, Northwestern University
Scott Morris, Illinois Institute of Technology
Mike Bass, Northwestern University
Matthew Lauritsen, Illinois Institute of Technology
Ideas in Testing Conference (October, 2017)
This work was supported by the National Library of Medicine of the National Institutes of Health, R01LM

2. Computer Adaptive Testing (CAT)
Initial Estimate
Prior trait distribution: N(0,1)
Based on previous responses, administer the most informative question
Based on current response, update posterior trait distribution
Point estimate
(expected a posteriori (EAP) or maximum a posteriori (MAP)
SE (Standard error / deviation) of posterior distribution

3. Standard Error (SE) Stopping Rule
Stop when SE falls below cutoff, usually 0.3.
Advantage
When item pool information function is relatively flat, yields similar measurement precision across trait levels
Disadvantages
Administers unnecessary items to examinees for whom the pre-determined SE cannot be reached
Limits precision for examinees for whom additional questions may further decrease SE

4. PROMIS Patient Reported Outcomes
Symptom severity
Health-related quality of life
Patient Reported Outcomes Measurement Information System (PROMIS)
Funded by NIH to improve assessment of self-reported symptoms and other health-related quality of life domains
3 domains: physical, mental and social health

5. Promis Anxiety Bank Sample Question:
In the past 7 days I had difficulty calming down
1 = Never 2 = Rarely 3 = Sometimes 4 = Often 5 = Always

6. PROMIS Anxiety Bank

7. For traits like anxiety the questions in the bank are aimed at differentiating among levels of anxiety
There are few, if any questions, that differentiate among individuals that don’t have anxiety
For individuals who do not have anxiety, the CAT will repeatedly ask questions without bringing down the SE
The SE will never fall below the cutoff (0.3)
All 28 questions will be asked
If the subject did not have anxiety before the test, they might after being subjected to 28 such questions

8. PROMIS ameliorates this problem by asking a maximum of 12 questions
A formal rule for determining when it is pointless to ask more questions should perform better than this ad hoc rule

9. Choi et al. (2010) developed the Predicted
Standard Error Reduction (PSER) Stopping Rule
θ is the trait
We are taking the minimum over all items X

10. PSER Algorithm
Parameter hyper allows us to ask more questions if we can
expect substantially more information by asking another question
Parameter hypo allows us to exit if we cannot expect to get a
minimum amount of information by asking another question

11. Simulation Sampled true θ values from empirical trait distribution
Generate simulated response to each item according the IRT model and the true θ
Administer CAT
Save # items and RMSE when each stopping rule is met
SE < .03 or # Items = (PROMIS rule)
Hypo = 0, .005, .01, .015, .02, .025, .03
Hyper = Hypo 0, .005, .01, .015

12. Results

13. Multivariate Computer Adaptive Testing (MCAT)
In MCAT we simultaneously assess correlated traits
The answers provide us with information about all the traits simultaneously
PROMIS has the correlated traits anxiety, depression, anger
We implemented an MCAT system for PROMIS and applied it to these 3 traits
In simulations our MCAT decreased the test length by 50% relative to unidimensional CAT

14. The stopping rule used in the simulations was to to stop when all 3 SEs were < 0.3 or 24 questions were asked
A formal PSER for MCAT should perform better than this ad hoc rule
We developed the Multivariate PSER

15. Sum SEs or Sum Variances?
In univariate CAT we choose the item expected to minimize the SE
This is the same as choosing the item expected to minimize the variance
In MCAT we could minimize the sum of the SEs or the sum of the variances
I illustrate the situation for two traits.
The sum of variance method minimizes the following:
The sum of the SEs minimizes the following:

16. Suppose one item is expected to bring the SEs for the two traits down
to 6 and 1, and another item is expected to bring them down to 4 and 4
Minimize sum of the SEs:
We choose Item1, which does not bring down the first SE very much
Minimize sum of the variances:
We choose Item 2

17. The Multivariate PSER

18. We are currently running simulations to evaluate the MPSER

19. MSPER Simulation Outcomes 3-Dimensional CAT 1000 Simulated examines
PROMIS Anxiety, Depression & Anger Banks (79 Items)
1000 Simulated examines
Theta distribution based on representative sample of the general US population
Items selected to minimize sum of posterior variance
Stopping Rule
MPSER using sum of variance
Hypo: 0, .005, .01, .015, .02, .025, .03, .035, .04, .045, .05, .055, .06
Hyper: 0, .005, .01, .015, .02, .025, .03, ∞
Traditional: Stop when SE < .3 on all traits
Outcomes
Average number of items administered
RMSE across all traits

20. Simulation Results (θ < 0)

21. Simulation Results (θ > 0)

22. Simulation Results (All θ)