A Hierarchical Framework for Modeling Speed and Accuracy on Test Items Van Der Linden

Outline Introduction of speed-accuracy tradeoff Current Models Discussion of Models General Hierarchical Framework Priors Distributions Empirical Example Discussion

Introduction of speed-accuracy tradeoff working faster with lower accuracy or more slowly with higher accuracy

Current models M1: M2: M3: M4: Weibull distribution M5:

Discussion (1) the within-person level, at which the value of the person parameters is allowed to change over time (2) the fixed-person level, at which the parameters remain constant; (3) the level of a population of fixed persons, for which we have a distribution of parameter values across persons M1 & M2: Confound the within-person and the fixed-person level.

(4) speed parameters and time parameters should be separated & incorporated M2: directly equate speed with the response time M3 & M5: contain time parameters M1, M2 & M4: no time parameters

General Hierarchical Framework Key assumptions (a) a test taker operates at a fixed level of speed & accuracy; (b) for a fixed test taker, the response and the time on an item are random variables; (c) separate item and person parameters for the distribution of the response on the items and the time (d) given the person’s ability and speed, the responses and time are conditional independent (e) model the relations between speed and accuracy for a population of test takers separately from the impact of these parameters on the responses and times of the individual test takers

Levels of Modeling First-level models: a response model and a response-time model (3PNO) (lognormal) Joint distribution (locally dependent given a person’s ability and speed)

Second-level model: For person parameters ( ), it was assumed that the and are multivariate normal distributed.

Likewise, for item parameters ( ), it was assumed that,,,, and are multivariate normal distributed

For the full model, the sampling distribution becomes

Identifiablity Alternative models response model: use IRT models response time model: exponential model, Weibull distribution, constrain some parameters for the second model

Priors Distribution Posterion distribution

Empirical Example 1104 test takers on 96 items in the computerized CPA examination. The items were shown to have a good fit to the 3PL logistic model and the response times to the lognormal model.

Results:, indicating the more able test takers tended to work faster (tradeoff between speed and accuracy)

Discussion Response time (1) improve testing routines that are traditionally based on the response only. (2) improve item selection in CAT; (3) diagnose differential speededness and thus to adjust the problem. (4) detect aberrant behavior (5) The hierarchical framework can be used to equate the result from different experiments or results from different conditions of speed.

Questions & Future Studies Scale problem for response model and response time model Use different models for response model (e.g., 3PL logistic model) and response time model Consider situations of cheating when response time is rather short…