1 An Investigation of The Response Time for Maths Items in A Computer Adaptive Test C. Wheadon & Q. He, CEM CENTRE, DURHAM UNIVERSITY, UK Chris Wheadon and Qingping He (CEM Centre, Durham University, UK)

2 Background The CEM Centre’s Baseline Assessments on Primary and Secondary School Students The CEM Baseline Tests for Secondary Schools (for students aged 11-18: three separate projects) Paper based, Efficiency, Cost Use of CTT and IRT in Computerised Testing and to Interpret Test Data To Develop A Computerised Test to Replace the Existing Paper-based Baseline Tests for the Three Secondary Projects C. Wheadon & Q. He, CEM CENTRE, DURHAM UNIVERSITY, UK

3 CABT comprises a calibrated item bank and a question display system The adaptive test contains an adaptive maths test and an adaptive English vocabulary test Testing is delivered through the Web or from the school’s local area network CABT-The CEM Computer Adaptive Baseline Test C. Wheadon & Q. He, CEM CENTRE, DURHAM UNIVERSITY, UK

4 Establishing A Calibrated Item Bank C. Wheadon & Q. He, CEM CENTRE, DURHAM UNIVERSITY, UK Testing of items by school students from different year groups through the administration of a series of tests containing common items Calibration of items in each test using the Rasch model (items meeting the Rasch requirements are used) Setting of the reference test and equating of different tests using the Rasch model and common items Further new item calibration through the embedding of new items in the adaptive test

5 Developing The Question Display System - Conducting Adaptive Testing C. Wheadon & Q. He, CEM CENTRE, DURHAM UNIVERSITY, UK Realisation of adaptivity is achieved through the implementation in the question display system of the Rasch model for ability estimation (MLE method is used) and question selection Rules for stopping test: minimum number of question; maximum number of questions; and convergence value Variable starting difficulty for the first question for different year groups.

6 C. Wheadon & Q. He, CEM CENTRE, DURHAM UNIVERSITY, UK Preliminary Results from the Adaptive Maths Test There are over 500 items in the maths item bank Effort has been made to make the items as curriculum- free as possible Items are content-independent to each other Item types include MCQ, short free text entry questions and interactive questions Items cover a wide range of difficulties in order for the three secondary projects (for three different year groups) to use the same adaptive test as their baseline test

7 C. Wheadon & Q. He, CEM CENTRE, DURHAM UNIVERSITY, UK Individual Student: The Distribution of Ability and the Distribution of Item Response Time

8 C. Wheadon & Q. He, CEM CENTRE, DURHAM UNIVERSITY, UK Individual Student: The Distribution of Ability and the Distribution of Item Response Time

9 C. Wheadon & Q. He, CEM CENTRE, DURHAM UNIVERSITY, UK All Items: The Effect of Item Difficulty on Response Time

10 C. Wheadon & Q. He, CEM CENTRE, DURHAM UNIVERSITY, UK A Selection of Items: The Effect of Age of Test Takers on Item Response Time

11 C. Wheadon & Q. He, CEM CENTRE, DURHAM UNIVERSITY, UK A Selection of Items: The Relationship between Response Time for Correct Answers and Response Time for Incorrect Answers

12 C. Wheadon & Q. He, CEM CENTRE, DURHAM UNIVERSITY, UK Individual Item: The Effect of Ability and Age on Response Time (Difficulty: -0.6 logits) – Easy Item

13 C. Wheadon & Q. He, CEM CENTRE, DURHAM UNIVERSITY, UK

14 C. Wheadon & Q. He, CEM CENTRE, DURHAM UNIVERSITY, UK Ability Bands within a Year Group for the Specific Item (Year 7)

15 C. Wheadon & Q. He, CEM CENTRE, DURHAM UNIVERSITY, UK Individual Item: The Effect of Ability and Age on Response Time (Difficulty: 1.1 logits) – Medium- difficulty Item

16 C. Wheadon & Q. He, CEM CENTRE, DURHAM UNIVERSITY, UK

17 C. Wheadon & Q. He, CEM CENTRE, DURHAM UNIVERSITY, UK Ability Bands within a Year Group for the Specific Item (Year 10)

18 C. Wheadon & Q. He, CEM CENTRE, DURHAM UNIVERSITY, UK Individual Item: The Effect of Ability and Age on Response Time (Difficulty: 2.9 logits) – Difficult Item

19 C. Wheadon & Q. He, CEM CENTRE, DURHAM UNIVERSITY, UK

20 C. Wheadon & Q. He, CEM CENTRE, DURHAM UNIVERSITY, UK Ability Bands within a Year Group for the Specific Item (Year 12)

21 Conclusions and Further Work Response time for all the maths items in the test generally increases with item difficulty but shows great variability Item difficulty levels and the age and ability of test takers have significant influence on item response time The information obtained in this study can be used for constructing more efficient tests Further work will involve investigating the relationship between CABT results and students’ subsequent academic performance C. Wheadon & Q. He, CEM CENTRE, DURHAM UNIVERSITY, UK