1 The Development of A Computer Assisted Design, Analysis and Testing System for Analysing Students’ Performance Q. He & P. Tymms, CEM CENTRE, UNIVERSITY OF DURHAM, UK Qingping He and Peter Tymms (CEM Centre, University of Durham, UK)

2 The Context Classic and Modern Test Theories Limitations of Classic Test Theory Advantages of Item Response Theory (IRT) Progress in Computer Assisted Assessment Which Incorporates Test Theories Existing Computer Software Systems Expensive Expertise Functionalities Q. He & P. Tymms, CEM CENTRE, UNIVERSITY OF DURHAM, UK

3 The Aim Q. He & P. Tymms, CEM CENTRE, UNIVERSITY OF DURHAM, UK Develop A System That is Easy and Economic for Schools (and other test organisations) to Use and Can Provide a Range of Useful Functionalities

4 Build Items and Construct Item Banks Design Tests Effectively (both Classic and IRT Model – based Tests) Analyse Test Results – Detailed Diagnostic Analysis of Students’ Performance: Individual Student Level School/Class level Analyse Test Items and Equate Different Tests (Rasch) Conduct Classic and IRT- based Tests Including Computer Adaptive Tests (CATs) on Computers CADATS – Computer Assisted Design, Analysis and Testing System Q. He & P. Tymms, CEM CENTRE, UNIVERSITY OF DURHAM, UK

5 The Architecture Item and Bank Building, Test Designing and Results Analysis Component Testing Component Test XML Files Response XML Files Q. He & P. Tymms, CEM CENTRE, UNIVERSITY OF DURHAM, UK

6 System Implementation Both Components Have Been Developed as A Microsoft Excel Visual Basic for Application Project Embedded with Macromedia Flash Player for Easy Use and Provision of Intuitive Graphic Representation of Test Results Q. He & P. Tymms, CEM CENTRE, UNIVERSITY OF DURHAM, UK

7 Item and Bank Building, Test Designing and Results Analysis Component – Excel Workbook: The Main Menu Q. He & P. Tymms, CEM CENTRE, UNIVERSITY OF DURHAM, UK

8 Writing Items and Building an Item Bank

9 Designing Tests Effectively

10 Analysing Test Results: Report on Individual/School Overall Scores + More

11 Item Calibration and Equating Different Tests

12 The Test Delivery Component – Excel Workbook: The Test Interface Q. He & P. Tymms, CEM CENTRE, UNIVERSITY OF DURHAM, UK

13 Q. He & P. Tymms, CEM CENTRE, UNIVERSITY OF DURHAM, UK Analysis of Students’ Performance – The Hong Kong Year 7 Baseline Maths Test Case Study Hong Kong Schools Involved in Middle Years Information System (MidYIS) Project The Year 7 Baseline Test For Hong Kong Schools Analysing Maths Test Results for Students from 5 Schools in Hong Kong Taking the Baseline Test in 2003

14 Q. He & P. Tymms, CEM CENTRE, UNIVERSITY OF DURHAM, UK Maths Questions: Different Subject Areas

15 Q. He & P. Tymms, CEM CENTRE, UNIVERSITY OF DURHAM, UK Item Difficulty and Student Ability Distributions: All 5 Schools

16 Individual Level: Individual Performance and Diagnosis Analysis Q. He & P. Tymms, CEM CENTRE, UNIVERSITY OF DURHAM, UK

17 Individual Level: Individual Performance and Diagnosis Analysis Q. He & P. Tymms, CEM CENTRE, UNIVERSITY OF DURHAM, UK

18 Individual Level: Individual Performance and Diagnosis Analysis Q. He & P. Tymms, CEM CENTRE, UNIVERSITY OF DURHAM, UK

19 Individual Level: Individual Performance and Diagnosis Analysis – Comparison within School Q. He & P. Tymms, CEM CENTRE, UNIVERSITY OF DURHAM, UK

20 Individual Level: Individual Performance and Diagnosis Analysis – Comparison with A Norm (e.g. National Average) Q. He & P. Tymms, CEM CENTRE, UNIVERSITY OF DURHAM, UK

21 School Level: School/Class Performance and Diagnosis Analysis- Comparison with A Norm (e.g. National Average) Q. He & P. Tymms, CEM CENTRE, UNIVERSITY OF DURHAM, UK

22 Conclusions The system is easy to use and provides a variety of functions The system can be used to generate information on the performance of students at both individual and school/class levels and the performance of test items The information generated by the system can be used to identify curriculum areas where students are under performing The use of the system will help schools improve students’ performance Q. He & P. Tymms, CEM CENTRE, UNIVERSITY OF DURHAM, UK