Ülle Türk Estonian National Defence College Managing the test construction process STANAG 6001 Testing Conference SHAPE, Mons, Belgium September 2013

Processes in testing The test development process The test assembling process The test delivery process Marking, grading and reporting results Monitoring and review 2

The test development process Try-out Informing stakeholders Decision to provide test PlanningDesignTry-out Informing stake- holders Final test specifcat- ions 3

The test assembling process Test specifi- cationas Producing materials Quality control Constructing tests Live test materials 4

The test delivery process Live test materials Arranging venues Registering test takers Preparing materials Administering the test Returning the materials Returned materials 5

Marking, grading and reporting results Returned test materials MarkingGrading Reporting of results 6

Monitoring and review The purpose is to establish whether the test is working properly and, if not, what needs to be changed. Monitoring is part of the routine process of producing and using a test Periodic reviews are done occasionally, outside of the regular operation of a test 7

The test assembling process Test specifi- cationas Producing materials Quality control Constructing tests Live test materials 8

Timeline Testing session 1 (February-March) Setting grade boundaries Analysing the task and item performance Producing new tasks Initial review Improving tasks Second review Testing session 2 (June-July) Improving tasks Third review 9

Item review questions Is the task clear in each item? Is it likely that the person attempting an item will know what is expected? Are the items expressed in the simplest possible language? Is each item a fair item for assessment at the level tested? Is the wording appropriate to the level tested? Are there unintended clues to the correct answer? Is the format reasonably consistent so that test-takers know what is required from item to item? Is there a single clearly correct (or best) answer for each item? Is the type of item appropriate to the information required? Are there statements in the items which are likely to offend? Is there content which reflects bias on gender, racial, or other grounds? Are there enough representative items to provide an adequate sample of the behaviours to be assessed? 10

Timeline (continued) Compiling the pre-test Producing pre-test materials Pre-teting Testing session 3 (November-December) Analysing pre-test results Compiling the live tests Producing the live test materials Administering the new test 11

Sources ALTE Manual for Language Test Development and Examining for Use with the CEFR. Strasbourg: Language Policy Division, Council of Europe, Izard, John. Overview of Test Construction. Quantitative Research Methods in Educational Planning, Module 6. UNESCO International Institute for Educational Planning,