Constructing a Test We now know what makes a good question: Valid - it tests the thing it sets out to test Reliable - it delivers consistent results for students of similar ability Clear - Direct and concise - the question uses the simplest language appropriate to the task and is direct in what it wants the student to do - Unambiguous marking guide - the marking guide makes it easy for the teacher to see what should be credited and what should not. How might it be possible to have written nine good questions that meet all of these criteria but when you put them together they would not add up to a good test? One could say that a test is like a cake and the questions are the ingredients. If you have a key ingredient missing, or you add too much of one and not enough of another, or mix them in the wrong order the cake will be spoiled. Restricted

Constructing a Test Things to consider when planning a test Does it test the whole unit? Check the test against the assessment criteria for the unit. Is the test balanced? The number of questions or marks should reflect the weighting of the learning objectives. Is the test at an appropriate level of difficulty? There should be a range of question difficulty, including competences from the lower, medium and higher bands. Does the demand of the questions increase through the test? Start with the most accessible questions and finish with the most demanding. Does it test the whole unit? Imagine in Computer science you were writing a test on Boolean algebra and logic gates. You could have nine questions that were excellent individually, but if they are all on Boolean algebra and there are none on logic gates then it’s not a good test of the unit. Remember, the test as a whole should address the assessment criteria for the unit – when you have constructed your test, check what the questions add up to. Do they tell you enough about the whole of the assessment criteria? If there are too many learning objectives to assess in nine or ten questions, have you asked questions on the most important ones? Is the test balanced? While all the key parts of a unit should be represented by questions in the test it doesn’t necessarily mean that if there are two main themes, ideas or concepts in the unit that there should be even numbers of questions or marks. For example, if one of the areas of the unit is more important than the other or has more learning objectives then it is fine to have more questions on it. Remember, we suggested ticking off the leaning objectives in a unit as you wrote questions for them. Use this to help check you have covered the most important points or if there are gaps. Is the test at an appropriate level of difficulty? It may sound obvious, but a test made up of questions that nearly all the students can answer, or hardly any of the students can answer won’t give you any useful formative information. There needs to be a variety of demand in the questions across a test so you can see at what point different students start to find the questions difficult. The questions you write should be accessible to all your students, even the less able (this means they should be able to understand what they are asking and attempt them even if they get the answer wrong); but there still needs to be some questions that will challenge the most able students. One way to monitor this is by checking that the test contains a range of different competences from across the different bands we discussed earlier: Lower (knowledge), Medium (understanding and applying) and Higher (analysing, reviewing and creating) Does the demand of the questions increase through the test? The order of the questions is important- even though these are formative tests and not high stakes, students may be nervous and unfamiliar with the e-assessment format. We don’t want that to affect the student’s performance so it is best where possible to start with more straightforward questions so they can get used to the format and build their confidence. Restricted

Constructing a Test Things to consider when planning a test Are there a range of question formats? Try to avoid having a run of questions with the same format. Are the questions distinct? Every question should assess something unique within the unit and should not give away the answer to another question. Do the questions address any of the generic competencies? For example do they require critical or problem solving skills? Are there a range of question formats? We have discussed choosing the best question format to suit what is being assessed- if there isn’t a range then it may be a sign that you are not taking full advantage of the different types available. A variety of question types is also a good way of keeping the students engaged and stopping them guessing (e.g. if you have too many multiple choice questions in a row, they might start looking for patterns – “the last two answers were a, so it can’t be a again this time”. Are the questions distinct? While it is ok to have several questions on one area of the unit (e.g. in a unit for physics you might have several questions on projectile motion, but they should be assessing different aspects of it – one question projectile motion related to linear and another on projectile motion in vertical components). If questions are too similar they won’t give you any additional formative information- if a student can do one, they can probably do the other, if they can’t do one, they probably can’t do the other. There will be a limited number of questions in the test so we have to make sure they maximise the coverage of the unit and what they tell us about what the student can do and where they need to improve. Another thing to check when you are putting the test together is questions which may give the answer to another question away- for example If one question asks ‘When was the treaty of Versailles signed?’ you do not want a later question to ask something like ‘When the first world war ended in 1919 with the treaty of Versailles, what were the main outcomes for Germany?- these questions are often called ‘enemies’ Do the questions address any of the generic competencies? The generic competences are an important part of the CBC and shouldn’t be forgotten- it’s not possible to assess all of them in an online test (we can only test their co-operation in group work or their lifelong learning over time) but it is possible that questions in the test may give some evidence for communication (can they use appropriate language and relevant vocabulary); research (can they use information to answer questions); Creativity (use imagination and generate new ideas) or critical or problem solving skills. Remember- no matter how good the individual questions are, if they are not selected and sequenced appropriately or don’t cover the most important parts of the unit they will lead to a poor test. When writing a question, you should be asking yourself- what does this contribute to the test as a whole? What information will it give me that I can use formatively? When constructing the test you should be asking yourself- what do the questions add up to? Do they adequately reflect the unit and its assessment criteria? Restricted

Constructing a Test Task: Planning your own test 1. Review the questions you have written so far using the criteria we have just discussed: What learning objectives have you not yet covered? What question formats have you not yet tried? What competences have you not yet addressed (Lower, medium and higher)? 2. Make a note of any gaps that need to be filled to good test of the unit. 3. Plan the test- nine questions what have you got already? What questions are needed to fill gaps? What order should the questions come in 4. Start writing the questions. Restricted