ICCS Field Trial Marking of open-ended response items Hamburg, July 2007

ICCS Marker Training, Hamburg July 2007 Purpose of Marker Training 1.To develop a common, shared understanding of the purpose of each item and the corresponding item codes for scoring student responses. 2.To confirm the procedures for setting up and running a marking operation. –The aim for us is that after 1+n days there are shared common understandings of these two aspects Day 1 – confirming the scoring of the open-ended response items Day 2 – confirming the marking operations procedures

ICCS Marker Training, Hamburg July 2007 Why are there open-response questions? (CIVED only MCQ) Open-response questions provide: –Information about broader conceptual understandings (e.g. citizenship motivations, benefits to community) –Information about students’ capacity to generate rather than recognise concepts (useful in CCE where correct responses can often stand-out) Still partly exploratory

ICCS Marker Training, Hamburg July 2007 The Open-response item scoring guides Crucial aspects: - the descriptions of acceptable responses - the scored sample responses

ICCS Marker Training, Hamburg July 2007 Considerations when using the guides Spelling, quality of expression, grammar and punctuation are not assessed. HOWEVER – Civics and Citizenship Education provides an unusual set of challenges when considering the language of student responses.

ICCS Marker Training, Hamburg July 2007 Considerations when using the guides In many countries students have not had extensive formal classroom experience of CCE. Many, many students do not have the explicit CCE vocabulary to express conceptual understandings efficiently. One significant challenge in assessing written student responses is to ‘unpack’ student attempts to express conceptual understandings using their ordinary (i.e. non-technical) language.

ICCS Marker Training, Hamburg July 2007 Considerations when using the guides This challenge has two dangers that can affect marker reliability. 1.Some markers can mistakenly believe that because students do not have the language skill to express answers succinctly that all answers are correct. 2.Some markers believe that unless students use appropriate CCE language they cannot be credited with providing correct responses.

ICCS Marker Training, Hamburg July 2007 Considerations when using the guides BOTH misconceptions must be dealt with in the marker training. Student responses can only be scored on the basis of what they have written – markers should not try to infer student responses on the basis of what has not been written. To this end, the students’ choice of the words they use MUST be given careful attention.

ICCS Marker Training, Hamburg July 2007 Considerations when using the guides For example –What are two different benefits of people discussing issue even if they may never agree with each other’s opinions? (CI2RR01) –‘They may learn to agree with each other’ (0) –‘They may learn to get along with each other’ (1) –‘They might end up happy’. (0) –The first and third responses are too vague to receive credit (and the first response contradicts the stem).

ICCS Marker Training, Hamburg July 2007 Considerations when using the guides Many students provide ‘feel-good’ responses (expressions of value rather than understanding) that can be very well expressed but still without substance. Many students provide complex re- wordings of the question stem without answering the question – these can be deceptive as often they are eloquent.

ICCS Marker Training, Hamburg July 2007 Questions or comments?

ICCS Marker Training, Hamburg July 2007 Comments It doesn’t matter where the students write their responses (1, 2) We will introduce a ‘spare’ code for response types that are not accounted for at all in the scoring guide but are worthy of credit. We will detail the way to use this in the SOP(III). You will be asked to document how the codes work for consideration in the main study.