How well did the assessment task do what we wanted it to do? Janina Drazek Manager — Assessment & Comparability, QCAR Queensland Studies Authority

Skilled diagnosticians, i.e. teachers diagnose and assess against declared standards. Central role of the teacher

Diagnosticians require reliable instruments designed to produce valid information. ASSESSMENT is the teacher’s instrument.

What is assessment? Assessment is the purposeful, systematic and ongoing collection of information as EVIDENCE for use in making judgments about student learning. »(Education Queensland 2001, Years 1-10 Curriculum framework for Education Queensland schools, Department of Education, Brisbane, p.13)

To develop valid and reliable tasks three reflective steps are required : 1.Evaluation 2.Modification 3.Trialling

When and how do I EVALUATE? ALL the time. In particular, 1.once a task has been created do the task yourself. 2.once you trial the task with learners—give them the possibility of commenting on the task. Using all the information you have gained modify the task or reject entirely.

Q. BUT I did all that, now I want to know how do I evaluate whether the task worked? A. Evaluate by applying exactly the same principles used for the development of quality assessments.

Sound assessment design Selecting a proper method to: suit the purpose and the target elicit the right performance or product provide enough evidence to support the decision avoids sources of bias.

Evaluating Did they enjoy the task? Was it easy/difficult? Did they understand what to do? How well did they think they responded to the task? Ask focused questions about how they did the various sections of the task. Ask the students

Evaluating Mark the responses according to the criteria and standards matrix (schema, rubric etc) Note the range of standards in the responses (consider and analyse) Do the responses reflect your indicative response? If not, why? Look at the evidence (student work)

Evaluating C redibility–valid and reliable assess what it purports to assess I ntellectual rigor focus on higher order thinking A uthenticity have purpose U ser friendliness feasible, appropriate and engaging fair and equitable Review the task against the principles of quality assessment

Credibility Does the task generate sound evidence about student performance? Are the criteria explicit? Are the standards clearly stated? Is it clearly aligned with the targeted intentions? Are the conditions of assessment clear? How fair and equitable to all students? Does it provide opportunity for students to demonstrate what they know and can do?

Evaluating Did they reflect what is valued in the assessment? How were they organised? Were they organised under: Knowing and understanding; Inquiring; Responding; Reflecting (or some other constructs)? Look at the number of criteria – too many or too few? Are they sufficiently different from, and independent of, each other? Did the students understand what was expected? Criteria

Evaluating Standards Descriptors cover a range of performances on a particular criterion are sufficient in number to enable differentiation of performances can be seen in student work are written in positive terms are written in language that is suitable for students

Intellectual quality Is the task intellectually challenging? draws in sufficient depth & breadth upon the targeted knowledge, concepts and skills engages students in a range of thinking skills encourages students to demonstrate critical analysis has clear cognitive expectations

Authenticity Does the task address a realistic and complex problem? has an appropriate real life context for all students is engaging and motivating has genuine and valued purpose has a context that is appropriate for the cognitive demands

User friendliness Does the task clearly communicate to students what is needed for producing their best performance? has an accessible & appropriate format has layout, cues, visual design, format and choice of words that clarify what is expected has examples and resources that are helpful to students provides a complete set of guidelines (including models) that allow students to reflect on, rehearse and review their responses

Alignment of Assessment & Curriculum 1.Intent (What do we want students to learn?) 2.Enactment (What are the appropriate learning experiences?) 3.Evidence (What are the contexts in which students demonstrate their learning?) 4.Judgment (What is being valued?) 5.Moderation (Is there consistency in the way we apply standards?)

Effective assessment design depends upon a clear and complete explication of the claims one wishes to make on the basis of assessment, the evidence needed to support those claims, the student behaviour that will comprise that evidence, and the tasks needed to elicit those behaviours. That chain of reasoning, once established and documented becomes part of the validity argument in support of the assessment (Matters, 2006:14).

QCAR contacts Website: