LANGUAGES The links between the: Standards Texts Questions and Schedules

The Curriculum MoE The Curriculum MoE Achievement standards MoE Achievement standards MoE The Grade score NZQA The Grade score NZQA The hierarchy Assessment Task NZQA Assessment Task NZQA A happy student A happy student 2

The Listening standards Level 1 (1.1) Demonstrate understanding of a variety of spoken LANGUAGE texts on areas of most immediate relevance Level 2 (2.2) Demonstrate understanding of a variety of spoken LANGUAGE texts on familiar matters Level 3 (3.1) Demonstrate understanding of a variety of extended spoken LANGUAGE texts 3

The reading standards Level 1 (1.4) Demonstrate understanding of a variety of LANGUAGE texts on areas of most immediate relevance Level (2.4) Demonstrate understanding of a variety of written and/or visual LANGUAGE texts on familiar matters Level 3 (3.4) Demonstrate understanding of a variety of extended written and/or visual LANGUAGE texts 4

The standards, a colourful summary The colours indicate Grade: AchievementMerit Excellence NCEA level:Level 1 Level 2 Level 3 The listening standard Demonstrate clear / thorough understanding of a variety of extended spoken Tongan texts on areas of most immediate relevance / on familiar matters The reading standard Demonstrate clear / thorough understanding of a variety of extended written and/or visual Tongan texts on areas of most immediate relevance / on familiar matters 5

The definitions that matter For Merit Clear understanding means the relevant information ideas and/or opinions from the texts are selected and unambiguously communicated. For Excellence Thorough understanding means relevant information, ideas and/or opinions from the texts are expanded on with supporting detail to show understanding of the implied meanings or conclusions within the text. 6

A BAD TEXT and A BAD QUESTION Example of a bad text - Liz made this up, it’s not taken from a real exam Below is some information for the exchange student who is coming in a month to stay with your family and go to your school. Our family consists of five people. My mother, my father, myself, my older brother called William who is at university studying to be an accountant and my younger brother called Thomas who goes to the same school as I do. He wants to be a musician when he grows up so he can be rich and famous. I have lots of aunts and uncles and cousins, and many of them live in the Auckland like I do. We often meet up for birthdays and other family celebrations. When you come to our you will be able to choose some interesting subjects to study such as Drama, Media studies, Technology, German, French, Japanese, Māori and Music. Other classes are compulsory, for example Maths, Science, English and PE. Our school day starts at 8/45 with a Form class meeting, and finishes at 3/15. We have an hour for lunch, and can buy food at the canteen but we are not allowed to go to the shops. We have a lot of sports teams at my school. Mostly we practice at lunchtime and after school, and play against other schools on Saturday. In summer I like to play tennis, and in winter I like to play hockey. My father likes to come and watch me play sport, and afterwards we often go to a café for a delicious lunch. If you don’t like playing sport we have all sorts of other clubs such as drama, music and debating. I hope you join one of our after school clubs, you will enjoy the activities and the company of the other students. In the weekends and holidays there are lots of places you could visit with me and my family. We could go to the beaches or the hot springs, there are museums and art galleries and concerts, and many interesting small towns to visit, we could even stay on a farm and help with the animals. I hope very much you enjoy your stay in New Zealand, I am looking forward to your visit. Kim A bad question to match Describe Kim’s family Explain what the exchange student can do during the school day during class time outside of class time Describe a typical weekend for Kim. Use information from the text to support your answer. 7

Consider now: 1.What’s the problem with the text? 2.Can it be improved without too much hassle? 3.What’s the problem with the question? 4.Can it be improved without too much hassle? 5.What’s more important / difficult to fix, the text or the questions? And, going back a couple of steps: 6.What’s the function of an exam question? 7.What things characterise a good exam question? 8

The Assessment Schedule The schedule is formatted the same way for both standards at all three levels. It is intended to enable holistic judgement for each question, and avoid focussing on counting up fragmented pieces of evidence / lexical items. It is a guideline to how to make judgements, not a black and white list of ‘right’ and ‘wrong’ answers. Judgements between grades (that is N and A, A and M, M and E) must be qualitative Judgements within grades may be qualitative or quantitative Although final cut-scores may vary from exam to exam (depending on relative difficulty) adherence to the intent of the standard supersedes all other considerations. That is, final judgements must be made against the standard. 9

Grade scores Each question receives a grade first, then a score – a grade score! There are nine possible grade scores: Between grades – steps up are always qualitative. Within grades – steps up are quantitative and/or qualitative The scores are totalled once all the questions have been marked The total score is written on the front of the answer booklet 10

Top-down Marking All standards are marked using top-down marking which conforms to the fundamental principle on which standards based assessment sits. Top down marking is fairer and more valid for the candidates because: It recognises all evidence (including N1 or N2) and aggregates it across the paper. The absence of lower level evidence does not ‘knock’ a candidate out of a higher level of achievement.

The steps in top down marking 1.Markers look for Excellence evidence. If there is sufficient to indicate the candidate is performing at E level, they: –award E –decide if it is ‘a good E’ (E8) or ‘just an E’ (E7) –write the appropriate grade score in the box provided –move on to the next question. 2.If Excellence evidence is not there, they look for Merit evidence 3.If Merit evidence is not there, they look for Achievement evidence 4.If Achievement evidence is not there, they decide whether to award NØ, N1 or N2 5.Finally they total the scores and write the total on the front of the answer booklet.

Quick analysis of a test On one page, draw up a tally table like the one below. Go through the class results, one student and a time. Enter (as a tally) each student’s results for the four questions. You can now see which questions : are discriminating well seem to be too ‘hard’ or too ‘easy’ have a very strange spread. 13

Cut scores The languages exams all have four questions, therefore the highest possible total score is 32. While there is some variation possible in cut-scores, there are limits. For example: 8 must always be N, because it could be gained from N2, N2, N2 and N2. Clearly that’s not an A. Equally, 12 must be an A because it could be gained from A3, A3, A3 and A3 which cannot logically or validly be N or M. Task Work out the highest and lowest possible scores that for the valid award of A and M and E in a four question exam. 14

Cut score ranges – an almost complete illustration Why must Merit start at 17 and not 16? Achievement could start at which four scores? One of the above ‘starting/minimum’ scores is extremely unlikely. What score is that? Why is it unlikely? Which 4 scores could M and E start at? 15

The nuts and bolts of adapting * Texts * Questions * Schedules so they are valid for the new standards

The hierarchy What order do you think these should be put in? Curriculum Cut score decision Grade scores Questions Schedule Standard Student Text Top down marking Total score 17

This order Student Curriculum Standard Text Questions Schedule Top down marking Grade scores Total score Cut score decision 18

The text The text is the most important thing, it must contain: straight forward information (Achievement) material that can used to justify opinions / draw conclusions / infer / link / analyse (Merit and Excellence) Texts and questions can link in several ways: one text, one question one text, more than one question several linked texts, one question If the text is good, the questions ‘fall out’ of it easily.

The question – some guidelines The question should elicit sufficient information for the marker to be able to validly judge the level of understanding the student is demonstrating; don’t ask more just because you can. As you are writing / critiquing a question, bear in mind two students: the ‘just achieved’, literal, conscientious one the excellence one who thinks outside the box If your question can be successfully attempted by both those candidates, it will discriminate successfully at all levels. Scaffold if possible, and try to avoid very broad ‘drag net’ questions; candidates at all levels tend to respond to them by writing everything.

A BAD TEXT and A BAD QUESTION Example of a bad text - Liz made this up, it’s not taken from a real exam Below is some information for the exchange student who is coming in a month to stay with your family and go to your school. Our family consists of five people. My mother, my father, myself, my older brother called William who is at university studying to be an accountant and my younger brother called Thomas who goes to the same school as I do. He wants to be a musician when he grows up so he can be rich and famous. I have lots of aunts and uncles and cousins, and many of them live in the Auckland like I do. We often meet up for birthdays and other family celebrations. When you come to our you will be able to choose some interesting subjects to study such as Drama, Media studies, Technology, German, French, Japanese, Māori and Music. Other classes are compulsory, for example Maths, Science, English and PE. Our school day starts at 8/45 with a Form class meeting, and finishes at 3/15. We have an hour for lunch, and can buy food at the canteen but we are not allowed to go to the shops. We have a lot of sports teams at my school. Mostly we practice at lunchtime and after school, and play against other schools on Saturday. In summer I like to play tennis, and in winter I like to play hockey. My father likes to come and watch me play sport, and afterwards we often go to a café for a delicious lunch. If you don’t like playing sport we have all sorts of other clubs such as drama, music and debating. I hope you join one of our after school clubs, you will enjoy the activities and the company of the other students. In the weekends and holidays there are lots of places you could visit with me and my family. We could go to the beaches or the hot springs, there are museums and art galleries and concerts, and many interesting small towns to visit, we could even stay on a farm and help with the animals. I hope very much you enjoy your stay in New Zealand, I am looking forward to your visit. Kim A bad question to match Describe Kim’s family Explain what the exchange student can do during the school day during class time outside of class time Describe a typical weekend for Kim. Use information from the text to support your answer.

Some questions apropos the previous 1.What’s the problem with the text? 2.Can it be improved without too much hassle? (Pause and consider) 3.What’s the problem with the question? 4.Can it be improved without too much hassle? (Pause and consider) 5.What’s more important / easier to fix, the text or the questions? And, going back a couple of steps: 6.What’s the function of an exam question? 7.What things characterise a good exam question?

The Assessment Schedule The schedule is formatted the same way for both standards, all languages, at all three levels. It is intended to enable holistic judgement for each question, and avoid focussing on counting up fragmented pieces of evidence / lexical items. It is a guideline to how to make judgements, not a black and white list of ‘right’ and ‘wrong’ answers. Judgements between grades (that is N and A, A and M, M and E) must be qualitative Judgements within grades may be qualitative or quantitative Although final cut-scores may vary from exam to exam (depending on relative difficulty) adherence to the intent of the standard supersedes all other considerations. That is, final judgements must be made against the standard. 23

Adapting a schedule Schedules for the old standards have: bolds underlines question judgements that specify how many ‘pieces of evidence’ are required Schedules for new standards have: each level of achievement, as defined in the standards, at the top (this can me omitted in an adapted schedule) nine ‘grade score descriptors’ a specific evidence box brief examples for each of the nine levels 24

Quick analysis of a test On one page, draw up a tally table like the one below. Go through the class results, one student and a time. Enter (as a tally) each student’s results for the four questions. You can now see which questions: are discriminating well seem to be too ‘hard’ or too ‘easy’ have a very strange spread. Score Question One llllllllllllll llllll ll Two llllllll lllllll lllllllll Three llllllll llll lll l Four lllllllllll llll lllllllll ll 25