1. NAPLAN and MySchool – impact on Assessment and Curriculum Margaret Wu 1

2. “There are three kinds of lies: lies, damned lies, and statistics.” Mark Twain, based on part of a phrase attributed to Benjamin Disraeli 2

3. Statistical complexity – to stop debate When situations become calculable it is taken to indicate the fact that political contestation has ended. (Andrew Barry, 2002, p. 272) 3

4. Statistical complexity – to stop debate (Julia Gillard, in reply to a journalist, National Press Club, 24/2/10, cited in Williss, 2010) We have obviously had public debate about the ICSEA index... I do have a standing offer to any journalist who has read Barry McGaw’s book on meta‐analysis and would like to sit through and work through the regression equations with him, anybody who wants to do that, a standing invitation to come to my office for the number of days necessary to get that done. 4

5. Claims about NAPLAN data:  Smh: Nov 16, 2010 Giving a preview of My School version two on Wednesday, Ms Gillard said the additional facts would help parents make more informed decisions about where to send their children.  The Australian: Nov 18, 2010 The extra data will give principals greater ability to identify poorly performing teachers.  Smh: Nov 17, 2010 PARENTS will be able to measure how much an individual school improves the performance of their child on the federal government's My School website. 5

6. Key Issues Large margins of error at student and class levels. (reliability) Student factors contribute more to student achievement than teacher/school factors do. validity ICSEA does NOT explain away all student level factors. 6

7. (a) 16 × 10 = (j) 139.2 ÷ 1000 = ( a) The temperature was 7°. It fell by 4°. The temperature was then (c) (-11) + (+3) = A class starts at 10:30. The class is 40 minutes long. What time does the class finish? Each apple weighs around 160 grams. How many apples together will weigh close to half a kilogram? A class starts at 10:30. The class is 40 minutes long. What time does the class finish? Possible Grade 5 Mathematics Item Pool – Many questions can be asked NAPLAN 2008 Test 40 questions are sampled from the large item pool David’s score: 20/40 NAPLAN 2009 Test David’s score: 25/40 NAPLAN 2010 Test David’s score: 28/40 David can answer 60% of the items (if we have the opportunity to administer all items) David’s test scores on similar NAPLAN tests will have a range of 10 score points (e.g., between 20/40 to 30/40)

8. 8 On the NAPLAN scale…

9. 9

10. 10 Teacher effect A high performing teacher can raise student standards by one more year of growth as compared to a low performing teacher. excellent teacher average teacher Ineffective teacher 1.5 year of growth 1 year of growth 0.5 year of growth Margin of error surrounding class average is as large as the whole range of teacher effect.

11. Summary about margin of error One single test, once a year Not much information that the teacher does not already know NAPLAN was not designed to inform about individual students. NAPLAN is a BLUNT instrument for measuring teaching effect. 11

12. “Torture numbers, and they will confess to anything.” Gregg Easterbrook 12

13. ICSEA Idea of ICSEA is to control for student level factors Assumption is that after adjusting for ICSEA, what is left is school effect. 13

14. Everyone equal ?% variation + Variation due to natural ability/ personality differences ?% variation + Variation due to school/ teacher effect ?% variation + Variation due to student background Factors contributing to student achievement 14 10% of the student scores variation is due to school/teacher effect

15. Variation of school mean scores After ICSEA- Variation within like- school group Red/Green bars do NOT reflect school effects 15 School effect Non-School effect

16. Donnelly (April 14, 2011, ABC The Drum) On the back of schools being forced to implement Australia’s National Assessment Program Literacy and Numeracy tests (NAPLAN) at years three, five, seven and nine beginning in 2008, Australia’s education ministers have called for a similar testing and accountability regime to monitor and evaluate Australian parents. The new tests, called National Assessment Program Mums and Dads (NAPMAD), will be taken annually by all parents and will improve parent productivity by benchmarking Australian homes against world best practice. 16

17. Drawing statistical inference What does “accurate” mean? NAPLAN tests students Used to make inferences about teachers A leap of faith I want to know if the principal at Eastbourne primary school is effective! There is 36% chance that he is effective, 48% chance of being average, and 16% chance of being ineffective And I am only 90% confident of the above statement 17

18. Conviction by Numbers Given any group, there will always be someone at the lower end, by chance. After examining a whole group, the chance of finding someone at the lower end is a certainty. It ain’t statistics. I want to know the truth! Darrell Huff, How to Lie with Statistics 18

19. Conviction by Numbers Statistical information alone can never be used to make high stakes decisions Certainly not to label schools/teachers as effective or not effective. Teacher pay linked to student performance? Punitive actions against teachers based on test results must be stopped. 19

20. Finally… 20 Darrell Huff