1. 1 The Appropriate Use of NAPLAN Data National Symposium, 23 July, 2010 Margaret Wu University of Melbourne m.wu@unimelb.edu.au 1

2. 2 NAPLAN Tests Conducted once a year About 40 test questions per subject area Test scores are used to infer ◦ the achievement levels of students How reliable can NAPLAN test scores reflect ◦ Student achievement level? ◦ School performance? 2

3. 3 Margin of error in measuring student performance David - a Grade 5 student in 2008. Reading score was 25 out of 40. David’s reading test scores could vary between 20 and 30, out of 40. ◦ if similar tests are administered (e.g., 2009, 2010 tests ) One test collects only a small sample of performance. Variation in scores is called Measurement Error. 3

4. 4 How big an error size is acceptable? The answer is ◦ It depends. An example ◦ Effectiveness of a weight loss program ◦ Expect a loss of 0.5 kg after one week. ◦ Measurement scale is accurate to 1kg. ◦ Not good enough for measuring individual change ◦ OK for a group change, if group size is ‘large’. 4

5. 5 On the NAPLAN scale… 5

6. 6 6

7. 7 Measuring Growth Growth measure? Margin of error of growth measure ± 76 points Expected growth is 50 points 7

8. 8 Class mean scores Average score for a class ◦ Effect of measurement error reduces New source of error ◦ Sampling error Cohort of students changes from year to year Variation in class mean score because of the sample of students in a class Class mean ± 20 points ◦ (1 year’s growth) 8

9. 9 Teacher effect A high performing teacher can raise student standards by one more year of growth as compared to a low performing teacher. excellent teacher average teacher poor teacher 50 points Margin of error of teacher effect based on two testing occasions: ± 20 points 9

10. 10 MySchool Website It is a league table ◦ It compares and ranks schools It is the worst kind of league table ◦ Because it is claimed that the red bars reflect “underperforming schools” ◦ Simple league tables do not have this claim.

11. 11 Summary - 1 NAPLAN results are NOT suitable for measuring Student achievement level  ◦ beyond a rough “lower”, “average”, “higher” groups Student progress  Teacher effect  School performance  11

12. 12 Summary - 2 NAPLAN results are for the systems, e.g. ◦ Compare girls and boys ◦ Compare rural and urban ◦ Trends, if equating design is improved NAPLAN results should NEVER be published. Parents/caregivers should not be encouraged to use the results to judge schools. 12

13. 13 Finally… Conflicting advice from different experts? An easy way to check out: Ask proponents of MySchool website to publicly name one underperforming school. 13

14. 14 References Wu, M.L. (2010). Measurement, sampling and equating errors in large-scale assessments. Educational Measurement: Issues and Practice, (In press: Volume 29 Number 4). Nye, B., Konstantopoulos, S., & Hedges, L. (2004). How Large Are Teacher Effects? Educational Evaluation and Policy Analysis, Vol. 26, No. 3 (Autumn, 2004), pp. 237-257.

15. 15 Leigh, A. (2009). Estimating teacher effectiveness from two-year changes in students’ test scores. Economics of Education Review. Byrne, Coventry, Olson, Wadsworth, Samuelsson, Petrill, Willcutt and Corley. (2009). Teacher Effects in Early Literacy Development: Evidence From a Study of Twins. Journal of Educational Psychology, 2009.