1. Assessment to assure quality Barry McGaw Professorial Fellow, University of Melbourne Former Chair, Australian Curriculum, Assessment and Reporting Authority Former Director for Education, OECD The 1 st Public Education Evaluation Commission Conference Riyadh, Saudi Arabia 3-5 November 2015

2. mcgaw group pty ltd ABN 34 117 491 228  Countries provoked to change by comparisons  Schools provoked to change by comparisons  Assessing to reveal weaknesses and then finding ways to improve  So, what have we seen? Outline of presentation

3. Countries provoked to change by comparisons

4. mcgaw group pty ltd ABN 34 117 491 228 You need to assess only a sample of students.

5. A low-performing country challenged to improve

6. mcgaw group pty ltd ABN 34 117 491 228 Trends in PISA reading means Finland OECD Poland

7. mcgaw group pty ltd ABN 34 117 491 228 Trends in PISA reading, mathematics & science means Finland OECD Poland Reading Mathematics Science Finland OECD Poland Finland OECD Poland

8. mcgaw group pty ltd ABN 34 117 491 228 How did Poland do it?. 1999 Reform Ceased separating students into academic and vocational tracks before age 15. Strengthened role of local government. Increased autonomy for schools Introduction of national exams for accountability 2007-2008 Curriculum Reform Comprehensive education extended from 8 to 11 years. Academic studies added to vocational education. Universal preschool education introduced. CONCLUSIONS Effective reform takes time. Without data: You don’t know what you need to do You don’t know if you are succeeding..

9. A high-performing country challenged to improve its best performing students

10. mcgaw group pty ltd ABN 34 117 491 228 Percent of students at each PISA 2000 reading level 1 st 2 nd 3 rd 4 th 5 th 6 th Rank on means shown in red. Level 5 Level 4 Level 3 Level 2 Level 1 Below Level 1 7 th 8 th 9 th

11. mcgaw group pty ltd ABN 34 117 491 228 Changes in top nine in PISA 2000 reading to 2003 Finland Austria New Zealand Canada Korea Australia Japan Ireland Sweden

12. mcgaw group pty ltd ABN 34 117 491 228 How did South Korea do it?. Shift in distribution of results PISA 2000 – Korea showed had the smallest variation in student performances PISA 2003 – Korea’s variation was at the OECD average, with more high- performing students. CONCLUSIONS This reform worked more quickly than Poland’s It was a smaller change The country was starting from a high base. Without data: You don’t know what you need to do You don’t know if you are succeeding.. YES, BUT how did Korea increase the spread New curriculum More emphasis on essay tests More use of essays in assessments for university entrance.

13. Schools provoked to change by comparisons

14. mcgaw group pty ltd ABN 34 117 491 228 You have to assess the full cohort of students.

15. mcgaw group pty ltd ABN 34 117 491 228 Performance vs socio-educational advantage Index of socio-educational advantage (ICSEA) Performance School B School A School C

17. Band 7 Band 6 Band 5 Band 4 Band 3 20082009 School’s mean scores The lines indicate the 90% error ranges for the estimate of the means

18. Band 7 Band 6 Band 5 Band 4 Band 3 20082009 National mean scores

19. Band 7 Band 6 Band 5 Band 4 Band 3 20082009 Mean scores for 60 schools with similar students

20. mcgaw group pty ltd ABN 34 117 491 228 The school was coasting, doing well compared with the national average but relatively badly compared with other schools with students with the same level of socio-educational advantage. Which are those schools?

21. Schools in dark purple are 0.5 standard deviation or more ahead of Indooroopilly Indooroopilly Primary School 430 417-443 Schools in light purple are more than 0.2 and less than 0.5 standard deviation ahead of Indooroopilly Comparison among individual schools - 2008.

22. mcgaw group pty ltd ABN 34 117 491 228 What happened next?

23. Band 7 Band 6 Band 5 Band 4 Band 3 20082009 20102011201220132014

24. Assessing to reveal weaknesses and then finding ways to improve

25. Overcoming common weaknesses at the school level

26. Developed by New South Wales Department of Education Used in New South Wales and the Australian Capital Territory  Government schools  Catholic schools  Independent schools Provides diagnostic information and remedial suggestions  For schools  For students

27. Each circle is a student. At year 3, school is doing better in reading than numeracy.

28. By year 5, school performance in reading matches that in numeracy.

29. Here is a difficult item on which school does less well than might be expected.

30. Here is the item and information on which students were incorrect. Clicking here leads to suggested teaching strategies to remediate weakness.

31. This is the page on suggested teaching strategies to which the user is taken.

32. Overcoming specific weaknesses for individual students

33. Here is a representation of the growth from Grade 3 to Grade 5 of each student in the class. Student: Emma Wood Score Grade 3: 493 Score Grade 5: 531 Growth: 38 Expected growth: 47 State average: 76 School average: 84 25 th percentile 75 th percentile The students whose growth is shown with a blue arrow grew less than expected,

34. Emma Wood answered 13 of the 39 questions incorrectly. This display shows, item by item, in the test which she answered correctly and which incorrectly and which incorrect response she had chosen.

35. So, what have we seen?

36. mcgaw group pty ltd ABN 34 117 491 228 Without assessment you cannot have the evidence with which to identify strengths and weaknesses. Comparative assessment can show schools, teachers and parents how they are doing compared with others. They can show what is possible and even challenge a high-performing, coasting school to do better. Assessment can diagnose weaknesses for which remediation can be achieved. That is the role of summative assessment. That is the role of formative assessment.

37. Thank you.