Assessment to assure quality Barry McGaw Professorial Fellow, University of Melbourne Former Chair, Australian Curriculum, Assessment and Reporting Authority Former Director for Education, OECD The 1 st Public Education Evaluation Commission Conference Riyadh, Saudi Arabia 3-5 November 2015

mcgaw group pty ltd ABN  Countries provoked to change by comparisons  Schools provoked to change by comparisons  Assessing to reveal weaknesses and then finding ways to improve  So, what have we seen? Outline of presentation

Countries provoked to change by comparisons

mcgaw group pty ltd ABN You need to assess only a sample of students.

A low-performing country challenged to improve

mcgaw group pty ltd ABN Trends in PISA reading means Finland OECD Poland

mcgaw group pty ltd ABN Trends in PISA reading, mathematics & science means Finland OECD Poland Reading Mathematics Science Finland OECD Poland Finland OECD Poland

mcgaw group pty ltd ABN How did Poland do it? Reform Ceased separating students into academic and vocational tracks before age 15. Strengthened role of local government. Increased autonomy for schools Introduction of national exams for accountability Curriculum Reform Comprehensive education extended from 8 to 11 years. Academic studies added to vocational education. Universal preschool education introduced. CONCLUSIONS Effective reform takes time. Without data: You don’t know what you need to do You don’t know if you are succeeding..

A high-performing country challenged to improve its best performing students

mcgaw group pty ltd ABN Percent of students at each PISA 2000 reading level 1 st 2 nd 3 rd 4 th 5 th 6 th Rank on means shown in red. Level 5 Level 4 Level 3 Level 2 Level 1 Below Level 1 7 th 8 th 9 th

mcgaw group pty ltd ABN Changes in top nine in PISA 2000 reading to 2003 Finland Austria New Zealand Canada Korea Australia Japan Ireland Sweden

mcgaw group pty ltd ABN How did South Korea do it?. Shift in distribution of results PISA 2000 – Korea showed had the smallest variation in student performances PISA 2003 – Korea’s variation was at the OECD average, with more high- performing students. CONCLUSIONS This reform worked more quickly than Poland’s It was a smaller change The country was starting from a high base. Without data: You don’t know what you need to do You don’t know if you are succeeding.. YES, BUT how did Korea increase the spread New curriculum More emphasis on essay tests More use of essays in assessments for university entrance.

Schools provoked to change by comparisons

mcgaw group pty ltd ABN You have to assess the full cohort of students.

mcgaw group pty ltd ABN Performance vs socio-educational advantage Index of socio-educational advantage (ICSEA) Performance School B School A School C

Band 7 Band 6 Band 5 Band 4 Band School’s mean scores The lines indicate the 90% error ranges for the estimate of the means

Band 7 Band 6 Band 5 Band 4 Band National mean scores

Band 7 Band 6 Band 5 Band 4 Band Mean scores for 60 schools with similar students

mcgaw group pty ltd ABN The school was coasting, doing well compared with the national average but relatively badly compared with other schools with students with the same level of socio-educational advantage. Which are those schools?

Schools in dark purple are 0.5 standard deviation or more ahead of Indooroopilly Indooroopilly Primary School Schools in light purple are more than 0.2 and less than 0.5 standard deviation ahead of Indooroopilly Comparison among individual schools

mcgaw group pty ltd ABN What happened next?

Band 7 Band 6 Band 5 Band 4 Band

Assessing to reveal weaknesses and then finding ways to improve

Overcoming common weaknesses at the school level

Developed by New South Wales Department of Education Used in New South Wales and the Australian Capital Territory  Government schools  Catholic schools  Independent schools Provides diagnostic information and remedial suggestions  For schools  For students

Each circle is a student. At year 3, school is doing better in reading than numeracy.

By year 5, school performance in reading matches that in numeracy.

Here is a difficult item on which school does less well than might be expected.

Here is the item and information on which students were incorrect. Clicking here leads to suggested teaching strategies to remediate weakness.

This is the page on suggested teaching strategies to which the user is taken.

Overcoming specific weaknesses for individual students

Here is a representation of the growth from Grade 3 to Grade 5 of each student in the class. Student: Emma Wood Score Grade 3: 493 Score Grade 5: 531 Growth: 38 Expected growth: 47 State average: 76 School average: th percentile 75 th percentile The students whose growth is shown with a blue arrow grew less than expected,

Emma Wood answered 13 of the 39 questions incorrectly. This display shows, item by item, in the test which she answered correctly and which incorrectly and which incorrect response she had chosen.

So, what have we seen?

mcgaw group pty ltd ABN Without assessment you cannot have the evidence with which to identify strengths and weaknesses. Comparative assessment can show schools, teachers and parents how they are doing compared with others. They can show what is possible and even challenge a high-performing, coasting school to do better. Assessment can diagnose weaknesses for which remediation can be achieved. That is the role of summative assessment. That is the role of formative assessment.

Thank you.