Potential impact of PISA Organisation for Economic Cooperation and Development (OECD) Potential impact of PISA Michael Davidson OECD/Directorate for Education

A three-yearly assessment that… What is PISA? A three-yearly assessment that… … examines the performance of 15-year-olds in key subject areas as well as a wider range of educational outcomes Including students attitudes to learning, their beliefs about themselves, and their learning strategies … collects contextual data from students schools, parents and systems to identify policy levers Contextual data explain 71% of the performance variation among schools internationally Coverage Representative samples of between 3,500 and 50,000 15-year-old students drawn in each country Most federal countries also draw regional samples

Results from PISA 2006 Scientific knowledge and use of that knowledge in… … identifying scientific issues, … explaining scientific phenomena, … using scientific evidence

PISA provides key benchmarks for the quality of education systems 1. Overall performance of education systems 2. Equity in the distribution of learning opportunities Measured by the impact students’ and schools’ socio-economic background has on performance… … not merely by the distribution of learning outcomes 3. Consistency of performance standards across schools 4. Gender differences

Mean science scores The centre line is the mean The boundaries indicate the 95% confidence limits There is only a significant difference if there is no overlap of scores OECD (2007), PISA 2006 – Science Competencies for Tomorrow’s World, Table 2.1c

Is it all innate ability? Variation in student performance Performance variation across schools 20 Consistent standards across schools is a gaol for many education systems. PISA allows for an analysis of the variation of student performance across schools. OECD (2007), Learning for tomorrow’s world: First results from PISA 2006, Table 4.1a

Variation in student performance The variation in country performance between the high performers and the low performers can be seen clearly in this chart. For example, the United States, New Zealand and the United Kingdom have a much higher degree of variation than Turkey or Mexico. However, what is even more interesting is if we take the same chart and separate the variation into two different types – that which occurs within schools and that which occurs between schools … PISA 2006: Science Competencies for Tomorrow’s World, Figure 4.1a.

Variation in student performance Variation of performance within schools Variation of performance between schools Between-school variation is an indication of how much a student’s performance depends on the school that he or she attends. It can be seen in the diagram that in countries such as Finland, Iceland and Norway that the between-school variation is relatively small, meaning that standards across schools in these countries are very consistent, whereas in the countries at the other end of the chart there are large differences between schools. This can be a result of the design of the education system where there may be different types of schools that students attend (as in Germany). PISA 2006: Science Competencies for Tomorrow’s World, Figure 4.1a.

High science performance Average performance of 15-year-olds in science – extrapolate and apply Equity In PISA 2006 we also looked at students’ attitudes towards science. Why is this important? Competing successfully in a globalised world increasingly depends on countries’ ability to innovate. This in turn will require major investments in scientific infrastructure and the ability to attract qualified individuals into science-related professions. Governments have to secure broad public support for scientific endeavour. Science and technology have enabled remarkable achievement over the past 100 years, but addressing these challenges successfully will require countries to make major investments in scientific infrastructure and to attract qualified individuals into science-related professions, as well as to secure broad public support for scientific endeavour and the capacity of all citizens to use science in relation to their lives. Peoples’ attitudes to science thus play a key role. In general, 15-year-olds report a fairly strong level of appreciation of science, and they reported an above average level of awareness of environmental issues. … 18 countries perform below this line Low science performance

Impact of selected student and school factors on school performance (after accounting for all other factors in the model) Schools with greater autonomy (resources) School activities to promote science learning One additional hour of self-study or homework One additional hour of science learning at school School results posted publicly Academically selective schools but no system-wide effect Schools practising ability grouping for all subjects 20 The combined effect of a number of different factors including autonomy are summarised in this chart. Values which have a negative association with student performance are displayed below the zero line and those that have a positive association with performance are displayed above the line. PISA tells us that the most important factors for success they are not the ones most closely associated with finite material resources, such as the distribution of good teachers. Rather, what matters is how schools and the school system are run – for example, the amount of time that students spend in class and the extent to which schools are accountable for their results. Delivering such advantages to one student is not obviously at the expense of another. This, in itself, is an important conclusion from PISA. It underlines once more that quality, equity and coherence in educational standards are indeed achievable policy goals. USE TEXT ON CHART Effect after accounting for the socio-economic background of students, schools and countries OECD (2007), PISA 2006 – Science Competencies from Tomorrow’s World, Table 6.1a

Evaluating the policy impact of PISA

External evaluation of the policy impact of PISA Objective To what degree does PISA impact policies, meet expectations and priorities of the participants? Evaluation criteria Relevance Effectiveness Sustainability Unexpected / Unplanned impacts

External evaluation of the policy impact of PISA Team of three external evaluators David Hopkins, Dianne Pennock, Jo Ritzen Two-step procedure Questionnaire to stakeholders in the country In-depth evaluation of 5 case studies Stakeholder groups Policy makers at the national/federal and local level as well as members of the PISA Governing Board School principals, representatives of teacher and parent organisations Academics and researchers Representatives of the business community and the media Evaluation report Conclusions and recommendations based on the empirical results obtained

Strategy for an evaluation of the policy impact of PISA Effectiveness – Specific questions To what extent are the participating countries and other stakeholders aware of the output results of PISA? To what extent are these output results being used as input into the policy-making processes of clients and other stakeholders? How significant is the influence of the PISA output results used in the policy-making processes? What is the role of PISA in the context of national assessment strategies, i.e. to what extent have countries used PISA to add to, or complement, their national assessments or data? To what extent has PISA provided added value to the national assessment strategies? What impact has PISA had on policy coherence in participating countries? Effectiveness in terms of visibility, use and impact – both with regards to PISA itself, and in its relation to relevant national assessments: To what extent are the participating countries and other stakeholders aware of the output results of PISA? This will include an analysis of the dissemination strategy for PISA 2006 and of the PISA reports both at international and national levels. To what extent are these output results being used as input into the policy-making processes of clients and other stakeholders? This will include a review of the involvement of various national stakeholders in PISA, as well as a more general analysis of the reasons for countries to participate in PISA. How significant is the influence of the PISA output results used in the policy-making processes? This will include a review of the delivery chain at national and international levels from the PISA data through their utilisation for policy-making at national levels up to the impact of policy decisions. Specifically: To what extent has PISA impacted on school level practices and student learning? What is the role of PISA in the context of national assessment strategies, i.e. to what extent have countries used PISA to add to, or complement, their national assessments or data? To what extent has PISA provided added value to the national assessment strategies? What impact has PISA had on policy coherence in participating countries?

Outcomes – Relevance of PISA Policy makers are considered the most significant stakeholder group in relation to PISA and its results. Policy makers are also considered as the major group responsible for implementing policies in light of PISA. The various stakeholder groups assume relatively low levels of responsibility for the PISA results in their countries.

Outcomes – Relevance of PISA Overall, PISA is seen as a useful tool for identifying and addressing nationally relevant themes and problems In many countries PISA is used to monitor and evaluate the quality and equity of the education system Impact of PISA varies from country to country: In countries with unsatisfactory outcomes often direct policy impact In high-performing systems: PISA as an evaluation mechanism

Important aspects of PISA

Outcomes – Effectiveness and sustainability of PISA In all countries, PISA is seen as an important instrument for policy making with a high level of credibility Impact of PISA on policy formation increasing from PISA 2000 to PISA 2006 Influence of PISA greater at the national / federal level than at the local level or in school practices and classroom instruction Policy makers, academics and researchers and the media most aware of PISA

Addressing policy needs

Contributing to change

Outcomes – Effectiveness and sustainability of PISA According to respondents, the influence of PISA could be increased by: a better coordinated and strategic approach for the dissemination of PISA results; further support for various stakeholder groups in interpreting PISA results and in designing strategies for improvement in light of PISA; In some countries, clearer linkage of PISA results to national or federal assessment strategies; and, greater utilisation of the PISA results by participating countries.

Unexpected / unplanned outcomes Great public interest and debate PISA as an instrument to justify reform “Culture of blame” Increased interest in empirical educational research

Next step

Thank you very much for your attention. http://www. pisa. oecd Thank you very much for your attention! http://www.pisa.oecd.org Michael.davidson@oecd.org