PISA OECD Programme for International Student Assessment Science Competencies for Tomorrow’s World Science competencies for tomorrow’s world Seeing school systems through the prism of PISA OECD Programme for International Student Assessment (PISA) Kyiv, 27 March 2008 Ian Whitman OECD Directorate for Education

PISA OECD Programme for International Student Assessment Science Competencies for Tomorrow’s World Today

PISA OECD Programme for International Student Assessment Briefing of Council 14 November 2007 OECD’s Programme for International Student Assessment (PISA) What PISA seeks to accomplish How PISA works

PISA OECD Programme for International Student Assessment Science Competencies for Tomorrow’s World In the dark… …all students, schools and education systems look the same… But a little light….

PISA OECD Programme for International Student Assessment Science Competencies for Tomorrow’s World But a little light…. …can show important differences…. In the dark… …all students, schools and education systems look the same…

PISA OECD Programme for International Student Assessment Science Competencies for Tomorrow’s World PISA A three-yearly global assessment that… … examines the performance of 15-year-olds in key subject areas as well as a wider range of educational outcomes Including students attitudes to learning and their learning behaviour …collects contextual data from… …students, parents, schools and systems… …in order to identify policy levers Coverage Representative samples of between 3,500 and 30, year-old students drawn in each country Most federal countries also draw regional samples PISA covers roughly 90% of the world economy.

PISA OECD Programme for International Student Assessment Science Competencies for Tomorrow’s World 1998 PISA countries in Coverage of world economy 77%81% 83% 85%86%87%

PISA OECD Programme for International Student Assessment Science Competencies for Tomorrow’s World r A strong international network of expertise among the participating countries… From establishing the assessment frameworks… –The PISA assessments include tasks from more than 40 countries …developing the instruments… –Cross-national and cross-cultural validity …to analysing and interpreting the results –National, regional and international analyses and reports –In-depths country peer reviews …supported by a consortium of leading research institutions… ACER, CITO, ETS, NIER, WESTAT …co-ordinated through the OECD in collaboration with other international organisations. How PISA works

PISA OECD Programme for International Student Assessment Briefing of Council 14 November 2007 Key findings from PISA 2006 Where we are – how students perform across countries Where we can be – the top performers How we can get there – some school and system factors

PISA OECD Programme for International Student Assessment Science Competencies for Tomorrow’s World PISA 2006 r The latest PISA assessment emphasizes science competencies, defined in terms of an individual’s: Scientific knowledge and use of that knowledge to… …identify scientific issues, …explain scientific phenomena, and …draw evidence-based conclusions about science-related issues Understanding of the characteristic features of science as a form of human knowledge and enquiry Awareness of how science and technology shape our material, intellectual and cultural environments Willingness to engage with science-related issues r A large proportion of complex open-ended tasks.

PISA OECD Programme for International Student Assessment Science Competencies for Tomorrow’s World Deciding what to assess... looking back at what students were expected to have learned …or… looking ahead to how well they can extrapolate from what they have learned and apply their knowledge and skills in novel settings. For PISA, the OECD countries chose the latter.

PISA OECD Programme for International Student Assessment Science Competencies for Tomorrow’s World How the demand for skills has changed Economy-wide measures of routine and non-routine task input (US) (Levy and Murnane) Mean task input as percentiles of the 1960 task distribution

PISA OECD Programme for International Student Assessment Briefing of Council 14 November 2007 Quality in learning outcomes Science performance

PISA OECD Programme for International Student Assessment Science Competencies for Tomorrow’s World Average performance of 15-year-olds in science – extrapolate and apply High science performance Low science performance

PISA OECD Programme for International Student Assessment Science Competencies for Tomorrow’s World Strengths and weaknesses of countries in science relative to their overall performance Poland OECD (2007), PISA 2006 – Science Competencies for Tomorrow’s World, Figure 2.13 Science competencies Science knowledge

PISA OECD Programme for International Student Assessment Science Competencies for Tomorrow’s World Strengths and weaknesses of countries in science relative to their overall performance Russian Federation OECD (2007), PISA 2006 – Science Competencies for Tomorrow’s World, Figure 2.13 Science competencies Science knowledge

PISA OECD Programme for International Student Assessment Science Competencies for Tomorrow’s World Strengths and weaknesses of countries in science relative to their overall performance Serbia OECD (2007), PISA 2006 – Science Competencies for Tomorrow’s World, Figure 2.13 Science competencies Science knowledge

PISA OECD Programme for International Student Assessment Science Competencies for Tomorrow’s World Strengths and weaknesses of countries in science relative to their overall performance Montenegro OECD (2007), PISA 2006 – Science Competencies for Tomorrow’s World, Figure 2.13 Science competencies Science knowledge

PISA OECD Programme for International Student Assessment Science Competencies for Tomorrow’s World Strengths and weaknesses of countries in science relative to their overall performance Kyrgyzstan OECD (2007), PISA 2006 – Science Competencies for Tomorrow’s World, Figure 2.13 Science competencies Science knowledge

PISA OECD Programme for International Student Assessment Science Competencies for Tomorrow’s World Gender differences in science performance OECD (2007), PISA 2006 – Science Competencies for Tomorrow’s World, Tables 2.1c, 2.2c, 2.3c, 2.4c, 2.7, 2.8, 2.9, 2.10 Boys do better Girls do better PISA score points

PISA OECD Programme for International Student Assessment Science Competencies for Tomorrow’s World Gender differences in science performance OECD (2007), PISA 2006 – Science Competencies for Tomorrow’s World, Tables 2.1c, 2.2c, 2.3c, 2.4c, 2.7, 2.8, 2.9, 2.10 Boys do better Girls do better PISA score points

PISA OECD Programme for International Student Assessment Science Competencies for Tomorrow’s World Gender differences in science performance OECD (2007), PISA 2006 – Science Competencies for Tomorrow’s World, Tables 2.1c, 2.2c, 2.3c, 2.4c, 2.7, 2.8, 2.9, 2.10 Boys do better Girls do better PISA score points

PISA OECD Programme for International Student Assessment Science Competencies for Tomorrow’s World Gender differences in science performance OECD (2007), PISA 2006 – Science Competencies for Tomorrow’s World, Tables 2.1c, 2.2c, 2.3c, 2.4c, 2.7, 2.8, 2.9, 2.10 Boys do better Girls do better PISA score points

PISA OECD Programme for International Student Assessment Science Competencies for Tomorrow’s World Gender differences in science performance OECD (2007), PISA 2006 – Science Competencies for Tomorrow’s World, Tables 2.1c, 2.2c, 2.3c, 2.4c, 2.7, 2.8, 2.9, 2.10 Boys do better Girls do better PISA score points

PISA OECD Programme for International Student Assessment Briefing of Council 14 November 2007 Distribution of student performance

PISA OECD Programme for International Student Assessment Science Competencies for Tomorrow’s World Top and bottom performers in science These students often confuse key features of a scientific investigation, apply incorrect information, mix personal beliefs with facts in support of a position… These students can consistently identify, explain and apply scientific knowledge, link different information sources and explanations and use evidence from these to justify decisions, demonstrate advanced scientific thinking in unfamiliar situations… Large proportion of top performers Large prop. of poor perf.

PISA OECD Programme for International Student Assessment Science Competencies for Tomorrow’s World Top performers matter Excellence in education and countries’ research intensity

PISA OECD Programme for International Student Assessment Science Competencies for Tomorrow’s World Money matters - but other things do too

PISA OECD Programme for International Student Assessment Science Competencies for Tomorrow’s World Student performance on the science scale and national income Relationship between performance in science and GDP per capita, in US dollars, converted using purchasing power parities (PPPs).

PISA OECD Programme for International Student Assessment Science Competencies for Tomorrow’s World Investments and outcomes r Since 2000, expenditure per primary and secondary student increased across OECD countries by 39% (in real terms) … …while PISA outcomes generally remained flat… …but there are notable exceptions…

PISA OECD Programme for International Student Assessment Science Competencies for Tomorrow’s World Poland raised its reading performance by 28 PISA points, equivalent to ¾ of a school year - What happened? OECD (2007), Learning for tomorrow’s world: First results from PISA 2006, Table 6.1a 20 Between PISA 2000 and 2003 Poland delayed the separation of students into different school types beyond the age of 15 years In 2003, performance variation among schools had fallen from 51% to 16% of the variation of student performance But did this lead to genuine improvements of school performance? Between 2000 and 2003 showed the second- largest increase in reading (17 points) and a further 11 point increase since 2003 Most of that increase resulted from smaller proportions at the bottom level (23% in 2000, and three-quarters in vocational tracks, 17%in 2003) Did this harm the better performers?

PISA OECD Programme for International Student Assessment Briefing of Council 14 November 2007 Students attitudes to science and their awareness of the life opportunities science may open

PISA OECD Programme for International Student Assessment Science Competencies for Tomorrow’s World Students generally value science… OECD (2007), PISA 2006 – Science Competencies for Tomorrow’s World, Figure 3.2 and Figure 3.13 % …but only a minority report interest in a scientific career %

PISA OECD Programme for International Student Assessment Science Competencies for Tomorrow’s World Students generally value science… OECD (2007), PISA 2006 – Science Competencies for Tomorrow’s World, Figure 3.2 and Figure 3.13 % …but only a minority report interest in a scientific career %

PISA OECD Programme for International Student Assessment Science Competencies for Tomorrow’s World Students generally value science… OECD (2007), PISA 2006 – Science Competencies for Tomorrow’s World, Figure 3.2 and Figure 3.13 % …but only a minority report interest in a scientific career %

PISA OECD Programme for International Student Assessment Science Competencies for Tomorrow’s World Students generally value science… OECD (2007), PISA 2006 – Science Competencies for Tomorrow’s World, Figure 3.2 and Figure 3.13 % …but only a minority report interest in a scientific career %

PISA OECD Programme for International Student Assessment Science Competencies for Tomorrow’s World Students generally value science… OECD (2007), PISA 2006 – Science Competencies for Tomorrow’s World, Figure 3.2 and Figure 3.13 % …also majority report interest in a scientific career %

PISA OECD Programme for International Student Assessment Science Competencies for Tomorrow’s World Higher mean performance in science, but smaller proportion of students expecting a science- related career at age 30 Higher mean performance in science and larger proportion of students expecting a science- related career at age 30 Lower mean performance in science and smaller proportion of students expecting a science- related career at age 30 Lower mean performance in science, but larger proportion of students expecting a science- related career at age 30 OECD mean Figure Performance in science and proportions of students expecting a science-related career at age 30 Source: OECD PISA 2006 database, Tables 3.12 and 2.1c.

PISA OECD Programme for International Student Assessment Briefing of Council 14 November 2007 Equity in educational opportunities

PISA OECD Programme for International Student Assessment Science Competencies for Tomorrow’s World Durchschnittliche Schülerleistungen im Bereich Mathematik Low average performance Large socio-economic disparities High average performance Large socio-economic disparities Low average performance High social equity High average performance High social equity Strong socio- economic impact on student performance Socially equitable distribution of learning opportunities High science performance Low science performance Bulgaria -- Serbia

PISA OECD Programme for International Student Assessment Briefing of Council 14 November 2007 Coherence of educational standards across schools

PISA OECD Programme for International Student Assessment Science Competencies for Tomorrow’s World Variation in student performance OECD (2007), Learning for tomorrow’s world: First results from PISA 2006, Table 4.1a 20

PISA OECD Programme for International Student Assessment Science Competencies for Tomorrow’s World Variation of performance between schools Variation of performance within schools OECD (2004), Learning for tomorrow’s world: First results from PISA 2003, Table 4.1a Variation in student performance

PISA OECD Programme for International Student Assessment Briefing of Council 14 November 2007 Some levers for policy

PISA OECD Programme for International Student Assessment Science Competencies for Tomorrow’s World Some myths r No relationship between size of countries and average performance r No relationship between proportion of immigrants and average performance r Few difference in students’ reported test motivation r Limited impact of national item preferences.

PISA OECD Programme for International Student Assessment Science Competencies for Tomorrow’s World Pooled international dataset, effects of selected school/system factors on science performance after accounting for all other factors in the model OECD (2007), PISA 2006 – Science Competencies from Tomorrow’s World, Table 6.1a 20 Schools practicing ability grouping (gross and net) Academically selective schools (gross and net) but no system-wide effect School results posted publicly (gross and net) One additional hour of science learning at school (gross and net) One additional hour of out- of-school lessons (gross and net) One additional hour of self-study or homework (gross and net) School activities to promote science learning (gross and net) Schools with greater autonomy (resources) (gross and net) Each additional 10% of public funding (gross only) Schools with more competing schools (gross only) School principal’s perception that lack of qualified teachers hinders instruction (gross only) School principal’s positive evaluation of quality of educational materials (gross only) Measured effect Effect after accounting for the socio-economic background of students, schools and countries

PISA OECD Programme for International Student Assessment Science Competencies for Tomorrow’s World PISA score in science School autonomy, standards-based examinations and science performance School autonomy in selecting teachers for hire

PISA OECD Programme for International Student Assessment Science Competencies for Tomorrow’s World PISA score in science School autonomy, standards-based examinations and science performance School autonomy in deciding on budget allocation within the school

PISA OECD Programme for International Student Assessment Science Competencies for Tomorrow’s World Durchschnittliche Schülerleistungen im Bereich Mathematik Low average performance Large socio-economic disparities High average performance Large socio-economic disparities Low average performance High social equity High average performance High social equity Strong socio- economic impact on student performance Socially equitable distribution of learning opportunities High science performance Low science performance Early selection and institutional differentiation High degree of stratification Low degree of stratification

PISA OECD Programme for International Student Assessment Science Competencies for Tomorrow’s World Why care? r Progress Concerns about skill barriers to economic growth, productivity growth and rates of technological innovation –One additional year of education equals to between 3 and 6% of GDP –Rising college-level qualifications seem generally not to have led to an “inflation” of the labour-market value of qualifications (in all but three of the 20 countries with available data, the earnings benefit increased between 1997 and 2003, in Germany, Italy and Hungary by between 20% and 40%) r Fairness Concerns about the role of skills in creating social inequity in economic outcomes –Both average and distribution of skill matter to long-term growth (high percentages of low skill impede growth) r Value for money Concerns about the demand for, and efficiency and effectiveness of, investments in public goods

PISA OECD Programme for International Student Assessment Science Competencies for Tomorrow’s World –All national and international publications –The complete micro-level database …and remember: Without data, you are just another person with an opinion