PISA OECD Programme for International Student Assessment IRAs 53 rd Annual Convention Atlanta, 6 May 2008 Literacy skills for tomorrows world Seeing school systems through the prism of international comparisons OECD Programme for International Student Assessment (PISA) IRAs 53 rd annual convention, Atlanta, 6 May 2008 Andreas Schleicher Head, Indicators and Analysis Division OECD Directorate for Education

PISA OECD Programme for International Student Assessment IRAs 53 rd Annual Convention Atlanta, 6 May 2008 In the dark… …all students, schools and education systems look the same… But with a little light….

PISA OECD Programme for International Student Assessment IRAs 53 rd Annual Convention Atlanta, 6 May 2008 But with a little light…. …important differences become apparent…. In the dark… …all students, schools and education systems look the same…

PISA OECD Programme for International Student Assessment IRAs 53 rd Annual Convention Atlanta, 6 May 2008 Today

PISA OECD Programme for International Student Assessment Briefing of Council 14 November 2007 There is nowhere to hide How the global talent pool has changed Why literacy skills matter more today

PISA OECD Programme for International Student Assessment IRAs 53 rd Annual Convention Atlanta, 6 May 2008 A world of change in baseline qualifications Approximated by percentage of persons with high school or equivalent qualfications in the age groups 55-64, 45-55, und years % 1. Excluding ISCED 3C short programmes 2. Year of reference Including some ISCED 3C short programmes 3. Year of reference

PISA OECD Programme for International Student Assessment IRAs 53 rd Annual Convention Atlanta, 6 May 2008 High school completion rates Percentage of graduates to the population at the typical age of graduation %

PISA OECD Programme for International Student Assessment IRAs 53 rd Annual Convention Atlanta, 6 May 2008 College-level graduation rates Percentage of tertiary type A graduates to the population at the typical age of graduation % 15 2 Decline of the relative position of the US from 1995 to 2005

PISA OECD Programme for International Student Assessment IRAs 53 rd Annual Convention Atlanta, 6 May 2008 Moving targets Future supply of high school graduates

PISA OECD Programme for International Student Assessment IRAs 53 rd Annual Convention Atlanta, 6 May 2008 Future supply of high school graduates Future supply of college graduates

PISA OECD Programme for International Student Assessment IRAs 53 rd Annual Convention Atlanta, 6 May 2008 The quality of skills matters How the demand for skills has changed

PISA OECD Programme for International Student Assessment IRAs 53 rd Annual Convention Atlanta, 6 May 2008 Why literacy skills matter r Reading is the currency in the knowledge society Just as those with little money have difficulty meeting their basic needs, those with limited literacy are likely to find it more challenging to pursue their goals Like most currencies, reading literacy has been subject to inflation over the years r Despite the rapid growth in the supply of qualifications, demand grows even faster Such that the earnings and employment gap continues to grow

PISA OECD Programme for International Student Assessment IRAs 53 rd Annual Convention Atlanta, 6 May 2008 Why literacy skills matter r Approximately three quarters of adults with the lowest level of reading literacy in IALS were either not working or, if working, in relatively low-paying jobs (in the bottom 40% of wage earners) r Adults in the two lowest reading literacy levels were typically twice as likely to be unemployed as those in the highest three levels r Lower skills place individuals at higher risks of dependency and also influence civic, community and political behaviour

PISA OECD Programme for International Student Assessment IRAs 53 rd Annual Convention Atlanta, 6 May 2008 How the demand for skills has changed Economy-wide measures of routine and non-routine task input (US) (Levy and Murnane) Mean task input as percentiles of the 1960 task distribution To analyse, compare, contrast, and evaluate To think imaginatively To apply knowledge in real-life situations To communicate thoughts and ideas effectively OECD concept of literacy Accessing, managing, integrating and evaluating written information in order to develop ones knowledge and potential, and to participate in, and contribute to, society

PISA OECD Programme for International Student Assessment IRAs 53 rd Annual Convention Atlanta, 6 May 2008 How the demand for skills has changed Economy-wide measures of routine and non-routine task input (US) (Levy and Murnane) Mean task input as percentiles of the 1960 task distribution To analyse, compare, contrast, and evaluate To think imaginatively To apply knowledge in real-life situations To communicate thoughts and ideas effectively PISA concept of literacy Accessing, managing, integrating and evaluating written information in order to develop ones knowledge and potential, and to participate in, and contribute to, society To analyse, compare, contrast, and evaluate To think imaginatively To apply knowledge in real-life situations To communicate thoughts and ideas effectively Reading literacy Using, interpreting and reflecting on written material

PISA OECD Programme for International Student Assessment IRAs 53 rd Annual Convention Atlanta, 6 May 2008 How the demand for skills has changed Economy-wide measures of routine and non-routine task input (US) (Levy and Murnane) Mean task input as percentiles of the 1960 task distribution To analyse, compare, contrast, and evaluate To think imaginatively To apply knowledge in real-life situations To communicate thoughts and ideas effectively PISA concept of literacy Accessing, managing, integrating and evaluating written information in order to develop ones knowledge and potential, and to participate in, and contribute to, society To analyse, compare, contrast, and evaluate To think imaginatively To apply knowledge in real-life situations To communicate thoughts and ideas effectively Reading literacy Using, interpreting and reflecting on written material To analyse, compare, contrast, and evaluate To think imaginatively To apply knowledge in real-life situations To communicate thoughts and ideas effectively Mathematical literacy Emphasis is on mathematical knowledge put into functional use in a multitude of different situations in varied, reflective and insight-based ways

PISA OECD Programme for International Student Assessment IRAs 53 rd Annual Convention Atlanta, 6 May 2008 How the demand for skills has changed Economy-wide measures of routine and non-routine task input (US) (Levy and Murnane) Mean task input as percentiles of the 1960 task distribution To analyse, compare, contrast, and evaluate To think imaginatively To apply knowledge in real-life situations To communicate thoughts and ideas effectively PISA concept of literacy Accessing, managing, integrating and evaluating written information in order to develop ones knowledge and potential, and to participate in, and contribute to, society To analyse, compare, contrast, and evaluate To think imaginatively To apply knowledge in real-life situations To communicate thoughts and ideas effectively Reading literacy Using, interpreting and reflecting on written material To analyse, compare, contrast, and evaluate To think imaginatively To apply knowledge in real-life situations To communicate thoughts and ideas effectively Mathematical literacy Emphasis is on mathematical knowledge put into functional use in a multitude of different situations in varied, reflective and insight-based ways To analyse, compare, contrast, and evaluate To think imaginatively To apply knowledge in real-life situations To communicate thoughts and ideas effectively Scientific literacy Using scientific knowledge, identifying scientific questions, and drawing evidence-based conclusions to understand and make decisions about the natural world

PISA OECD Programme for International Student Assessment IRAs 53 rd Annual Convention Atlanta, 6 May 2008 PISA A three-yearly global assessment that… … examines the performance of 15-year-olds in key subject areas as well as a wider range of educational outcomes Including students attitudes to learning and their learning behaviour …collects contextual data from… …students, parents, schools and systems… …in order to identify policy levers Coverage Representative samples of between 3,500 and 50, year-old students drawn in each country Most federal countries also draw regional samples PISA covers roughly 90% of the world economy.

PISA OECD Programme for International Student Assessment IRAs 53 rd Annual Convention Atlanta, 6 May 2008 Deciding what to assess... looking back at what students were expected to have learned …or… looking ahead to how well they can extrapolate from what they have learned and apply their knowledge and skills in novel settings. For PISA, the OECD countries chose the latter.

PISA OECD Programme for International Student Assessment IRAs 53 rd Annual Convention Atlanta, 6 May PISA countries in Coverage of world economy 77%81% 83% 85%86%87%

PISA OECD Programme for International Student Assessment IRAs 53 rd Annual Convention Atlanta, 6 May 2008 Average performance of 15-year-olds in scientific literacy– extrapolate and apply (2006) High science performance Low science performance … 18 countries perform below this line

PISA OECD Programme for International Student Assessment IRAs 53 rd Annual Convention Atlanta, 6 May 2008 Average performance of 15-year-olds in science – extrapolate and apply Low average performance Large socio-economic disparities High average performance Large socio-economic disparities Low average performance High social equity High average performance High social equity Strong socio- economic impact on student performance Socially equitable distribution of learning opportunities High science performance Low science performance

PISA OECD Programme for International Student Assessment IRAs 53 rd Annual Convention Atlanta, 6 May 2008 Durchschnittliche Schülerleistungen im Bereich Mathematik Low average performance Large socio-economic disparities High average performance Large socio-economic disparities Low average performance High social equity High average performance High social equity Strong socio- economic impact on student performance Socially equitable distribution of learning opportunities High science performance Low science performance

PISA OECD Programme for International Student Assessment IRAs 53 rd Annual Convention Atlanta, 6 May 2008 School performance and socio-economic background Germany Student performance Advantage PISA Index of socio-economic background Disadvantage Schools proportional to size Student performance and students socio-economic background within schools School performance and schools socio-economic background Student performance and students socio-economic background

PISA OECD Programme for International Student Assessment IRAs 53 rd Annual Convention Atlanta, 6 May 2008 School performance and socio-economic background United States Student performance Advantage PISA Index of socio-economic background Disadvantage Schools proportional to size Student performance and students socio-economic background within schools School performance and schools socio-economic background Student performance and students socio-economic background

PISA OECD Programme for International Student Assessment IRAs 53 rd Annual Convention Atlanta, 6 May 2008 School performance and socio-economic background Finland Student performance Advantage PISA Index of socio-economic background Disadvantage Schools proportional to size Student performance and students socio-economic background within schools School performance and schools socio-economic background Student performance and students socio-economic background

PISA OECD Programme for International Student Assessment IRAs 53 rd Annual Convention Atlanta, 6 May 2008 Variation in student performance OECD (2007), Learning for tomorrows world: First results from PISA 2006, Table 4.1a 20

PISA OECD Programme for International Student Assessment IRAs 53 rd Annual Convention Atlanta, 6 May 2008 Variation of performance between schools Variation of performance within schools OECD (2004), Learning for tomorrows world: First results from PISA 2003, Table 4.1a Variation in student performance

PISA OECD Programme for International Student Assessment IRAs 53 rd Annual Convention Atlanta, 6 May 2008 Investments and outcomes r Since 2000, expenditure per primary and secondary student increased across OECD countries by 39% (in real terms) … …while PISA literacy outcomes generally remained flat… …but there are notable exceptions…

PISA OECD Programme for International Student Assessment IRAs 53 rd Annual Convention Atlanta, 6 May 2008 Poland raised its reading performance by 28 PISA points, equivalent to ¾ of a school year - What happened? OECD (2007), Learning for tomorrows world: First results from PISA 2006, Table 6.1a 20 In 2003, performance variation among schools had fallen from 51% to 16% of the variation of student performance But did this lead to genuine improvements of school performance? Between 2000 and 2003 showed the second- largest increase in reading (17 points) and a further 11 point increase since 2003 Most of that increase resulted from smaller proportions at the bottom level (23% in 2000, and three-quarters in vocational tracks, 17%in 2003) Did this harm the better performers?

PISA OECD Programme for International Student Assessment IRAs 53 rd Annual Convention Atlanta, 6 May 2008 Increased likelihood of postsec. particip. at age 19 associated with reading proficiency at age 15 (Canada) after accounting for school engagement, gender, mother tongue, place of residence, parental, education and family income (reference group Level 1)

PISA OECD Programme for International Student Assessment Briefing of Council 14 November 2007 How to get there Some policy levers that emerge from international comparisons

PISA OECD Programme for International Student Assessment IRAs 53 rd Annual Convention Atlanta, 6 May 2008 Some myths r No relationship between size of countries and average performance r No relationship between proportion of immigrants and average performance r Few difference in students reported test motivation r Limited impact of national item preferences.

PISA OECD Programme for International Student Assessment IRAs 53 rd Annual Convention Atlanta, 6 May 2008 High ambitions and universal standards Access to best practice and quality professional development r Sympathy doesnt raise standards – aspiration does PISA suggests that students and schools perform better in a climate characterised by high expectations and the readiness to invest effort, the enjoyment of learning, a strong disciplinary climate, and good teacher-student relations –Among these aspects, students perception of teacher-student relations and classroom disciplinary climate display the strongest relationships

PISA OECD Programme for International Student Assessment IRAs 53 rd Annual Convention Atlanta, 6 May 2008 Challenge and support Weak support Strong support Low challenge High challenge Strong performance Systemic improvement Poor performance Improvements idiosyncratic Conflict Demoralisation Poor performance Stagnation

PISA OECD Programme for International Student Assessment IRAs 53 rd Annual Convention Atlanta, 6 May 2008 School principals perceptions of parents expectations Percentage of students in schools where the principal reported that regarding high academic standards

PISA OECD Programme for International Student Assessment IRAs 53 rd Annual Convention Atlanta, 6 May 2008 The role of books and engagement with reading r Results from PISA show… Students from advantaged backgrounds… …have a greater chance of coming to school more engaged in reading and entering into a virtuous circle of increasing reading interest and improved reading performance …but not all engaged students come from privileged homes… …and those from more modest backgrounds who read regularly and feel positive about it are better readers than students with home advantages and weaker reading engagement Schools can make a significant difference to bring students into the virtuous circle –Seeking mutual reinforcement of cognitive skills and motivation, particularly for boys.

PISA OECD Programme for International Student Assessment IRAs 53 rd Annual Convention Atlanta, 6 May 2008 Access to books at home Low Performance High Performance OECD average performance of students who have 10 or fewer books at home Performance of students with 11 – 50 books at home Performance of students with 51 – 100 books at home Performance of students with 101 – 250 books at home OECD average READING performance of students with more than 250 books (PISA 2000) Much of the difference remains after accounting for social background, and… Cultural capital appears more closely related to student performance than family wealth.

PISA OECD Programme for International Student Assessment IRAs 53 rd Annual Convention Atlanta, 6 May 2008 Low Performance High Performance Public and school libraries OECD average performance of students who never or hardly ever borrow books to read for pleasure from a public library A few times per year About once a month or more (PISA 2000).

PISA OECD Programme for International Student Assessment IRAs 53 rd Annual Convention Atlanta, 6 May 2008 Reading for enjoyment Low Performance High Performance OECD average performance of students who never or hardly ever read for enjoyment 30 minutes or less each day 30 minutes to one hour One hour or more And moreover… Predictive power of reading activities almost equally strong for mathematics and science performance.

PISA OECD Programme for International Student Assessment IRAs 53 rd Annual Convention Atlanta, 6 May 2008 A profile of reading engagement Low Performance High Performance OECD average performance of least diversified readers (22%) (only magazines frequently read) Moderately diversified readers (27%) (Typical materials are magazines or newspapers) Diversified readers in short texts (28%) (Magazines, newspapers, comics and moderate readers of fiction and non-fiction) Diversified readers in long texts (22%) (Magazines, newspapers, demanding texts and books).

PISA OECD Programme for International Student Assessment IRAs 53 rd Annual Convention Atlanta, 6 May 2008 Reading engagement r Having diverse reading material at home is strongly associated with high overall student performane and engagement in reading, which includes positive attitudes towards reading… …but there is much schools can do to bring students into the virtuous circle of increasing reading interest and student performance r Improvement in literacy performance relies not just on improving student cognitive skills but also on increasing their engagement in reading r Engagement in reading may be an effective policy lever to mediate the impact of social background on performance r The emergence at relatively early ages of, for example, gender differences in reading performance and engagement underline the importance of an early start.

PISA OECD Programme for International Student Assessment IRAs 53 rd Annual Convention Atlanta, 6 May 2008 High ambitions Access to best practice and quality professional development Accountability and intervention in inverse proportion to success Devolved responsibility, the school as the centre of action

PISA OECD Programme for International Student Assessment IRAs 53 rd Annual Convention Atlanta, 6 May 2008 Public and private schools Private schools perform better Public schools perform better % Score point difference

PISA OECD Programme for International Student Assessment IRAs 53 rd Annual Convention Atlanta, 6 May 2008 Pooled international dataset, effects of selected school/system factors on science performance after accounting for all other factors in the model OECD (2007), PISA 2006 – Science Competencies from Tomorrows World, Table 6.1a 20 Schools practicing ability grouping (gross and net) Academically selective schools (gross and net) but no system-wide effect School results posted publicly (gross and net) One additional hour of science learning at school (gross and net) One additional hour of out- of-school lessons (gross and net) One additional hour of self-study or homework (gross and net) School activities to promote science learning (gross and net) Schools with greater autonomy (resources) (gross and net) Each additional 10% of public funding (gross only) Schools with more competing schools (gross only) School principals perception that lack of qualified teachers hinders instruction (gross only) School principals positive evaluation of quality of educational materials (gross only) Measured effect Effect after accounting for the socio-economic background of students, schools and countries 64% of US students in schools that compete with more than 2 schools in same area, 11% with one school, 26% with no school 91% of US students in schools posting achievement data publicly (OECD 38%) 26% of US students in schools with no vacant science teaching positions (OECD 38%), 71% where all vacant positions had been filled (OECD 59%), BUT 20% where principals report that instruction is hindered by a lack of qualified science teachers

PISA OECD Programme for International Student Assessment IRAs 53 rd Annual Convention Atlanta, 6 May 2008 PISA score in science School autonomy, standards-based examinations and science performance School autonomy in selecting teachers for hire

PISA OECD Programme for International Student Assessment IRAs 53 rd Annual Convention Atlanta, 6 May 2008 Strong ambitions Access to best practice and quality professional development Accountability Devolved responsibility, the school as the centre of action Integrated educational opportunities From prescribed forms of teaching and assessment towards personalised learning

PISA OECD Programme for International Student Assessment IRAs 53 rd Annual Convention Atlanta, 6 May 2008 Durchschnittliche Schülerleistungen im Bereich Mathematik Low average performance Large socio-economic disparities High average performance Large socio-economic disparities Low average performance High social equity High average performance High social equity Strong socio- economic impact on student performance Socially equitable distribution of learning opportunities High science performance Low science performance Early selection and institutional differentiation High degree of stratification Low degree of stratification

PISA OECD Programme for International Student Assessment IRAs 53 rd Annual Convention Atlanta, 6 May 2008 High ambitions Access to best practice and quality professional development Accountability and intervention in inverse proportion to success Personalized learning Devolved responsibility, the school as the centre of action Integrated educational opportunities

PISA OECD Programme for International Student Assessment IRAs 53 rd Annual Convention Atlanta, 6 May 2008 Why care? r Progress Concerns about skill barriers to economic growth, productivity growth and rates of technological innovation –One additional year of education equals to between 3 and 6% of GDP –Rising college-level qualifications seem generally not to have led to an inflation of the labour-market value of qualifications (in all but three of the 20 countries with available data, the earnings benefit increased between 1997 and 2003, in Germany, Italy and Hungary by between 20% and 40%) r Fairness Concerns about the role of skills in creating social inequity in economic outcomes –Both average and distribution of skill matter to long-term growth r Value for money Concerns about the demand for, and efficiency and effectiveness of, investments in public goods

PISA OECD Programme for International Student Assessment IRAs 53 rd Annual Convention Atlanta, 6 May –All national and international publications –The complete micro-level database …and remember: Without data, you are just another person with an opinion

PISA OECD Programme for International Student Assessment Briefing of Council 14 November 2007 Backup slides

PISA OECD Programme for International Student Assessment IRAs 53 rd Annual Convention Atlanta, 6 May 2008 School principals reports on vacant science teaching positions and their perceptions of the supply of qualified science teachers

PISA OECD Programme for International Student Assessment IRAs 53 rd Annual Convention Atlanta, 6 May 2008 Effort expended by students in PISA 2003 (Butler and Adams, 2007)

PISA OECD Programme for International Student Assessment IRAs 53 rd Annual Convention Atlanta, 6 May 2008 Effort expended by students in PISA 2003, relative to an important school test (Butler and Adams, 2007)

PISA OECD Programme for International Student Assessment IRAs 53 rd Annual Convention Atlanta, 6 May 2008 Ranks comparisons: Overall vs favourites Rank on favourites higher than overall rank Rank on favourites lower than overall rank Korea 3 rd overall 9 th on favourites Norway 13 th overall 10 th on favourites For all other countries, the ranks were not significantly different. Rank on all items Rank on own most appropriate items