Programme for International Student Assessment - PISA Organisation for Economic Cooperation and Development (OECD)

Origins of PISA r OECD work on education statistics and indicators major development commenced in late 1980s major development commenced in late 1980s r Network on educational outcomes Council decision in 1997 Council decision in 1997

OECD Partner countries (4) OECD countries (28) PISA 2000 country participation

OECD Partner countries (11) OECD countries (30) PISA 2003 country participation

OECD Partner countries (28) OECD countries (30) PISA 2006 country participation

Making international comparisons of achievement requires decisions about... what to assess, whom to assess.

Deciding what to assess... looking back at what they were expected to have learned OR looking ahead to what they can do with what they have learned. For PISA, the OECD countries chose the latter.

PISA assessments r Reading literacy Using, interpreting and reflecting on written material. Using, interpreting and reflecting on written material. r Mathematical literacy Recognising problems that can be solved mathematically, representing them mathematically, solving them. Recognising problems that can be solved mathematically, representing them mathematically, solving them. r Scientific literacy Identifying scientific questions, recognising what counts as scientific evidence, using evidence to draw conclusions about the natural world. Identifying scientific questions, recognising what counts as scientific evidence, using evidence to draw conclusions about the natural world.

Development of the PISA tests

Development of assessments r Frameworks by international experts r Assessment materials submitted by countries developed by research consortium screened for cultural bias translated into English & French originals trialled to check items working consistently in all countries r Final tests items shown in trial to be culturally biased removed best items chosen for final tests –balanced to reflect framework –range of difficulties –range of item types

Measuring mathematical literacy in PISA 2003

Mathematical literacy in PISA The real world The mathematical World A real situation A model of reality A mathematical model Mathematical results Real results Understanding, structuring and simplifying the situation Making the problem amenable to mathematical treatment Interpreting the mathematical results Using relevant mathematical tools to solve the problem Validating the results

Mathematical literacy in PISA r The capacity to: identify, understand and engage in mathematics; identify, understand and engage in mathematics; make well-founded judgements about the role that mathematics plays in an individual’s current and future: make well-founded judgements about the role that mathematics plays in an individual’s current and future: –private life –occupational life –social life with peers and relatives –life as a constructive, concerned and reflective citizen. r Seen as depending on… mathematical knowledge and skills, mathematical knowledge and skills, ability to think and work mathematically, ability to think and work mathematically, ability to apply the knowledge in a wide variety of contexts. ability to apply the knowledge in a wide variety of contexts.

Measuring mathematical literacy in PISA 2003 r Content Space and shape Space and shape Change and relationships Quantity Change and relationships Quantity Uncertainty Uncertainty r Process skills Reproduction: use of practised knowledge, routine procedures… Reproduction: use of practised knowledge, routine procedures… Connections: somewhat familiar but not routine… Connections: somewhat familiar but not routine… Reflection: insight, creativity in choosing mathematical concepts… Reflection: insight, creativity in choosing mathematical concepts…

Deciding whom to assess... grade-based sample OR age-based sample For PISA, the OECD countries chose the latter, selecting 15-year-olds in school as the population.

PISA sampling requirements r Population: all 15-year-olds in school r Sample minimum of 150 schools per country two random samples: schools and replacement schools if school declines, replacement school is invited stringent requirements set by countries (85% of selected schools, 80% of selected students within schools)

Key features of PISA 2003 assessment r Information collected each student –2 hours on paper-and-pencil tasks (subset of all questions) –½ hour for questionnaire on background, learning habits, learning environment, engagement and motivation school principals –questionnaire (school demography, learning environment quality) r Sample 275,000 students 41 participating countries

Results from PISA 2003

PISA provides five key benchmarks for the quality of education systems 1.Overall performance of education systems 2.Equity in the distribution of learning opportunities 3.Consistency of performance standards across schools 4.Gender differences 5.Foundations for lifelong learning

OECD (2004), Learning for tomorrow’s world: First results from PISA 2003, Table 2.5c, p.356. Mean mathematics scores – selected countries

OECD Level 6 Level 5 Level 4 Level 3 Level 2 Level 1 Below Level 1 OECD (2004), Learning for tomorrow’s world: First results from PISA 2003, Table 2.5a, p.354. What students can do in mathematics 15 % 21 % 22 % 18 % 10 % 4% 11 %

Percentage of students at each of the proficiency levels on the mathematics scale Level 3 Level 1 Below Level 1 Level 6 Level 5 Level 4 Level 2 OECD (2004), Learning for tomorrow’s world: First results from PISA 2003, Table 2.5a, p.354.

What students can do in reading 10% 22% 12% 6%6% 22% 29% OECD Average Level 5 Level 4 Level 3 Level 2 Level 1 Below Level 1 OECD (2004), Learning for tomorrow’s world: First results from PISA 2003, Table 6.1, p.443.

Percentage of students at each of the proficiency levels in reading OECD (2004), Learning for tomorrow’s world: First results from PISA 2003, Table 6.1, p.443.

Performance in all domains Mathematics ReadingScience Problem Solving

Securing an equitable distribution of learning opportunities Measured by the impact students’ and schools’ socio-economic background has on performance – not merely by the distribution of learning outcomes

High Student performance Social background and student performance Advantage PISA Index of social background Disadvantage Low OECD (2004), Learning for tomorrow’s world: First results from PISA 2003, Figure 4.8, p.176. On average, there is a 45 point change in mathematics score for a one standard deviation change in social background

Student performance School performance and schools’ socio- economic background - Sweden Advantage PISA Index of social background Disadvantage Figure 4.13 School proportional to size Student performance and student SES

Low Performance High performance Low performance Low social equity High performance Low social equity High performance High social equity Strong impact of social background Moderate impact of social background   Low performance High social equity

Student performance and spending per student Mexico Greece Portugal Italy Spain Germany Austria Ireland United States Norway Korea Czech republic Slovak republic Poland Hungary Finland Netherlands Canada Switzerland Iceland Denmark France Sweden Belgium Australia Japan R 2 = 0.28 Cumulative expenditure (US$) Performance in mathematics r Spending per student is positively associated with average student performance… …but not a guarantee for high outcomes Australia, Belgium, Canada, the Czech Republic, Finland, Japan, Korea and the Netherlands do well in terms of “value for money”… …while some of the big spenders perform below-average

Ensuring consistent performance standards across schools Between and within-school variation in performance

Is it all innate ability? Variation in student performance OECD (2004), Learning for tomorrow’s world: First results from PISA 2003, Table 4.1a, p.383.

Variation of performance between schools Variation of performance within schools Is it all innate ability? Variation in student performance in mathematics OECD (2004), Learning for tomorrow’s world: First results from PISA 2003, Table 4.1a, p.383.

Bridging the gender gap Performance, attitudes and motivation

Gender differences r In reading, girls are far ahead In all countries, girls significantly outperform boys in reading r In mathematics, boys tend to be somewhat ahead in most countries …However …

Performance in mathematics Females perform better Males perform better Performance in reading Females perform better Males perform better OECD (2004), Learning for tomorrow’s world: First results from PISA 2003, Tables 2.5c, 6.3, pp.356, 445. Gender differences

Governance of the school system r In many of the best performing countries Decentralised decision-making is combined with devices to ensure a fair distribution of substantive educational opportunities The provision of standards and curricula at national/subnational levels is combined with advanced evaluation systems Process-oriented assessments and/or centralised final examinations are complemented with individual reports and feed-back mechanisms on student learning progress

Support systems and professional teacher development r In the best performing countries Effective support systems are located at individual school level or in specialised support institutions Teacher training schemes are selective The training of pre-school personnel is closely integrated with the professional development of teachers Continuing professional development is a constitutive part of the system Special attention is paid to the professional development of school management personnel

Student approaches to learning r The ability to manage one’s learning is both an important outcome of education and a contributor to student literacy skills at school Learning strategies, motivation, self-related beliefs, preferred learning styles r Different aspects of students’ learning approaches are closely related Well-motivated and self-confident students tend to invest in effective learning strategies and this contributes to their literacy skills r Immigrant students tend to be weaker performers …but they do not have weaker characteristics as learners r Boys and girls each have distinctive strengths and weaknesses as learners Girls stronger in relation to motivation and self- confidence in reading Boys believing more than girls in their own efficacy as learners and in their mathematical abilities

Thematic Reports r To complement the initial report. r In different areas of interest often based on options parts of the questionnaire r Two of particular interest: Where Immigrant Students Succeed Are Students Ready for a Technology Rich World

Key Issues r Policy attention is shifting from managing and containing migration inflows to addressing challenges of integration r Schools can play a central role in integration processes Preparation for school-work transitions Overcoming language barriers Transmission of norms and values r PISA provides first-time comparative data on cognitive and non-cognitive learning outcomes of immigrant students… Comparison with native peers Comparison with immigrant student populations across countries …and thus provides an opportunities to review policies and practices in this area

r The report compares three student populations… Native students are students who were born in the country of assessment or who had at least one parent born in that country Second-generation immigrant students are students who were born in the country of assessment, but whose parents were born in another country, i.e. students who have followed their entire school career in the country of assessment First-generation immigrant students are students who were not born in the country of assessment and whose parents were also born in another country

OECD average = 500 Mathematics performance Native students Second-generation students First-generation students Where immigrant students succeed – A comparative review of performance and engagement in PISA 2003: Figure 2.2a. r Key findings On average across the 17 countries, 15- year-old first-generation immigrants score in mathematics more than one school year behind their native counterparts The performance disadvantage varies widely across countries from negligible amounts to… …more than 90 score points in Belgium and Sweden for first-generation students …more than 90 score points in Belgium and Germany for second-generation students The performance of immigrant students also varies in absolute terms …with second-generation immigrants in Canada outperforming their German counterparts by 111 score points

Unemployment rates by immigration background % Where immigrant students succeed – A comparative review of performance and engagement in PISA 2003: Table 1.4.

Larger immigrant populations do not imply lower overall performance Percentage of immigrant students in the country Mathematics performance

Levels 5 and 6 Level 4 Level 3 Level 2 Level 1 Below 1 PISA Proficiency Levels Percentage of native students Percentage of first-generation immigrant students Where immigrant students succeed – A comparative review of performance and engagement in PISA 2003: Figure 2.4a. Mathematics performance by proficiency levels In PISA Level 2 demonstrates an essential foundation of mathematics skills

Students’ interest in and enjoyment of mathematics (OECD average) Native students Second- generation immigrant students First- generation immigrant students I enjoy reading about mathematics I look forward to my mathematics lessons I do mathematics because I enjoy it I am interested in the things I learn in mathematics Stronger in 9 countries Effect size 0.16 Stronger in 14 countries Effect size 0.32 Where immigrant students succeed – A comparative review of performance and engagement in PISA 2003: Figures 4.2 and 4.9.

Students’ anxiety in mathematics (OECD average) Native students Second- generation immigrant students First- generation immigrant students I often worry that it will be difficult for me in mathematics classes I get very tense when I have to do mathematics homework I get very nervous doing mathematics problems I feel helpless when doing a mathematics problem I worry that I will get poor marks in mathematics Weaker in 9 countries Effect size Weaker in 8 countries Effect size Where immigrant students succeed – A comparative review of performance and engagement in PISA 2003: Figures 4.6 and 4.9.

Are students ready for a technology-rich world? r First internationally comparative data on: The opportunities 15-year-old students have for using computers at home and at school How they use computers and their attitudes to them; The relationship between computer use and performance in key school subjects.

Access to computers at school has increased rapidly between PISA 2000 and PISA 2003…

…but in some countries students still have only limited opportunity to use computers at school.

Number of computers per student (PISA 2003) More than 10 students per computer Source: OECD (2005) Are students ready for a technology-rich world? What PISA studies tell us, Figure 2.8, p or fewer students per computer 1. Response rate too low to ensure comparability.

Access to computers at school is more universal than access to computers at home, but students report using computers much more frequently at home.

Percentage of students using a computer at least a few times each week 100% 0% At home At school Percentage of students reporting they use computers “Almost every day” or “A few times each week”: Source: OECD (2005) Are students ready for a technology-rich world? What PISA studies tell us, Figure 3.2, p Response rate too low to ensure comparability.

What do students use computers to do? r PISA asked students how often they used: The Internet to look up information about people things or ideas Games on a computer Word processing (e.g. ) The Internet to collaborate with a group or team Spreadsheets (e.g. ) The Internet to download software (including games) Drawing, painting or graphics programs on a computer Educational software such as mathematics programs The computer to help learn school material The Internet to download music The computer for programming A computer for electronic communication (e.g. or “chat rooms”) r Students could choose from the following answers: Almost every day, A few times each week, Between once a week and once a month, Less than once a month, Never

Students use computers for a wide range of purposes and not just to play games…

1. Response rate too low to ensure comparability. Students' use of computers (1) Percentage of students reporting they use the following “Almost every day” or “A few times each week”: Source: OECD (2005) Are students ready for a technology-rich world? What PISA studies tell us, Figures 3.3 and 3.4, pp.39 and 41. Games – 53% on average Internet research – 55% on average Word processing – 48% on average

… a minority of students frequently use educational software on computers…

1. Response rate too low to ensure comparability. Students' use of computers (2) Percentage of students reporting they use the following “Almost every day” or “A few times each week”: Source: OECD (2005) Are students ready for a technology-rich world? What PISA studies tell us, Figure 3.4, p.41. To learn school material - 30% on average Educational software - 13% on average

In general, students are confident in performing routine and Internet tasks on computers.

Routine tasks on a computer – percentage of students who are confident (OECD average) Open a file907 Play computer games907 Start a computer game8610 Save a computer document or file888 Delete a computer document or file888 Draw pictures using a mouse8510 Print a computer document or file869 Scroll a document up and down a screen878 Create/edit a document8013 Move files from one place to another on a computer7617 Copy a file from a floppy disk7516 I can do this… By myself With help Source: OECD (2005) Are students ready for a technology-rich world? What PISA studies tell us, Table 3.9, p.110.

Internet tasks on a computer – percentage of students who are confident (OECD average) I can do this… By myself With help At least 90% of students report confidence in these tasks in Australia, Canada, Iceland, Korea, New Zealand, Sweden and the United States. Get onto the Internet887 Write and send s7912 Copy or download files from the Internet7019 Download music from the Internet6621 Attach a file to an message5824 Source: OECD (2005) Are students ready for a technology-rich world? What PISA studies tell us, Table 3.11, p.112.

In general, 15-year-old boys report higher confidence than girls do in performing computing tasks and these differences are particularly apparent for the more demanding computing tasks...

High-level tasks on a computer – percentage of students who are confident to perform these tasks by themselves or with help (OECD average) Source: OECD (2005) Are students ready for a technology-rich world? What PISA studies tell us, Table 3.14, p.115. BoysGirls Use software to find and get rid of computer viruses7954 Create a multi-media presentation (with sound, pictures, video)7762 Create a computer program (e.g. in Logo, Pascal, Basic)6348 Construct a Web page7161 Create a presentation (e.g. using 7970 Use a spreadsheet to plot a graph7970 Use a database to produce a list of addresses8579

Students who are established computer users perform better than students with limited computing experience.

… and diminishes somewhat when socio-economic background factors are taken into account

If more experience counts, more frequent use does not necessarily Looking at a wide range of students’ use of computers, moderate users perform better than students who are either not using computers/using them rarely or are using computers very often…

Students reporting a moderate use of computers to perform a range of tasks Frequency of use of computer to perform a wide range of tasks and student performance Source: OECD (2005) Are students ready for a technology-rich world? What PISA studies tell us, Figure 4.6, p.65. Index of ICT Internet/entertainment use Index of ICT program/software use

Other research r “Northern lights” r Regional studies r Longitudinal studies r Science attitudes r Reading engagement r Mathematics anxiety r Indigenous students r Rural education r Selection practices

Other research r Social background r Teaching and learning strategies – cumulative study (caution) r Problem solving

Some country interests r Germany - social background, regional effects, effect of tracking, migration r Netherlands – social background, migration r Australia – longitudinal, indigenous, rural, regions r Japan – attitudes to science, r US – reading interest, difference in performance of students in TIMSS and in PISA r Belgium – regions, social background r Switzerland – grade sample

Some country interests r Denmark – longitudinal r Luxembourg – language background r Italy – regions r Ireland – relationship of PISA with National examinations r Turkey – school variation r Canada – longitudinal, province differences r Iceland – gender differences

Further information –All national and international publications –The complete database –Data analysis manuals (SPSS, SAS)