1. Knowledge Economy Forum Organisation for Economic Cooperation and Development (OECD) Results from PISA 2003 Istanbul March 22, 2005 Dr John Cresswell OECD/ Directorate for Education

2. In the dark all education systems look the same

3. In the light, differences between education systems can be seen

4. In the light, differences between education systems can be seen

5. Origins of PISA r OECD work on education statistics and indicators major development commenced in late 1980s most of it funded by voluntary contributions substantial Member engagement through networks r Network on educational outcomes led by US developed a proposal for measurement of outcomes Education Committee formally initiated activity in 1996 –11 Members initially committed Council decision in 1997 –established decentralised Part II programme (became PISA) –Council required that OECD face no costs or financial risks –virtually all OECD Members signed on by then –Education Ministries have paid all costs and bear all risks

6. OECD Partner countries OECD countries PISA 2000 country participation

7. OECD Partner countries OECD countries PISA 2003 country participation

8. OECD Partner countries OECD countries PISA 2006 country participation

9. Making international comparisons of achievement requires decisions about... what to assess, whom to assess.

10. Deciding what to assess... looking back at what they were expected to have learned OR looking ahead to what they can do with what they have learned. For PISA, the OECD countries chose the latter.

11. PISA assessments r Reading literacy Using, interpreting and reflecting on written material. r Mathematical literacy Recognising problems that can be solved mathematically, representing them mathematically, solving them. r Scientific literacy Identifying scientific questions, recognising what counts as scientific evidence, using evidence to draw conclusions about the natural world.

12. Development of the PISA tests

13. Development of assessments r Frameworks by international experts r Assessment materials submitted by countries developed by research consortium screened for cultural bias –by countries –by expert, international panel –items with prima facie cultural bias removed at this stage translated from English & French originals trialled to check items working consistently in all countries r Final tests items shown in trial to be culturally biased removed best items chosen for final tests –balanced to reflect framework –range of difficulties –range of item types (constructed response, multiple choice)

14. Measuring mathematical literacy in PISA 2003

15. Mathematical literacy in PISA r The capacity to: identify, understand and engage in mathematics; make well-founded judgements about the role that mathematics plays in an individual’s current and future: –private life –occupational life –social life with peers and relatives –life as a constructive, concerned and reflective citizen. r Seen as depending on… mathematical knowledge and skills, ability to think and work mathematically, ability to apply the knowledge in a wide variety of contexts.

16. Measuring mathematical literacy in PISA 2003 r Content Space and shape (assessed in PISA 2000) Change and relationships (assessed in PISA 2000) Quantity Uncertainty r Process skills Reproduction: use of practised knowledge, routine procedures… Connections: somewhat familiar but not routine… Reflection: insight, creativity in choosing mathematical concepts… r Context Personal Educational or occupational Public Scientific

17. Space & shape item Answers: Yes, No, Yes, Yes Process skill: connections Context: educational quasi-realistic problem typical in maths classes not genuine occupational problem Form: complex multiple-choice Source: OECD (2004) Learning for tomorrow’s world: First results from PISA 2003, Figure 2.4a, p.52.

18. Change and relationships item Scores: 1 for n = 140x0.8 = 112 but no further work shown 2 for correct steps/min but not m/min; correct m/min but not km/hr; correct method but error of calculation; correct km/hr but not giving m/sec 3 for correct m/min (89.6) and m/hr (5.4), rounding acceptable. Process skill: score 1=connections score 2=connections score 3=reflection Context: personal Form: open-constructed

19. Quantity item Form: open constructed response Answer: Yes, with adequate explanation Process skill: reflection Context: public Form: short constructed response Answer: 12 600 ZAR (unit not required) Process skill: reproduction Context: public Form: short constructed response Answer: 975 SGD (unit not required) Process skill: reproduction Context: public

20. Uncertainty item Scores: 1 for “No, not reasonable” but explanation lacking detail (e.g. focusing on exact increase in number of robberies without comparison with total) 2 for “No, not reasonable” with argument focusing on only small part of graph shown, ratio or percentage increase, or need for trend data. Process skill: connections Context: personal Form: open- constructed

21. Deciding whom to assess... grade-based sample OR age-based sample For PISA, the OECD countries chose the latter, selecting 15-year-olds in school as the population.

22. Key features of PISA 2003 assessment r Information collected each student –2 hours on paper-and-pencil tasks (subset of all questions) –½ hour for questionnaire on background, learning habits, learning environment, engagement and motivation school principals –questionnaire (school demography, learning environment quality) r Sample 275,000 students 41 participating countries

23. PISA sampling requirements r Population: all 15-year-olds in school r Sample minimum of 150 schools per country two random samples: schools and replacement schools if school declines, replacement school is invited stringent requirements set by countries (85% of selected schools, 80% of selected students within schools)

24. Results from PISA 2003

25. PISA provides five key benchmarks for the quality of education systems 1.Overall performance of education systems 2.Equity in the distribution of learning opportunities Measured by the impact students’ and schools’ socio-economic background has on performance… …not merely by the distribution of learning outcomes 3.Consistency of performance standards across schools 4.Gender differences 5.Foundations for lifelong learning Learning strategies, motivation and attitudes

26. OECD (2004), Learning for tomorrow’s world: First results from PISA 2003, Table 2.5c, p.356. Mean mathematics scores – selected countries

27. OECD Level 6 Level 5 Level 4 Level 3 Level 2 Level 1 Below Level 1 OECD (2004), Learning for tomorrow’s world: First results from PISA 2003, Table 2.5a, p.354. What students can do in mathematics 15 % 21 % 22 % 18 % 10 % 4% 11 %

28. Percentage of students at each of the proficiency levels on the mathematics scale Level 3 Level 1 Below Level 1 Level 6 Level 5 Level 4 Level 2 OECD (2004), Learning for tomorrow’s world: First results from PISA 2003, Table 2.5a, p.354.

29. What students can do in reading 10% 22% 12% 6%6% 22% 29% OECD Average Level 5 Level 4 Level 3 Level 2 Level 1 Below Level 1 OECD (2004), Learning for tomorrow’s world: First results from PISA 2003, Table 6.1, p.443.

30. Percentage of students at each of the proficiency levels in reading OECD (2004), Learning for tomorrow’s world: First results from PISA 2003, Table 6.1, p.443.

31. Performance in all domains Mathematics ReadingScience Problem Solving

32. Securing an equitable distribution of learning opportunities Measured by the impact students’ and schools’ socio-economic background has on performance – not merely by the distribution of learning outcomes

33. High Student performance Social background and student performance Advantage PISA Index of social background Disadvantage Low OECD (2004), Learning for tomorrow’s world: First results from PISA 2003, Figure 4.8, p.176.

34. Ensuring consistent performance standards across schools Between and within-school variation in performance

35. Is it all innate ability? Variation in student performance OECD (2004), Learning for tomorrow’s world: First results from PISA 2003, Table 4.1a, p.383.

36. Variation of performance between schools Variation of performance within schools Is it all innate ability? Variation in student performance in mathematics OECD (2004), Learning for tomorrow’s world: First results from PISA 2003, Table 4.1a, p.383.

37. Variation in student performance in mathematics Variation of performance between schools Variation of performance within schools Variation explained by socio-economic level of students and schools OECD (2004), Learning for tomorrow’s world: First results from PISA 2003, Table 4.1a, p.383.

38. Student performance School performance and schools’ socio- economic background - Japan Advantage PISA Index of social background Disadvantage Figure 4.13 Student performance and student SES Student performance and student SES within schools School performance and school SES School proportional to size

39. Student performance School performance and schools’ socio- economic background – Sweden Advantage PISA Index of social background Disadvantage Figure 4.13 Student performance and student SES Student performance and student SES within schools School performance and school SES School proportional to size

40. Bridging the gender gap Performance, attitudes and motivation

41. Gender differences r In reading, girls are far ahead In all countries, girls significantly outperform boys in reading r In mathematics, boys tend to be somewhat ahead In most countries, boys outperform girls …but mostly by modest amounts… … within classrooms and schools, the gender gap is often larger Strong problem-solving performance for girls suggests… …that it is not the cognitive processes underlying mathematics that give boys an advantage… …but the context in which mathematics appears in school Gender differences in interest and attitudes towards mathematics are significantly greater than the observed performance gap –Girls report much lower interest in mathematics, more negative attitudes and much greater anxiety with mathematics… …and this may well contribute to the significant gender difference in educational and occupational pathways in mathematics-related subjects

42. Performance in mathematics Females perform better Males perform better Performance in reading Females perform better Males perform better OECD (2004), Learning for tomorrow’s world: First results from PISA 2003, Tables 2.5c, 6.3, pp.356, 445. Gender differences

43. Anxiety in mathematics and performance in mathematics Females Males LowHigh OECD (2004), Learning for tomorrow’s world: First results from PISA 2003, Table 3.8, p.374.

44. Creating strong foundations for lifelong learning Performance, attitudes and motivation

45. Interest in and enjoyment of mathematics OECD (2004), Learning for tomorrow’s world: First results from PISA 2003, Table 3.4, p.367 and Figure 3.4, p.126.

46. Anxiety in mathematics OECD (2004), Learning for tomorrow’s world: First results from PISA 2003, Table 3.8, p.374 and Figure 3.8, p.139.

47. Some features of successful education systems Insights from earlier PISA analysis

48. Student performance and spending per student Mexico Greece Portugal Italy Spain Germany Austria Ireland United States Norway Korea Czech republic Slovak republic Poland Hungary Finland Netherlands Canada Switzerland Iceland Denmark France Sweden Belgium Australia Japan R 2 = 0.28 Cumulative expenditure (US$) Performance in mathematics

49. Pre-school attendance and performance Percentage of students who attended pre-school Difference in performance between those who attended pre-school for more than one year and those with no pre-school Difference in performance between those who attended pre-school for one year or less and those with no pre-school 38 score points is the average performance difference associated with one school year

50. Governance of the school system r In many of the best performing countries Decentralised decision-making is combined with devices to ensure a fair distribution of substantive educational opportunities The provision of standards and curricula at national/subnational levels is combined with advanced evaluation systems –That are implemented by professional agencies Process-oriented assessments and/or centralised final examinations are complimented with individual reports and feed-back mechanisms on student learning progress

51. Public and private schools Private schools perform better Public schools perform better

52. Organisation of instruction r In many of the best performing countries Schools and teachers have explicit strategies and approaches for teaching heterogeneous groups of learners –A high degree of individualised learning processes –Disparities related to socio-economic factors and migration are recognised as major challenges Students are offered a variety of extra- curricular activities Schools offer differentiated support structures for students –E.g. school psychologists or career counsellors Institutional differentiation is introduced, if at all, at later stages –Integrated approaches also contributed to reducing the impact of students socio-economic background on outcomes

53. Mathematics performance Decreasing effect of socioeconomic background High performance Low SES effect Low performance Low SES effect Low performance High SES effect High performance High SES effect

54. Support systems and professional teacher development r In the best performing countries Effective support systems are located at individual school level or in specialised support institutions Teacher training schemes are selective The training of pre-school personnel is closely integrated with the professional development of teachers Continuing professional development is a constitutive part of the system Special attention is paid to the professional development of school management personnel

55. Student approaches to learning r The ability to manage one’s learning is both an important outcome of education and a contributor to student literacy skills at school Learning strategies, motivation, self-related beliefs, preferred learning styles r Different aspects of students’ learning approaches are closely related Well-motivated and self-confident students tend to invest in effective learning strategies and this contributes to their literacy skills r Immigrant students tend to be weaker performers …but they do not have weaker characteristics as learners r Boys and girls each have distinctive strengths and weaknesses as learners Girls stronger in relation to motivation and self- confidence in reading Boys believing more than girls in their own efficacy as learners and in their mathematical abilities

56. Teacher support in mathematics Students’ views OECD (2004), Learning for tomorrow’s world: First results from PISA 2003, Table 5.1a, p.403 and Figure 5.1, p.213.

57. Disciplinary climate Students’ views OECD (2004), Learning for tomorrow’s world: First results from PISA 2003, Table 5.3a, p.408 and Figure 5.3, p.217.

58. Teachers’ morale and commitment Principals’ views OECD (2004), Learning for tomorrow’s world: First results from PISA 2003, Table 5.5a, p.412 and Figure 5.5, p.223.

59. Further information www.pisa.oecd.org –All national and international publications –The complete micro-level database email: pisa@oecd.org john.cresswell@oecd.org