1. OECD World Forum “Statistics, Knowledge and Policy”, Palermo, 10-13 November 2004 1

2. OECD World Forum “Statistics, Knowledge and Policy”, Palermo, 10-13 November 2004 2 Development, validation and policy use of international, comparative indicators on education Barry McGaw Director for Education, OECD

3. OECD World Forum “Statistics, Knowledge and Policy”, Palermo, 10-13 November 2004 3 OECD Indicators of Education Systems Framework for indicators –current state of development –longer-term strategic objectives –comment on collaborative nature of the work Illustrations of development and use –educational attainment –quality of educational outcomes

4. OECD World Forum “Statistics, Knowledge and Policy”, Palermo, 10-13 November 2004 4 Framework for OECD education statistics National educ, social and economic context Structures, resource alloc and policies Social & economic outcomes of education Community and school characteristics Student learning, teacher working conditions Socio-economic background of learners Antecedents contextualise or constrain ed policy The learning environment at school Teaching, learning practices and classroom climate Individ attitudes, engagement and behaviour Output and performance of institutions Quality of instructional delivery Quality and distribution of knowledge & skills Policy Levers shape educational outcomes Outputs and Outcomes impact of learning Individual learner Level A Instructional settings Level B Schools, other institutions Level C Country or system Level D Domain 3Domain 2 Domain 1

5. OECD World Forum “Statistics, Knowledge and Policy”, Palermo, 10-13 November 2004 5 Well-established areas National educ, social and economic context Structures, resource alloc and policies Social & economic outcomes of education Community and school characteristics Student learning, teacher working conditions Socio-economic background of learners Antecedents contextualise or constrain ed policy The learning environment at school Teaching, learning practices and classroom climate Individ attitudes, engagement and behaviour Output and performance of institutions Quality of instructional delivery Quality and distribution of knowledge & skills Policy Levers shape educational outcomes Outputs and Outcomes impact of learning Individual learner Level A Instructional settings Level B Schools, other institutions Level C Country or system Level D Domain 3Domain 2 Domain 1 Educational finance, students, entrants, graduates, personnel Well-established and build on national systems… –Internationally agreed standards, definitions, and methods. –Joint OECD, UNESCO, EUROSTAT collection. …but need further consolidation –finance and graduates need refined international methodology and more coherent national implementation. –improved meta-data. …and an extension of their coverage, e.g. –Early childhood education. –International student mobility.

6. OECD World Forum “Statistics, Knowledge and Policy”, Palermo, 10-13 November 2004 6 Well-established areas National educ, social and economic context Structures, resource alloc and policies Social & economic outcomes of education Community and school characteristics Student learning, teacher working conditions Socio-economic background of learners Antecedents contextualise or constrain ed policy The learning environment at school Teaching, learning practices and classroom climate Individ attitudes, engagement and behaviour Output and performance of institutions Quality of instructional delivery Quality and distribution of knowledge & skills Policy Levers shape educational outcomes Outputs and Outcomes impact of learning Individual learner Level A Instructional settings Level B Schools, other institutions Level C Country or system Level D Domain 3Domain 2 Domain 1

7. OECD World Forum “Statistics, Knowledge and Policy”, Palermo, 10-13 November 2004 7 Areas under development National educ, social and economic context Structures, resource alloc and policies Social & economic outcomes of education Community and school characteristics Student learning, teacher working conditions Socio-economic background of learners Antecedents contextualise or constrain ed policy The learning environment at school Teaching, learning practices and classroom climate Individ attitudes, engagement and behaviour Output and performance of institutions Quality of instructional delivery Quality and distribution of knowledge & skills Policy Levers shape educational outcomes Outputs and Outcomes impact of learning Individual learner Level A Instructional settings Level B Schools, other institutions Level C Country or system Level D Domain 3Domain 2 Domain 1 Educational processes Annual collections –statutory teacher salaries –teachers’ working conditions –instruction time –pre-service training Five-yearly collection –distribution of decision-making responsibilities Strategic objective –survey on teachers, teaching and learning Quality review planned

8. OECD World Forum “Statistics, Knowledge and Policy”, Palermo, 10-13 November 2004 8 Areas under development National educ, social and economic context Structures, resource alloc and policies Social & economic outcomes of education Community and school characteristics Student learning, teacher working conditions Socio-economic background of learners Antecedents contextualise or constrain ed policy The learning environment at school Teaching, learning practices and classroom climate Individ attitudes, engagement and behaviour Output and performance of institutions Quality of instructional delivery Quality and distribution of knowledge & skills Policy Levers shape educational outcomes Outputs and Outcomes impact of learning Individual learner Level A Instructional settings Level B Schools, other institutions Level C Country or system Level D Domain 3Domain 2 Domain 1

9. OECD World Forum “Statistics, Knowledge and Policy”, Palermo, 10-13 November 2004 9 Areas under development National educ, social and economic context Structures, resource alloc and policies Social & economic outcomes of education Community and school characteristics Student learning, teacher working conditions Socio-economic background of learners Antecedents contextualise or constrain ed policy The learning environment at school Teaching, learning practices and classroom climate Individ attitudes, engagement and behaviour Output and performance of institutions Quality of instructional delivery Quality and distribution of knowledge & skills Policy Levers shape educational outcomes Outputs and Outcomes impact of learning Individual learner Level A Instructional settings Level B Schools, other institutions Level C Country or system Level D Domain 3Domain 2 Domain 1 Educational outcomes –PISA established 3-yearly surveys of 15-year-olds; benchmarking countries/regions –Longer-term objective longitudinal or multi-cohort to identify factors related to success

10. OECD World Forum “Statistics, Knowledge and Policy”, Palermo, 10-13 November 2004 10 Areas under development National educ, social and economic context Structures, resource alloc and policies Social & economic outcomes of education Community and school characteristics Student learning, teacher working conditions Socio-economic background of learners Antecedents contextualise or constrain ed policy The learning environment at school Teaching, learning practices and classroom climate Individ attitudes, engagement and behaviour Output and performance of institutions Quality of instructional delivery Quality and distribution of knowledge & skills Policy Levers shape educational outcomes Outputs and Outcomes impact of learning Individual learner Level A Instructional settings Level B Schools, other institutions Level C Country or system Level D Domain 3Domain 2 Domain 1

11. OECD World Forum “Statistics, Knowledge and Policy”, Palermo, 10-13 November 2004 11 Areas under development National educ, social and economic context Structures, resource alloc and policies Social & economic outcomes of education Community and school characteristics Student learning, teacher working conditions Socio-economic background of learners Antecedents contextualise or constrain ed policy The learning environment at school Teaching, learning practices and classroom climate Individ attitudes, engagement and behaviour Output and performance of institutions Quality of instructional delivery Quality and distribution of knowledge & skills Policy Levers shape educational outcomes Outputs and Outcomes impact of learning Individual learner Level A Instructional settings Level B Schools, other institutions Level C Country or system Level D Domain 3Domain 2 Domain 1 Labour market and social outcomes –Evidence from national labour force surveys earnings and employment continuing education and training … but to be done to harmonise them consistent implementation of ISCED international alignment of national taxonomies –Strategic objective: beyond qualifications to competencies measure adult competencies evaluate capacity of education/training to deliver competencies assess impact of competencies on economic & social outcomes … currently developing a detailed strategy

12. OECD World Forum “Statistics, Knowledge and Policy”, Palermo, 10-13 November 2004 12 It is a collaborative effort… With countries –National Co-ordinators (34) –Technical Group (72) –Network on Educational Outcomes (37) –Network on Education and Socio-economic Outcomes (40) –Network on School Features and Processes (36) With other international organisations –EUROSTAT –UNESCO –World Bank –Euridyce

13. OECD World Forum “Statistics, Knowledge and Policy”, Palermo, 10-13 November 2004 13 Data and indicators on educational attainment with international comparisons that provoke national policy debate.

14. OECD World Forum “Statistics, Knowledge and Policy”, Palermo, 10-13 November 2004 14 University-level attainment by age-group Source: OECD (2004) Education at Glance, Table A3.3, p.71 1 st 2 nd 13 th 22 nd 17 th 3 rd

15. OECD World Forum “Statistics, Knowledge and Policy”, Palermo, 10-13 November 2004 15 With data on secondary school completion suggesting further change in tertiary completion rates… -US has dropped from 1 st to 10 th -South Korea has risen from 24 th to 1 st

16. OECD World Forum “Statistics, Knowledge and Policy”, Palermo, 10-13 November 2004 16 Data and indicators on student learning OECD/PISA

17. OECD World Forum “Statistics, Knowledge and Policy”, Palermo, 10-13 November 2004 17 OECD/PISA Target populations –15-year-olds in school Assessments –Reading literacy Using, interpreting and reflecting on written material –Mathematical literacy Recognising a problem can be solved mathematically Mathematising problems to solve them –Scientific Literacy Identifying scientific questions Knowing what counts as evidence for dealing with such questions Drawing evidence-based conclusions

18. OECD World Forum “Statistics, Knowledge and Policy”, Palermo, 10-13 November 2004 18 20002003200620092012 Reading Maths Science Problem solving ICT literacy?? PISA assessment schedule

19. OECD World Forum “Statistics, Knowledge and Policy”, Palermo, 10-13 November 2004 19 PISA 2003 results will be released at 00.01 Paris time, Tuesday 7 December 2004.

20. OECD World Forum “Statistics, Knowledge and Policy”, Palermo, 10-13 November 2004 20 Participation in OECD/PISA PISA 2000 –28 of then 29 OECD Member countries –15 Non-Members (some testing in 2002) PISA 2003 –30 OECD Member countries –10 Non-Members PISA 2006 –30 OECD Member countries –28 Non-Members

21. OECD World Forum “Statistics, Knowledge and Policy”, Palermo, 10-13 November 2004 21 Comparisons of achievement levels.

22. OECD World Forum “Statistics, Knowledge and Policy”, Palermo, 10-13 November 2004 22 Source: OECD (2001) Knowledge and skills for life: First results from PISA 2000, Fig. 2.4, p.53 PISA 2000 Reading literacy means: OECD

23. OECD World Forum “Statistics, Knowledge and Policy”, Palermo, 10-13 November 2004 23 Source: OECD, UNESCO (2003) Literacy skills for the world of tomorrow, Fig. 2.5, p.76 PISA 2000 Reading literacy means: All

24. OECD World Forum “Statistics, Knowledge and Policy”, Palermo, 10-13 November 2004 24 % at each reading proficiency level Level 3 Level 2 Level 1 Below Level 1 Source: OECD (2001) Knowledge and skills for life:, Appendix B1, Table 2.1a, p.246 Level 4 Level 5 Korea has a high average performance but few at the highest level. US is only average, on average, but has relatively high proportion at highest level. Korea has less than 6% below Level 2. Mexico has almost 45% below Level 2 – and less than 60% of 15-year-olds in school.

25. OECD World Forum “Statistics, Knowledge and Policy”, Palermo, 10-13 November 2004 25 Comparisons of equity in outcomes.

26. OECD World Forum “Statistics, Knowledge and Policy”, Palermo, 10-13 November 2004 26 Social background vs reading performance Social Advantage PISA Index of social background Each 20,000 students in the OECD area are represented by one dot in this diagram High Student performance in reading Low Source: OECD (2001) Knowledge and skills for life, Appendix B1, Table 8.1, p.308 There are disadvantaged students who do well and advantaged ones who do poorly BUT there is a strong tendency for social advantage to be associated with better performance

27. OECD World Forum “Statistics, Knowledge and Policy”, Palermo, 10-13 November 2004 27 Social background and reading performance Source: OECD (2001) Knowledge and skills for life, Appendix B1, Table 8.1, p.308 Social Advantage PISA Index of social background Low Student performance in reading High This gap is in the order of 3 years of schooling. Countries differ markedly in the equity of their outcomes.

28. OECD World Forum “Statistics, Knowledge and Policy”, Palermo, 10-13 November 2004 28 This kind of analysis is possible only with individual data.

29. OECD World Forum “Statistics, Knowledge and Policy”, Palermo, 10-13 November 2004 29 Reading mean vs social background Low quality High equity Source: OECD (2001) Knowledge and skills for life, Appendix B1, Table 2.3a, p.253 Low quality Low equity High quality High equity High quality Low equity

30. OECD World Forum “Statistics, Knowledge and Policy”, Palermo, 10-13 November 2004 30 Germany commissioned a multi-lateral study.

31. OECD World Forum “Statistics, Knowledge and Policy”, Palermo, 10-13 November 2004 31 Germany’s selection of countries to study Low quality High equity Source: OECD (2001) Knowledge and skills for life, Appendix B1, Table 2.3a, p.253 Low quality Low equity High quality High equity High quality Low equity

32. OECD World Forum “Statistics, Knowledge and Policy”, Palermo, 10-13 November 2004 32 Conclusions from German study Publication –What makes school systems perform? Seeing school systems through the prism of PISA Conclusions –building a culture of achievement social and cultural disparities are central to innovation strategies quality and equity are addressed at the same time –integrated not differentiated system with good quality support to students, teachers, schools –shifting responsibility downward from central authority BUT with system monitoring of performance AND, PROBABLY, publication of results at the school level

33. OECD World Forum “Statistics, Knowledge and Policy”, Palermo, 10-13 November 2004 33 Mean literacy by expenditure per student Countries below the line achieve less than could be expected, given how much they spend. Countries above the line achieve more than could be expected, given how much they spend. Source: OECD (2001) Knowledge and skills for life, Appendix B1, Table 2.3a, p.253 Denmark is among the high spenders but is only average in achievement.

34. OECD World Forum “Statistics, Knowledge and Policy”, Palermo, 10-13 November 2004 34 Denmark commissioned OECD to undertake a comparative policy review of Denmark, Finland, England and Alberta.

35. OECD World Forum “Statistics, Knowledge and Policy”, Palermo, 10-13 November 2004 35 Conclusions from review of Denmark Publication –Quality and equity of schooling outcomes in Denmark Recommendations (35 in all) –Learning standards, evaluation of students, schools clarify and raise expectations of student learning establish school improvement teams develop strategy for estimating value added by schools –Roles and competencies of school leaders mandatory courses before appointment mentoring during full first year –Professional development of teachers link initial and in-service education and research

36. OECD World Forum “Statistics, Knowledge and Policy”, Palermo, 10-13 November 2004 36 Thank you.