Are International Comparisons Useful or Simply Interesting? SICI Conference – Greenwich, London 7 June 2012 Linda Sturman Research Director, Centre for International Comparisons, NFER and Sigrid Boyd Head of Eurydice Unit for England, Wales and Northern Ireland, NFER

International comparisons What? Large-scale surveys of achievement Knowledge and skills Contextual data Why? Review and compare Improve outcomes and/or educational systems Who? OECD IEA Policy makers

Some examples 3 PISA TIMSS PIRLS ICCS ESLC PIAAC Curriculum-based Beyond the curriculum Students (primary and secondary schools) Adults Further information about these surveys is available on the NFER website:

More international comparisons What and who? Comparisons of other aspects of education systems (including QA) Desk reviews or fieldwork Lead agencies and country representatives

Some recent examples 5 NFER study of Computing in Schools for Royal Society NFER study of Maths take-up post-16 Eurydice’s Key Data on Education 2012 Eurydice’s Teaching Reading 2011 Eurypedia More information: ucation/policy/computing -in-schools ucation/policy/computing -in-schools ation.org/uk-outlier- upper-secondary-maths- education /education/eurydice/inde x_en.php

International survey ‘industry' 6 Large-scale Complex Time consuming Costly Breadth International reports National reports Released datasets Technical guidance and training Governments, academics and researchers

Other international comparisons 7 Smaller-scale? Less complex? Less expensive? Depth? Academics and researchers Information gatherers

Who’s interested? 8 Stakeholders, including governments Academics Educators Inspectorate Media/other commentators Why?

What’s interesting? 9 Overview of systemic similarities and differences Country’s level of performance International benchmarking Rank order Associations between context and achievement What seems to work

10 Overview of systemic similarities and differences Country’s level of performance and benchmarking Rank order Associations between context and achievement What seems to work Descriptive Snapshot/ exhaustive Context and change Causality Trends hide variation Challenges?

11 Overview of systemic similarities and differences Country’s level of performance and benchmarking Rank order Associations between context and achievement What seems to work Curriculum (Un)familiarity Translation and adaptation Validity Reliability Design differences Differential outcomes Challenges?

12 Overview of systemic similarities and differences Country’s level of performance and benchmarking Rank order Associations between context and achievement What seems to work Changes in participation Absolute vs relative achievement Significance Challenges?

13 Overview of systemic similarities and differences Country’s level of performance and benchmarking Rank order Associations between context and achievement What seems to work Significance Correlation/hidden effects Causality Challenges?

14 Overview of systemic similarities and differences Country’s level of performance and benchmarking Rank order Associations between context and achievement What seems to work Descriptive/ analytical Sufficient evidence? Causality Challenges?

15 Overview of systemic similarities and differences Country’s level of performance and benchmarking Rank order Associations between context and achievement What seems to work Overcoming the challenges Shared knowledge New ideas For high-performers? Satisfaction/pride Interest/visitors Media attention? For lower performers? Shock/reinforcement Media attention? Impetus for change? What follows?

Simply interesting or also useful? 16 Value for money? Implications of findings Highlight factors that might improve systems Consider different ways of doing things What works? What looks like it might work?

Intrinsically useful? 17 Important to learn from each other Depth, breadth, interpretation Dissemination vs impact Need critical evaluation Conclude with care Take account of limitations ‘Policy tourism’ (transfer and context) Cost of policy mistakes Explore further before acting? Further analysis? End point or starting point?

Useful countries for comparison… 18 High achievers ‘Countries like us’ Countries implementing change Dependent on the question…

And so, full circle… 19 Are international comparisons useful or simply interesting?

Finding out more about international comparisons and the NFER… 20 Webpages NFER: NFER International Comparisons: NFER International Evidence: comparisons/eurydice-and-inca.cfmhttp:// comparisons/eurydice-and-inca.cfm Linda Sturman: Sigrid Boyd: