Comments on Schleicher and Stanat AERA Session Impact of the Programme for International Student Assessment (PISA) on Educational Quality Daniel Koretz Harvard Graduate School of Education Annual Meeting of the American Educational Research Association Chicago April 13, 2007

2 The impact of PISA  Provides a rich database for exploring patterns of performance and its correlates  Has sparked a wide range of careful and informative analyses  Has contributed to an international wave of education reform

3 Three cautions  PISA is one test  Beware of unwarranted causal inference  Omitted variables: the special case of ILSA  Monitor impact; don’t trust scores alone

4 Inconsistencies across assessments Source: Excerpted from Grønmo & Olsen (2006)

5 Decomposition of variance, TIMSS 1995 Middle-school mathematics Source: Koretz, D., McCaffrey, D., and Sullivan, T. (2001)

6 Explaining differences between the assessments  Differences in results can stem from:  Intended differences in content and format  Unintended differences in attributes measured  Sampling: age of students, age vs. grade sampling, school vs. classroom sampling; differences in exclusions and participation rates  Empirical reasons for specific disparities are not always clear—no empirical linking study  Unless there is a strong reason to prefer one test, place greatest weight on results that replicate

7 Avoiding unwarranted causal inferences  Inadequacy of intra-national survey databases for causal inference widely acknowledged  More severe for explanation of international differences; e.g.:  Cultural and other unmeasured social differences among countries  Variation in the meaning of proxy measures across countries

8 Examples of appropriate language on causality: Stanat and Christensen (2003) The report attempts to identify factors that might contribute to between country differences… Although it is not possible to estimate the effects of these factors on immigrant students’ school success using the PISA data, the analyses presented in the report provide a description…

9 Responding to omitted variables in PISA  Avoid strong causal inference  The residual is just the residual  View analyses as generating tentative conclusions and hypotheses for further testing  Unfortunately, explanations of international differences are riskier than international replications of within- country relationships

10 Omitted variables and the problem of immigration  Why are PISA background variables adequate controls for ethnic German immigrants but not for Turkish?  Thought experiments from the U.S. experience:  1990s immigrants from HK and El Salvador  Italian and Jewish immigrants,  One possible explanation is culture, which is very hard to measure in a comparative survey  An illuminating example: Caplan, Nathan, et al. (1991). Children of the Boat People: A Study of Educational Success. Ann Arbor, MI: The University of Michigan Press.

11 Monitoring impact  PISA is sparking a remarkable range of reform efforts throughout OECD  More focus on monitoring of outcomes and accountability  Widespread interest in additional ‘PISA-like’ measures  How will we ascertain their effects  U.S. experience: impact is likely to be mixed, and scores may become seriously inflated

12 Advice from this side of the pond  Put monitoring of PISA-related reforms in place at the outset  Monitor effects on educational practice  Validate gains: do not trust increases in scores on tests used for accountability  Put auditing measures in place  Guard against corruption of PISA itself by the proliferation of PISA-like assessments with consequences