1. International Assessments: opportunities and challenges for educational policy in Latin America & the Caribbean Panel Presentation: Gilbert A. Valverde, Ph.D. University at Albany, State University of New York & PREAL Working Group on Evaluation and Standards gvalverde@albany.edu The Right to a Quality Education for All in Latin America and the Caribbean UNESCO – UNICEF Joint Seminar Cartagena de Indias, Colombia 21 de Setiembre, 2011

2. Why is interest in international assessments growing?  What explains the growing interest in large scale international tests on the part of countries such as those in Latin America and the Caribbean?  Are such countries..  Subjects?: ▪Forced to participate by international organizations or other forces?  Agents?: ▪Motivated by domestic agendas  ¿?

3. A hypothesis in development  It is possible that the explanation lies in features of an international political economy.  There are indication that large-scale international testing regimes satisfy demands similar to those satisfied by other types of international regimes.  They reduce transaction costs, ▪Associated with participation in international markets.  They alleviate problems associated with information asymmetries. ▪And there are a number of asymmetries involved!  They limit the uncertainties that countries face in trying to evaluate the policies (and their outcomes) followed in other countries.

4. Key differences between large-scale international tests and national evaluation systems in the region National Systems  Technical weaknesses. In the areas of psychometrics, statistics and sampling. In test R&D and validation. In year-to-year equivalence of scales.  Political weaknesses Tepid political commitment. Lack of technical documentation open to public scrutiny. Frequent not publishing of results. Results are not comparable to those of important peer nations, partners or aspirational peers. International tests  Led by psychometricians, statisticians and sampling statisticians using state-of-the-art techniques.  Rigorous adherence to international standards in test validation.  Internationally refereed sampling and instrument administration.  Test equating for year-to-year comparability.  Public availability of technical and procedural documentation.  (Publication of results)  Possibility of valid comparison with results in other countries.

5. Evaluating two aspects of PISA:  How well does it measure change in achievement levels from 2000 to 2009?  How do Latin American students behave when attempting PISA items in terms of their tendency to:  Not Answer  Answer correctly.  Answer incorrectly.

6. The Data  Taken from the PISA 2009 international report – print version.  Focusing on the 46 items common to the 2000 and 2009 tests (in reading and mathematics)

7. First: considering the evidence on trends in Reading  On the following slide, a graph:  The solid diagonal line represents the assumption that the scales for 2000 and 2009 are the same. ▪PISA contractors have worked hard to insure this!  If the assumption of perfect stability in the scale is correct. ▪Change (improvement, stability or deterioration in the levels of achievement) is represented by the vertical distance with respect to the diagonal.

8. Notable improvements in Peru and Chile (Brazil) No change in México Argentina decreases

9. … continuing..  The dotted line represents a regression of the 2009 scores against those of 2000.  This represents, in effect, a re-scaling of the two tests.  What effect does this have on our inferences about improvement, stability, o deterioration in achievement levels.

10. Improvement in countries like Peru seems less. Chile improves more than most, but improvement is less than in the prior model. Brazil shows no change Mexico has decreased The decrease in Argentina seems greater.

11. Implications  This and other evidence suggest that in PISA (and other evidence suggests something similar in TIMSS) the equating is centered around the median of the distribution of average student  The greater the distance from the median, the greater the likely equating error.  The error seems larger in the lower part of the distribution (where Latin American countries are) than in the higher part.

12. Mayor estabilidad en la equiparación en el centro de la distribución Inestabilidad en las “colas” Mayor inestabilidad en la parte inferior de la distribución. Los países de América Latina están en la parte posiblemente mas inestable de la distribución.

13. Which is the truth?  This simulation does not demonstrate that either the dotted or the solid lines are the “truth”.  It suggests that there are reasonable grounds to doubt the psychometric calibration of the test – especially at the tails of the distribution..  The rankings seem stable.  What we need to reevaluate are the inferences about trends over time in Latin America.  Perhaps the inferences about improvement in Chile and deterioration in Argentina are the most robust in comparison to the inferences that have been made about the rest of the region.  We need to investigate the magnitude and direction of change over time more carefully.

14. Second: examining student behavior regarding PISA items from 2000 to 2009 Omits (not attempted) incorrect correct

15. A mathematics item (M034Q0T) In various countries, from 2000 to 2009 The proportion of students reluctant to attempt items decreased. The proportion of students answering correctly increased. In various countries, from 2000 to 2009 The proportion of students reluctant to attempt items decreased. The proportion of students answering correctly increased. In Argentina and Brazil The proportion of students willing to attempt this item went up The tendency to answer incorrectly also went up. In Argentina and Brazil The proportion of students willing to attempt this item went up The tendency to answer incorrectly also went up.

16. A reading item (R104Q02)

17. We must study Latin America more closely!  It appears to be an outlier in terms of the increase in proportions of students unwilling to attempt items.  This does not appear to be related to the difficulty of items.  Regional patterns seem different from the rest of the world.  Maybe some students have decided not to participate in PISA!  What implications does this perhaps exceptional reluctance to attempt items on the part of the region’s students, have on our inferences regarding their levels of achievement?