USING THE OECD TEST FOR SCHOOLS (BASED ON PISA) FOR COMPARISON OF AMERICAN STUDENT PERFORMANCE AGAINST INTERNATIONAL BENCHMARKS Frank Rijmen, CTB

Symposium Discussion First there was PISA Then there was the OECD Test for Schools Let’s first look at PISA and how it is different from a summative assessment

Symposium Discussion PISAK-12 Summative Assessments Complex sampling designCensus Booklet test design (item blocks)Limited number of forms Extensive background questionnaires (Attitudes, non cogs) A few demographic variables Focus on group-level inferencesFocus on students Limited number of cognitive items per student Larger number of items per student Measurement error at individual level is accounted for Measurement error at individual level is ignored (but relatively small) when aggregating results at group level Age basedGrade based Not directly linked to a CurriculumAligned with (State) Standards

Symposium Discussion First there was PISA Then there was the OECD Test for Schools OECD Test for Schools adapts PISA to the school context – Preserves most PISA characteristics – Relies on the advanced statistical machinery underlying PISA

Symposium Discussion First there was PISA Then there was the OECD Test for Schools OECD Test for Schools adapts PISA to the school context – Preserves most PISA characteristics – Relies on the advanced statistical machinery underlying PISA – (First there was PISA, but before PISA there was NAEP)

Symposium Discussion First there was PISA Then there was the OECD Test for Schools OECD Test for Schools adapts PISA to the school context – Preserves most PISA characteristics – Relies on the advanced statistical machinery underlying PISA Hao and Joanna presented some aspects of the advanced statistical machinery underlying the assessment – Complex sampling – Balanced incomplete block test designs – Conditioning models – Plausible values (multiple imputations)

Symposium Discussion First there was PISA Then there was the OECD Test for Schools OECD Test for Schools adapts PISA to the school context – Preserves most PISA characteristics – Relies on the advanced statistical machinery underlying PISA Hao and Joanna presented some aspects of the advanced statistical machinery underlying the assessment – Complex sampling – Balanced incomplete block test designs – Conditioning models – Plausible values (multiple imputations) Not many people are interested in the engine, but we all like a reliable, fuel-efficient car

Symposium Discussion First there was PISA Then there was the OECD Test for Schools Noémie Le Donné showed some of the richness of information that is obtained with the OECD Test for Schools – International comparisons – Comparisons to schools with students of a similar socio- economic background – Non-cogs and how they relate to achievement

Symposium Discussion How do schools and school districts use the results of the OECD Test for Schools? Colin Martin showed how the results are used by the Gwinnett County Public Schools Variety of uses – (Inter)national bench marking – Close look at non-cogs E.g., discrepancy between ‘caring’ and helpfulness Improvements of performance through working on – School climate – Student engagement Development of communities of practice with other participating schools

Symposium Discussion Stakeholders use non-cogs in a variety of ways Are these uses supported by empirical findings? Joanna: Yes, they are Using these results may help in developing a holistic approach to learning – Not narrowly focused on Reading, Math, and Science only A mature use of test results