1. Obtaining International Benchmarks for States Through Statistical Linking: Presentation at the Institute of Education Sciences (IES) National Center for Education Statistics (NCES) Gary W. Phillips Chief Scientist American Institutes for Research May 30, 2008 This paper is intended to promote the exchange of ideas among researchers and policy makers. The views expressed in it are part of ongoing research and analysis and do not necessarily reflect the position of the National Center for Education Statistics, the Institute of Education Sciences, or the U.S. Department of Education.

2. Dr. Gary W. Phillips American Institutes for Research 2 Two ways of obtaining International Benchmarks for States Hard Way Each State is administered PISA, TIMSS or PIRLS, and each country is administered PISA, TIMSS or PIRLS. Very Expensive and impractical. Easy Way (statistical linking) Each State Is administered NAEP, and each country is administered TIMSS or PIRLS. Inexpensive and practical.

3. Dr. Gary W. Phillips American Institutes for Research 3 PISA Cannot be Statistically Linked to NAEP PISA is an age-based assessment (with grade 10 as the modal grade), NAEP is grade-based (grade 8) so PISA results cannot be statistically linked to NAEP. Even if NAEP conducted a special study to make it possible to link NAEP and PISA (e.g., an age-based special assessment of NAEP with a modal grade of 10), State-NAEP still cannot be compared to PISA because State- NAEP is still in the 8 th grade.

4. Dr. Gary W. Phillips American Institutes for Research 4 TIMSS was Purposefully Designed to be Statistically Linkable to NAEP Same grades as NAEP Similar content standards to NAEP Similar test design Matrix sampling of cognitive test items Policy/research related Background questions Similar nationally representative sampling Similar scaling (item-response theory) Similar analysis methods (plausible values)

5. Dr. Gary W. Phillips American Institutes for Research 5 What is Statistical Linking? If you link test X to Y, this means We are expressing the scores on one test (X) in terms of the metric of another test (Y) For example, we are expressing the scores on NAEP (X) in terms of the metric of TIMSS (Y) Four types of linking: equating, calibration, projection, moderation.

6. Dr. Gary W. Phillips American Institutes for Research 6 Assumptions in Linking Statistically linking test X to test Y EquatingCalibrationProjectionModeration High correlation between true scoresx1x1 x1x1 x Same Contentxx Equal Reliabilityx 1 assumes a perfect correlation between true scores

7. Dr. Gary W. Phillips American Institutes for Research 7 Why is it Important to Statistically Link NAEP to TIMSS or PIRLS? Allows policy makers to see how the U.S. (as a whole), the states and school districts stack up against the rest of the world. Provides a common metric that is familiar to US policy makers. It’s like converting world currencies to dollars. Seeing the results of TIMSS or PIRLS through the lens of NAEP Achievement Levels provides a familiar benchmark to interpret international educational performance

8. Dr. Gary W. Phillips American Institutes for Research 8 What are the Ideal Data Requirements for Statistically Linking NAEP to International Assessments? NAEP and the international assessment must be administered within the United States to groups of students which are: Randomly equivalent Nationally representative In the same grades In the same year NAEP and the international assessment must cover similar (but not identical) content

9. Dr. Gary W. Phillips American Institutes for Research 9 Possible Statistical Linkages With NAEP During This Decade

10. Dr. Gary W. Phillips American Institutes for Research 10 Using NAEP linked to TIMSS as an International Benchmark in Mathematics (Phillips, “Chance Favors the Prepared Mind”, AIR, 2007)

11. Dr. Gary W. Phillips American Institutes for Research 11 Using NAEP linked to TIMSS as an International Benchmark in Science (Phillips, “Chance Favors the Prepared Mind”, AIR, 2007)

12. Dr. Gary W. Phillips American Institutes for Research 12 One potential way of Obtaining State Estimates for PISA is Through Small Area Estimation If a survey has been carried out for the nation as a whole, the sample size may be too small for each state to generate accurate state estimates from the collected data. To deal with this problem, it may be possible to supplement the survey data with auxiliary data (such as the CCD) and use regression modeling techniques in order to obtain state estimates.

13. Dr. Gary W. Phillips American Institutes for Research 13 Statistical Linking Versus Small Area Estimation Statistical Linking - Two assessments (e.g., NAEP and TIMSS), measuring similar content, are administered to randomly equivalent national samples and the scales are statistically linked (sort of like converting Celsius to Fahrenheit). Small Area Estimation – One assessment (e.g., PISA) is administered to a national sample and auxiliary data (such as the CCD) is used with regression modeling techniques to obtain state estimates.