1. 1 Comparing International Assessments to NAEP May 30, 2008 Eugene Owen Senior Advisor, NCES International Activities Program This paper is intended to promote the exchange of ideas among researchers and policy makers. The views expressed in it are part of ongoing research and analysis and do not necessarily reflect the position of the National Center for Education Statistics, the Institute of Education Sciences, or the U.S. Department of Education.

2. 2 Goal of this Presentation Present results from comparison studies to elaborate on international assessments: –How each is related to NAEP and –What each can contribute to the knowledge base on student performance

3. 3 Recent Int’l and Nat’l Assessments 4 th grade8 th grade12 th grade ReadingNAEP 2007 PIRLS 2006 NAEP 2007NAEP 2005 MathematicsNAEP 2007 TIMSS 2007 NAEP 2007 TIMSS 2007 NAEP 2005 ScienceNAEP 2005 TIMSS 2007 NAEP 2005 TIMSS 2007 NAEP 2005 15 year olds PISA 2006* PISA 2006

4. 4 Overall Differences Assessments have subtle but important distinctions with regard to U.S. curricula International assessments provide benchmarks with different groups of countries The students being studied may represent different groups of students Different assessments are designed to measure at different levels of precision and disaggregation

5. 5 Content Differences: General Sources of difference –Framework Organization –Distribution of items across content areas and topics –Distribution of items across cognitive skills/ processes –General consistency with grade-specific content/topics in NAEP framework –Likelihood of being in U.S. curricula or on NAEP

6. 6 Mathematics Frameworks NAEP 2005/7 Content Strands Number Prop./Operations Measurement Geometry Data Analysis and Probability Algebra Mathematical Complexity Low Moderate High TIMSS 2007 Content Domains Number Geometric Shapes, Measures (4)/Geometry (8) Data Display (4)/Data and Chance (8) Algebra (8) Cognitive Domains Knowing Applying Reasoning PISA 2006 Overarching Ideas Quantity Space/shape Change and relationships Uncertainty Competency Clusters Reproductions Connections Reflections Situations/Contexts

7. 7 Notes and comments on TIMSS-NAEP 2007 math comparisons Some differences in distribution across cognitive domains (smaller between NAEP and TIMSS than PISA) PISA most consistent with 8 th grade topics in NAEP Vast majority of TIMSS items are “likely” in U.S. curricula Difficulty in placing some TIMSS items on NAEP framework reflects “holes” in NAEP framework TIMSS has notable number of pattern problems, and problems requiring drawing symmetrical figures

8. 8 Science Frameworks NAEP 2005/7 Fields of Science Life science Physical science Earth science “Knowing and Doing” Conceptual understanding Scientific investigation Practical reasoning Nature of science Themes TIMSS 2007 Content Domains Life science / biology Physics Chemistry Earth science Cognitive Domains Knowing Applying Reasoning Scientific Inquiry PISA 2006 Knowledge Of natural world: living systems, physical systems, earth/space systems, technology systems About science itself: scientific inquiry and explanations Competencies Identifying scientific issues Explaining phenomena scientifically Drawing conclusions based on evidence Contexts and Attitudes

9. 9 Notes and comments on NAEP-TIMSS science comparisons Vast majority of TIMSS items “likely in U.S. curricula” Difficulty in placing some TIMSS items on NAEP framework reflects “holes” in NAEP framework TIMSS has notable number of “science as context” items

10. 10 Notes and comments on PISA-NAEP science comparisons PISA items are equally consistent with the 8 th and 12 th grade NAEP frameworks (about 40% each) “Rate” as a concept does not appear in the NAEP framework, whereas it is in numerous PISA items The closer PISA comes to its objective of assessing scientific literacy, the harder it is to map items to NAEP –Over half rated as “not likely to be a NAEP item”

11. 11 Thank you