1. Describing the Academic Content of International Mathematics and Science Assessments & Their Relationship To State Academic Content Standards & Assessments Presented at the 2009 IES Conference, June 2009, Washington, DC John Smithson, Ph.D., Wisconsin Center for Education Research

2. Using a Comprehensive Descriptive Language and a Systematic Set of Procedures to Describe Academic Content A common, neutral language that can be applied to describe any curriculum- based materials or practices. A systematic procedure, utilizing teams of content experts to employ this common, neutral language as a tool for describing the content embedded in the selected materials. Content Analysis

3. The Language (Taxonomy) A two-dimensional framework for describing: 1)WHAT STUDENTS SHOULD KNOW (Represented by a detailed topic list organized into content areas.) & 2) BE ABLE TO DO (Represented by five categories of student performance expectations.)

4. The Process (Content Analysis) Coding Teams of 3-5 Content Experts Independent Coding by each Analyst w/ Group Discussion Goal: Describe the Assessed Content Data Quality: Generalizability not Inter-rater Reliability

5. BFEDC 101 105 104 103 102 104 E 106 The Description (Content Analysis)

6. From Coding Form Data Collection (Content Analysis)

7. The Results (Content Description) The 3 rd Dimension

8. Contour Interval = 1%Tile Interval = 2.5% K-12 Science ContentPISA 2006 Science Item Pool

9. Alignment Analysis Identifying similarities and differences between two content descriptions… Alignment Indices: A summary measure, or set of measures, designed to indicate the extent of similarity & difference between two content descriptions. … in order to answer the question: To what extent (and how) is the PISA similar or different to U.S. curriculum standards and/or other key assessments.

10. PISA 2006 2007 TIMSS Gr.8 I II III IV V I Memorize II Procedures III Communicate IV Analyze V Apply Concepts CG Align: 0.50 FG Align: 0.37 K-12 Science Content

11. Gr. 8 TIMSS Gr. 8 NAEP Mathematics Content: All Content Areas PISA 2006 0.34 0.62 0.380.59

12. TIMSS Mathematics Content Alignment

13. TIMSS Science Content Alignment

14. PISA Alignment

15. Alignment Levels to Key Targets Across All Documents PISA Math Gr.8 TIMSS Math Gr.4 TIMSS Math Total # Documents 267 101 PISA Science Gr.8 TIMSS Science Gr.4 TIMSS Science Total # Documents 195 77

16. www.SEConline.org