1. Design of the 2011 NAEP- TIMSS Linking Study Andrew Kolstad Senior Technical Advisor to the Assessment Division National Center for Education Statistics

2. For states and school districts, to project TIMSS mathematics and science scores from performance on NAEP (without the cost of participating in TIMSS) 2 Goals of the study in 2011 8 th Grade Mathematics (and Science) NAEP 8 th Grade Mathematics and Science TIMSS

3. Previous NAEP / TIMSS linkings - 1 Johnson’s statistical linkage to 1996 state NAEP No students took both NAEP and TIMSS assessments TIMSS-equivalent mathematics and science scores predicted from NAEP state scores using “statistical moderation” (matching score distributions) Evaluated with TIMSS in Minnesota and Colorado 1995 o The actual grade 8 means for Minnesota fell within the confidence intervals o The actual grade 4 means for both Colorado and Minnesota were higher than the predicted means 3

4. Previous NAEP / TIMSS linkings - 2 Johnson and Cohen’s projection linkage to 2000 grade 8 state NAEP mathematics and science A linking sample of students was assessed by NAEP in the winter of 2000 and again by TIMSS in the spring of 2000 TIMSS-equivalent mathematics and science scores predicted from NAEP state scores using “statistical projection” (estimating the average relationship between a student’s score on NAEP and on TIMSS and applying that relationship to state NAEP scores) Evaluated with grade 8 TIMSS in twelve states in 1999 o The performance differed: the actual grade 8 means generally fell above the confidence intervals for the predicted means No TIMSS score projections were published 4

5. Previous NAEP / TIMSS linkings - 3 Phillip’s statistical moderation linkage in 2007 International grade 8 TIMSS-equivalent scores of NAEP achievement level cut scores projected using “statistical moderation” (no student took both assessments) Based on national distributions of NAEP 2000 mathematics and science scores matched to U.S. score distributions on o TIMSS 1999 assessment o TIMSS 2003 assessment Result: Estimated percentage of basic, proficient, and advanced students in each country that participated in the 1999 TIMSS and 2003 TIMSS studies 5

6. Improving on previous linking designs How the current linking study differs 1.Both NAEP and TIMSS in 2011, the same year this time 2.Two linking samples of students; no student assessed in both winter and spring 3.Two booklet designs; one looks like NAEP in the winter and the other looks like TIMSS in the spring 4.Projected TIMSS mathematics and science scores predicted from NAEP state scores using “statistical projection” o (estimating the average relationship between a student’s score on NAEP and on TIMSS and applying that relationship to state NAEP scores) 5.Evaluated with grade 8 TIMSS in eight states in 2011 6

7. Components in linking NAEP & TIMSS 1.NAEP state-level science and mathematics data needed as a basis for projecting TIMSS mathematics and science scores ○ For the first time in NAEP history, all 50 states volunteered to participate in a voluntary state NAEP 2.TIMSS state-level grade 8 data needed to check the validity of the linking function 3.Linking samples of students will provide responses to items from both assessments; needed to estimate relationships ○ No students will take NAEP in the winter and TIMSS in the spring, to avoid the decline in scores found in the previous study 7

8. Booklet designs for NAEP & TIMSS 8 Main study booklets NAEP first blocksecond block BQ1BQ1 BQ2BQ2 TIMSS first block second blockBrkthird block fourth block Br kBQ Assessment time5101520253035404550556065707580859095 10 0… 130 Lege nd NAEP blocks TIMSS blocks NAEP booklets contain either science or mathematics, but not both TIMSS blocks contain both science and mathematics blocks

9. Math & Science Each of the 14 booklets includes two blocks of mathematics and two blocks of science items 1M01M02S01S02 2 S03M02M03 3 M04S03S04 4 S05M04M05 5 M06S05S06 6 S07M06M07 7 M08S07S08 8 S09M08M09 9 M10S09S10 10S10S11M10M11 11M11M12S11S12 12S12S13M12M13 13M13M14S13S14 14S14S01M14M01 Each block appears in two booklets TIMSS assessment booklets 9

10. Linking study approach 1.Administer braided booklets in both testing windows o NAEP in Winter (January to March, 2011) o TIMSS in Spring (April to June, 2011) 2.Select national samples independent of the main NAEP and TIMSS samples o 10,000 students in NAEP in Winter (half science, half math) o 7,500 students in TIMSS in Spring (half science, half math) 3.Collect the data using NAEP field staff in winter and TIMSS field staff in spring (Westat is the contractor for both) 4.Score the items using the same scoring staff with the same training and QC procedures that NAEP and TIMSS use 10

11. Booklet Designs for NAEP-TIMSS Linking Studies NAEP window first blocksecond block BQ 1 BQ 2 first block second block BQ 1 BQ 2 TIMSS window first block second block Br kthird block fourth blockBQ first block second block Br kthird block fourth blockBQ Assessment time5101520253035404550556065707580859095100…115 Lege nd NAEP blocks TIMSS blocks Booklet designs for braided study 11 NAEP-window booklets contain either science or mathematics blocks, but not both TIMSS-window booklets contain both science and mathematics TIMSS blocks TIMSS-window booklets contain three blocks of science or math, plus one block of NAEP

12. Example booklet configurations in the TIMSS window TIMSS M01NAEP M01TIMSS S13TIMSS M03 TIMSS S02NAEP S01TIMSS S05TIMSS M03 Mathematics Science TIMSS Math blocks NAEP Math block TIMSS Science block TIMSS Science blocks TIMSS Math block NAEP Science block Block composition of braided booklets 12

13. TIMSS at the state level Planned as a validity check on the NAEP projection methodology Administered as part of regular, national TIMSS Funded by NCES, not by states 29 states volunteered; ○ 8 states will participate ○ MA, MN, AL, CA, CT, IN, CO, & NC 13

14. NAEP at the state level A state-level grade 8 mathematics assessment was already planned All states have volunteered to participate in the new 2011 NAEP science assessment Science will not be a regular NAEP state-level assessment with full reporting (no urban districts) The braided linking samples are national only, with a very small sample in any given state 14

15. Timeline for NAEP-TIMSS linkingTimeline for NAEP-TIMSS linking Release of the national and state TIMSS results is scheduled for December 2012 Linking results will be reported a few months after the main TIMSS release 15

16. Benefits of the proposed approach Avoids interference with the conduct of regular operational assessments Provides a NAEP-TIMSS link through common items taken by the same student Measures the amount of growth during the period from winter to spring Provides states with international benchmarks in science and mathematics at grade 8, without the cost of conducting TIMSS at the state level 16