1. A Statistical Linkage Between NAEP and ECLS-K Grade Eight Reading Assessments Enis Dogan Burhan Ogut Young Yee Kim Sharyn Rosenberg NAEP Education Statistics Services Institute American Institutes for Research A Statistical Linkage Between NAEP and ECLS-K Grade Eight Reading Assessments Enis Dogan Burhan Ogut Young Yee Kim Sharyn Rosenberg NAEP Education Statistics Services Institute American Institutes for Research

2. Purpose  In Spring of 2007, about 1300 students took the NAEP and ECLS-K grade eight reading assessments  The purpose of this study is to establish a statistical link between these two assessments using this common sample  We also test the validity of the link by comparing the model projected results to reported NAEP results

3. BACKGROUND

4. Early Childhood Longitudinal Study Kindergarten Cohort (ECLS-K)  Provides longitudinal data on students’ academic achievement in reading, mathematics, and science  Began in the fall of 1998 with a national sample of 21,000 kindergartners  Each student is tested by a series of two-stage, 30-minute adaptive tests, administered face to face, on seven occasions  Item parameters and individual scores are estimated using IRT

5. NAEP 8 th Grade Reading 2007  Provides cross-sectional data on students’ academic achievement in reading  Nationally representative sample of more than 350,000 students participated in the 2007 reading assessment  Each student takes just a portion of the test (BIB Design), consisting of two 25-minute sections or one 50-minute section  Item parameters are estimated using IRT, but results are reported at group level. NAEP does not report individual scores

6. Differences between NAEP and ECLS-K  The two assessments have similar frameworks, but do not share any common items  NAEP uses a nationally representative sample  ECLS-K is representative of children enrolled in first grade in the U.S. in 1999-2000

7. ESTIMATING THE PROJECTION EQUATION

8. Linking through a common sample  In Spring of 2007, about 1300 public school students took NAEP and ECLS-K grade 8 reading assessments. o About ¼ were part of the operational NAEP sample o The rest took the NAEP assessment for the purposes of the common sample linking study

9. Method and challenges  NAEP scores in the form of plausible values were not available for about ¾ of the linking sample since they were not part of the operational NAEP assessment  Using item level data from the entire common sample, MML regression was used (with the AM software) to estimate a projection equation predicting NAEP score distributions from ECLS-K scores  The projection equation was where y is the NAEP score and x is the ECLS-K score

10. The projection equation  The projection equation was estimated using two different weights: o the original ECLS-K sampling weights, and o poststratification weights Parameter estimates predicting NAEP scale scores from ECLS-K scale scores 1 Using original weights: RMSE = 18.84, F (1,168) = 326.47 2 Using poststratification weights: RMSE = 18.60, F (1,168) = 370.64 ParameterOriginal estimate 1 Poststratified estimate 2 Intercept79.32 (10.46)75.86 (9.92) Slope 1.10 (0.06) 1.12 (0.06)

11. Variance of the projection  For the projection equation,  the a, b parameter estimates are denoted and their covariance as  The projected NAEP average reading score is  The variance of the projection at mean is

12. TESTING THE VALIDITY OF PROJECTION EQUATION

13. Comparing reported and projected NAEP mean National average scores and 95% confidence intervals for projected and reported 2007 NAEP reading scores: grade 8, public

14. Scores by gender and race/ethnicity  Left bar: Confidence Interval for the mean based on the projection using the original weights  Right bar: Confidence Interval for the reported NAEP mean NAEP Scale

15. AN APPLICATION OF THE LINK: READING PERFORMANCE AT EARLIER GRADES AND PROFICIENCY IN NAEP

16. What is the relationship between reading performance at first, third and fifth grades and Proficiency in eighth-grade NAEP reading assessment? Early childhood reading performance and proficiency in NAEP

17.  Show an overall understanding of the text, including inferential as well as literal information  Extend the ideas in the text by making clear inferences from it, by drawing conclusions, and by making connections to their own experiences— including other reading experiences  Identify some of the devices authors use in composing text Proficiency in eighth-grade NAEP reading assessment

18. Level 1 : Letter recognition: identifying upper- and lower-case letters by name Level 2 : Beginning sounds Level 3 : Ending sounds Level 4 : Sight words Level 5 : Comprehension of words in context Level 6 : Literal inference Level 7 : Extrapolation Level 8 : Evaluation Level 9 : Evaluating nonfiction Level 10: Evaluating complex syntax: evaluating complex syntax and understanding high-level nuanced vocabulary in biographical text. ECLS-K Reading performance levels

19. Percentage of students at Levels 5 through 9: Grade 5 Reading performance at grade 5 and Proficiency in 8 th grade NAEP

20. Percentage of students at Levels 5 through 9: Grade 5 Reading performance at grade 5 and Proficiency in 8 th grade NAEP 33 % 48 % 72 % Evaluation: demonstrating understanding of author’s craft … 13 %

21. Percentage of students at Levels 4 through 8: Grade 3 Reading performance at grade 3 and Proficiency in 8 th grade NAEP

22. Percentage of students at Levels 4 through 8: Grade 3 Reading performance at grade 3 and Proficiency in 8 th grade NAEP 11 % 25 % 46 %62 % Extrapolation: identifying clues used to make inferences, …

23. Percentage of students at Levels 3 through 7: Grade 1 Reading performance at grade 1 and Proficiency in 8 th grade NAEP

24. 13 % 34 % 49 % 73 % 85 % Percentage of students at Levels 3 through 7: Grade 1 Comprehension of words in context: reading words in context

25. Summary of findings  Using a common sample, a linking equation was estimated  Using the equation, projected NAEP scores were computed for ECLS-K 8 th graders  Mean projected NAEP scores for the nation and the gender and racial/ethnic groups were close to reported NAEP results  Mean projected NAEP scores were used to examine the relationship between proficiency in grade eight NAEP reading and earlier reading performance

26. Limitations  Composition of the ECLS-K sample and the population it represents  Projection results for Hispanic students Future analyses  Further validation of the link  Modeling growth in reading on the NAEP scale Discussion