1. Comparability of Assessment Results in the Era of Flexibility
Jessica Baghian, Louisiana Department of Education
Brian Gong, Center for Assessment
Jeffrey Nellhaus, Parcc Inc.
CCSSO National Conference on Student Assessment
Austin, Texas
June 28, 2017

2. Topics for this Session
Jeff Nellhaus, Parcc Inc.
How comparable are State and NAEP standards for proficiency?
What purposes are served by comparability?
Jessica Baghian, Louisiana Department of Education
How has Louisiana achieved comparability to other states?
Why is comparability to other states important to Louisiana?
Brian Gong, Center for Assessment
What are the challenges for achieving comparability?
What approaches can be taken to address the challenges?

3. How Comparable are State and NAEP Standards for Proficient Performance?
Beginning in 2005, NCES began to report states’ cut scores for proficiency in terms of their NAEP scale score equivalents
For example, if a state reported 60% of its students performing Proficient or above on its grade 8 math assessment, NCES determined what the cut score for Proficient on the NAEP scale would have to be for 60% of the students in the state to perform Proficient or above on the NAEP grade 8 math test
The NCES report helped answer the question: “Is State X’s standard for proficiency comparable to NAEP’s standard for proficiency?”

4. NAEP Scale Score Equivalent of State Cut Scores for Proficient Performance: 2009, Grade 8 Mathematics
299
300
262
229
*See slide 12 for notes and sources.

5. The Good News …
Between 2009 and 2015, states’ cut scores for proficiency, in terms of their NAEP scale score equivalents, became
Closer to the NAEP cut score for proficiency
Higher on average
Closer to each other
While most states’ cut scores for proficiency in 2015 remained in NAEP’s Basic Level, between 2009 and 2015, the number of states with cut scores in NAEP’s
Below Basic Level decreased
Proficient Level increased

6. Change in States’ NAEP Equivalent Cut Scores for Proficiency 2009–2015
*See slide 12 for notes and sources.

7. NAEP Scale Score Equivalent of State Cut Scores for Proficient Performance: 2009, Grade 8 Mathematics
299
300
262
229
*See slide 12 for notes and sources.

8. NAEP Scale Score Equivalent of State Cut Scores for Proficient Performance: 2013, Grade 8 Mathematics
*See slide 12 for notes and sources.

9. Some Reasons why States’ Performance Standards for Proficiency Have Become More NAEP-Like
Waivers provided by USED for setting improvement targets for school and district accountability
Public reporting of states’ NAEP equivalent cut scores for proficiency exposed the variability in states’ standards
New generation tests based on college- and career-ready content and performance standards
The use of NAEP and other external benchmarks of college- and career- readiness in standard-setting for new generation tests

10. Use of NAEP Results in Standard-Setting for PARCC
The PARCC Assessment RFP in 2013 required that its standard-setting process be informed by established benchmarks for proficiency and college- and career-readiness
“The offeror shall describe a set of benchmarks to inform standard setting, including the percentage of students at or above
proficient on the most recent NAEP assessments
the college-readiness benchmarks on ACT and SAT
relevant benchmarks on international assessments
the college- and career-ready benchmark on SBAC assessments”

11. What Purposes are Served by Comparability ?
Credibility
Comparability to build & maintain public support
Comparability to other states, NAEP, other tests – SAT, ACT, TIMSS
Accountability
Comparability for making high stakes decisions
Comparability across forms – within and across years, and across paper- and computer-based test forms
Trend
Comparability to report change over time
Comparability across forms, across years, and to former testing program
Research/Best Practice
Comparability for research, identify best practices
NAEP plays this role, but data is needed at school and district level and more frequently
Reason for Consortia – common measure

12. Notes and Sources Notes Sources
Figure on slide 4: In Nebraska, each district develops local assessments to report on standards. Therefore, the state was not included in the analyses. California was not included because the state does not test general mathematics.
Figure on slide 6: Nebraska was not included in the 2009 analysis because it did not offer a statewide assessment to report on standards.
Figure on slide 8: California and Virginia were not included because the states do not assess general mathematics in grade 8.
Sources
Phillips, G. (2016). National Benchmarks for State Achievement Standards. American Institutes for Research. Washington, DC
U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2013 Mathematics Assessments.
U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2009, 2011, 2013 Mathematics Assessments.
U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2009 Mathematics Assessments. U.S. Department of Education, Office of Planning, Evaluation and Policy Development, EDFacts SY 2008–09, Washington, DC, The National Longitudinal School-Level State Assessment Score Database (NLSLSASD) 2010.

13. Contact Information
Jeffrey Nellhaus