1. Scott Marion, Center for Assessment
Technical, policy, and practical considerations for dealing with opt-outs
Scott Marion, Center for Assessment
CCSSO’s National Conference on Student Assessment
Opt Out: Implications for Building Equitable Assessment Systems for All Students
June 22, 2016

2. Scott Marion_Opt-out_NCSA_June 22, 2016
Advance Organizer
Policy
Statutory leniency
Conflation with educator evaluation
Technical
Equating
Accountability calculations
Practical
Over-promising with the summative
Improving utility
Scott Marion_Opt-out_NCSA_June 22, 2016

3. Policy: Statutory Overview
From the National Association of State Boards of Education (2015)
Scott Marion_Opt-out_NCSA_June 22, 2016

4. Policy: Statutory Leniency
In spite of the laws, many states have way more opt- outs than might be predicted from the map (e.g., NY)
Challenge made tougher for state DOE personnel when legislative action allows opt-outs for almost any reason or none at all
Even in states where the state board or legislature has passed laws or regulations permitting opt-outs, many local school districts (e.g., Patchogue, NY; Springfield, OR) have passed resolutions encouraging opt-outs
Puts state DOEs between a rock and hard place when trying to deal with the federal participation rate requirements
Scott Marion_Opt-out_NCSA_June 22, 2016

5. Policy: Conflation with teacher evaluation
2015
New, more rigorous standards
New, more rigorous assessments
New, test-based teacher evaluation systems
What could go wrong?
Scott Marion_Opt-out_NCSA_June 22, 2016

6. Scott Marion_Opt-out_NCSA_June 22, 2016
Technical: Equating
Potential concerns:
Population of opt-outs varies from year-to-year
Population of test participants varies considerably from the year that the base scale was established
Could threaten the validity of the equating results
Potential approach:
Base equating on a sample of students selected to represent the population of students included in the base-year scale
Compare the equating results from the sample described above to equating results using the full set of participants to understand the effect of opt-out
Scott Marion_Opt-out_NCSA_June 22, 2016

7. Technical: Accountability calculations
One potential policy:
Once the participation rate drops below 90%, a statistical test* must be performed to document that the participating students are representative of the full school/district population.
*Based on a chi-square test associated for evaluating differences in proportions
If the school fails this test, no overall determination or other accountability score will be provided
Once a school/district has fewer than 80% of its students participating, no overall determination or other accountability score should be provided.
Scott Marion_Opt-out_NCSA_June 22, 2016

8. Practical: Mirror mirror…
We need to look in the mirror first!
We’ve brought much of this on ourselves!
Over-promising
Over-testing
Under-delivering
Irrelevant for students
Scott Marion_Opt-out_NCSA_June 22, 2016

9. Practical: Improving utility
Let’s not pretend that the summative assessment can do much more than produce an overall score used for accountability and perhaps program evaluation
We need considerably more effort into designing coherent assessments systems that can serve multiple purposes well
Intentional design; not slapping together various categories of tests
Be thoughtful about information flow through the system and how the whole can be greater than the sum of the parts
Scott Marion_Opt-out_NCSA_June 22, 2016

10. Scott Marion_Opt-out_NCSA_June 22, 2016
A look ahead…
It will get better…
ESSA takes the pressure off of the federal push for test-based teacher evaluation
We’re designing better and more interesting assessments
It will get worse…
Opt-out groups are feeling empowered
We’re still focused on the summative too much
Many summative tests are still really long
Need to meet alignment criteria
Desire for subscores
What do you think?
Scott Marion_Opt-out_NCSA_June 22, 2016

11. Scott Marion_Opt-out_NCSA_June 22, 2016
Scott Marion
Scott Marion_Opt-out_NCSA_June 22, 2016