1 Designing High Quality, Affordable Assessment Systems Edward Roeber Michigan State University National Research Council Board on Testing and Assessment April 6-7, 2010

2 Overview Balanced Assessment Systems Horizontally Balanced Vertically Balanced Summative Assessment Designs Purpose of this Study Typical Current Assessment Program High Quality Assessment Program Development and Administration Costs Cost Reduction Strategies Results Recommendations

3 Vertically-Balanced System Summative Component Broad array of types of assessment Interim Benchmark Component Instructionally-relevant, short-cycle summative assessments also using a broad array of assessment types Formative Assessment Component Adequate professional development High-quality pre-service education All three components present, and equally important to the teacher and the student

4 Vertically-Balanced System Portions of this system are out of balance: Interim benchmark assessments not thoughtfully created, acquired, or used Educators don’t understand how to use interim assessments Only conventional types of assessment used Lack of understanding about what formative assessment is and is not (e.g., it is not an item bank) Lack of opportunities for educators to learn how to use formative assessment strategies as they teach Educators don’t learn about classroom and formative assessment in pre-service programs State summative assessments overpower the interim and formative assessments in the minds of educators (and students)

5 Horizontally-Balanced Systems Multiple-choice items are the predominant form of assessment used by states Some states use constructed-response items Fewer still use performance assessments Emphasis is on speed of return of results, not quality or usefulness of the information once it gets there Schools feel compelled to “teach to the test” in some not-so-good ways - some subjects not taught at all and the rest are taught to the extent that they are on the test Only state-assessed skills are focused on in local assessments

6 Horizontally-Balanced Systems This system is out of balance in some important ways More types of items should be used in the system Short- and extended-response items Performance events Performance tasks This broader array of items could have a positive impact on teaching and learning - could be assessments worth teaching to These state assessments could also serve as a model for developing instructionally-sensitive interim assessments The prime question is can states can afford to use this broader array of assessment?

7 Purpose of This Cost Study Determine the cost of a typical state assessment program that primarily uses multiple-choice items Design a high quality summative assessment (HQA) as just described Note - HQA designed for Mathematics and Reading/Writing only. This excludes Science, alternate assessments for students with disabilities and ELPA assessments Determine the cost of such a HQA Determine whether several potential strategies could reduce costs of the HQA significantly Present various options and costs for an interim assessment system similar to the HQA

8 Current Typical Summative Assessment Design

9 Current Typical Interim Assessment Design

10 High Quality Summative Assessment Design

11 High Quality Interim Assessment Design

12 Development and Administration Costs Typical Assessment Program - $20/Student Cost per student for the current typical assessment calculated using the ASG cost model (includes Year 0 development costs) HQA Summative Assessment Program - $55/Student Cost per student for the high quality assessment calculated assuming a single state implementation and no cost reduction strategies (includes Year 0 development costs) Most states cannot afford a nearly tripling of their state assessment costs, so the result will be a limit on innovative assessment types, unless something occurs

13 Cost Reduction Strategies Participation in a consortium Model includes state consortium sizes of 10, 20, and 30 states Use of a state consortium reduces costs by an average of $15 per student Consortium approach represents a significant decrease in assessment cost Uses of technology for online test delivery, distributed human scoring of some of the open-ended items, and automated scoring for certain constructed response items Together, these innovations account for cost savings of about $3 to $4 per student Likely to account for more as efficiencies are developed in programming and using technology for these purposes Two approaches to the use of teacher-moderated scoring. Teacher-moderated scoring can net both substantial cost reductions as well as potential professional development benefits. We used two different models for teacher-moderated scoring

14 Cost Reduction Strategies Two different models for teacher-moderated scoring: Professional development model - no additional teacher compensation beyond that supported by the state or district for normal professional development days Stipend model - assume a $125/day stipend for teachers to score the performance items. Note: teachers were assumed to score all performance items in a distributed scoring model These strategies for using teachers as scorers reduce costs by an additional $10 to $20 per pupil (depending on whether teachers are engaged as part of professional development or are paid) Adopting all cost reduction strategies while paying teachers a $125/day stipend to score all performance tasks results in an assessment cost of $21

15 Overall Cost Reduction Results

16 Impact of Teacher Scoring Time on Costs

17 Cost Study Conclusions The development cost of a new HQA is relatively inexpensive relative to the total cost of the assessment A key factor in determining whether states can adopt and sustain new improved assessments is ongoing administration costs In order to reduce costs, states should participate in an assessment consortium to share the overhead associated with development, administration, and management of assessments Larger consortia are somewhat more cost-effective The majority of cost savings relative to the single state case are seen even in at a 10-state consortium size States should strongly consider being part of a large consortium where certain costs can be shared across many states, such as for item development and project management

18 Cost Study Conclusions Implementing a HQA system with performance items is affordable, with teacher scoring of performance items at a price comparable to today’s assessments, when procured by a consortium of states In order to implement and afford an HQA system that includes a variety of performance items, it is essential to have teachers involved in the scoring process The cost impact of increases in the time to score performance items is very significant The use of online technology (i.e., online assessments) should be encouraged It has the potential to reduce assessment cost and improve quality The procurement of PCs to improve the student- to-PC ratio should be encouraged at all levels of the educational system

19 Recommendations Developing and implementing an HQA will likely cost more than most current state assessments. It can be affordable for states if they look carefully at the design of the summative assessment component finding a balance in the number of CR items, PEs, and PTs used consider various cost-reduction strategies It is recommended that state consortia go about the process of designing a new assessment in a thoughtful manner, then use a comprehensive costing model to analyze and determine the price in advance of any new assessment system they would like to implement State consortia interested in implementing a HQA should make sure they can afford the ongoing administration costs of the assessment before they embark on developing it

20 For More Information Edward Roeber Michigan State University 263 Erickson Hall East Lansing, MI (517) For a copy of the complete paper for this study, go to the Assessment Solutions Group website to download a copy: