DISTRICT MANAGEMENT COUNCIL ACADEMIC RETURN ON INVESTMENT (A-ROI)

THE WHY OF A-ROI White Paper: Results-Driven Decision-Making and Academic Return on Investment “K-12 public education performance has been stagnant for years despite increased fiscal investments, both at the state and local level”. “…school districts have ramped up their focus on measuring academic outcomes, most measures do not promote looking at sustained improvement or recognize the need to differentiate programming..” “.. It’s hard to know what works, what works best for different students, how long a program or investment should be in place prior to producing intended outcomes.”

STRATEGIC FRAMEWORK How does A-ROI align with our score card and strategic framework?

THREE CRITICAL COMPONENTS We need: 1.A shift in mindset 2.An analytical tool 3.Align staff to develop and implement a new approach

EVOLUTION OF SCHOOL DISTRICT’S MINDSET Focus on OutcomesFocus on Cost Reduction Focus on Value No Child Left Behind forces states to set achievement standards and implement standardized tests The fiscal crisis strained local tax revenue Rising pressure on school districts to improve outcomes with reduced funding Governments created annual progress benchmarks and report cards. School districts faced a reduction in funds Simultaneous consideration of cost and outcomes to drive sustained improvement. School districts instituted multiple strategies to improve outcomes. Increasing competition for public funds

FOCUS ON OUTCOMES The focus on outcomes has resulted in a vast number of metrics being calculated and tracked across school districts in the country. Many metrics are not actionable and may mostly be compliance or accountability- related.

TYPES OF METRICS REQUESTED BY KEY STAKEHOLDERS State District School Community School Leaders Teachers

CHALLENGES WITH OUTCOME FOCUS The data is used in aggregate form and lacks granularity to make real improvements because it assumes most students have similar needs The data is not tied to instructional approaches and thus can not be replicated or improved The data does not show which teachers are effective with which students The data does not take into account other relevant attributes

FOCUS ON COSTS Different districts spend very different amounts to serve similar students populations. Within the same district, the cost of educating a student may vary based on teaching strategy or program (special ed. vs. Title vs. ADSIS vs. AVID or resource vs. co-teaching vs. inclusion)

CHALLENGES WITH COST APPROACHES Costs estimates typically only include teacher costs and leave out cost of administration, support services, etc. Incomplete Costs are calculated by time or by program and do not reflect whether or not students realize the intended outcome. Not tied to specific outcomes Costs are captured by department or by school, which are not granular enough to show changes in instructional process. Not captured at specific student level

DMC’S VALUE-BASED APPROACH DMC Value-Based Analysis Framework Student Segmentation Educational need Other relevant attributes Student Outcomes Mastery Costs Direct and indirect Duration for attainment

KEY BENEFITS OF A VALUE-BASED APPROACH Key Benefit 1.Focus on value and effectiveness not absolute results 1.Promote innovation and experimentation 1.Compare results across schools, district, and even regions Rationale Allows identification of cost- effective programs or interventions that have the greatest impact on student performance. Frees up resources to invest in other areas and allows the district to test newer strategies. Allows normalization of outcomes and costs across districts and may allow easy identification and replication.

STUDENT SEGMENTATION In most districts, segmentation of students takes place based on NCLB categories or funding streams and only classroom teachers have a student by student look. Issues This does not provide sufficient information for tailored interventions- instead, all students receive the same treatment. All th StudentMath ScoreCurrent Segmentation A59ESL B60F/R C61Spec. Ed. All three students receive math intervention

STUDENT OUTCOMES Potential outcome measures Improvement from level F to G in Fountas & Pinnell in 6 months 80% score on Algebra 1 final Score of 4 on 6 th grade writing rubric- persuasive essay Key Practices Monitors progress toward academic goals Measures outcomes associated with specific student needs Includes both absolute and growth metrics Assesses performance frequently. Good outcome measures are actionable

STUDENT OUTCOMES Students Formative Assessments Desired Results Student specific Individualized need and key attributes Allows for timely and targeted interventions

COSTS Identify Major Steps and Map Process Calculate Direct Costs Calculate Indirect Costs Compute Total Cost per student Complete Cost Analysis Choose an achievement Benchmark Create a map that outlines the major steps for student to reach benchmark Estimate the direct cost of each service Allocate services that indirectly support a student’s learning Add direct and indirect costs to obtain annual cost

KEY TAKE-AWAYS Schools have many measures of outcomes, but few actionable. In order to capture actionable outcome measures, schools need performance data segmented by student need and key attributes. Capture all costs connected with serving student, not just the easy ones. Calculate the cost to achieve an outcome, not just by the budget year.

USING A VALUE-BASED APPROACH 1.Find or create natural experiments to evaluate programs Existing programs: Use student-level data when analyzing programs that already exist and look for natural experiments to compare value. Proposed programs: Ensure that the rollout of new programs have measurable outcomes and meaningful comparisons. 2.Give value-based decision-making a seat at the table 3.Formalize the program approval, modify, or abandonment process Base on data and separate from the budget cycle

A-ROI TIMELINE AugSeptOctNovDecJanFebMar Select target programs Conduct value- based analyses Find meaning Budget Cycle

OUTCOME MEASURES Monitors progress toward academic goals Does not measure process or satisfaction metrics Includes both absolute and growth metrics Assess performance frequency Measures outcomes associated with specific student needs Key Attributes of Effective Outcome Measures

POSSIBLE PROGRAMS FOR A-ROI AVID ADSIS LLI