CONSORTIUM FOR EDUCATIONAL RESEARCH AND EVALUATION – NORTH CAROLINA The NC Race to the Top Evaluation Plan: An Introduction October 10, 2011 Gary T. Henry, Carolina Institute for Public Policy, UNC-CH Julie Marks, Carolina Institute for Public Policy, UNC-CH Trip Stallings, Friday Institute, NCSU Laura Simpson, SERVE Center, UNCG

CONSORTIUM FOR EDUCATIONAL RESEARCH AND EVALUATION – NORTH CAROLINA NC Race to the Top Evaluation NC Race to the Top is designed to be a “game changer ” NC RttT is a coordinated set of policy reforms and innovative activities that are designed collectively to dramatically improve the performances of students, teachers, leaders, and schools Evaluation contributes to NC RttT “changing the game” in two ways: 1. Program Evaluation:  Provide formative information on the implementation of NC RttT initiatives Inform decisions to improve implementation 2. Policy Evaluation:  Assess – from the perspective of students, teachers, leaders, and schools – the improvements that have occurred as a result of NC RttT initiatives collectively and, to the extent possible, the contributions of specific individual initiatives Inform decisions about sustainability and impacts 2

CONSORTIUM FOR EDUCATIONAL RESEARCH AND EVALUATION – NORTH CAROLINA NC RttT Evaluation: Team CERE-NC: SERVE Center, Carolina Institute for Public Policy, and Friday Institute Steering Committee: Gary Henry, Ludy van Broekhuizen, & Glenn Kleiman Principal Investigator: Gary Henry Management Committee: Trip Stallings, Laura Simpson, and Julie Marks Team Leaders: Teacher and Leader Effectiveness - Laura Simpson; Equitable Supply and Distribution of Teachers and Leaders - Trip Stallings; Professional Development - Jeni Corn and Karla Lewis; Turnaround of LEAs and Schools - Charles Thompson; Local-Level Implementation and Spending - Eric Houck; Overall Impact - Gary Henry and Julie Marks Other Leadership Roles: LEA Coordinator - Alexa Edwards 336 State Liaison - Trip Stallings 919 3

CONSORTIUM FOR EDUCATIONAL RESEARCH AND EVALUATION – NORTH CAROLINA Evaluation Overview NC Race to the Top Initiatives: Evaluation Organization 1.Teacher and leader effectiveness Integration of value-added student achievement measures into educator evaluation system 2.Equitable supply and distribution of teachers and leaders Teach for America, NC Teacher Corps, Regional Leadership Academies, Teacher Induction Program, Virtual Public School, Incentives 3.Professional development All professional development activities in support of RttT initiatives, including: PD for standards and assessment, IIS, and data use; and PD delivery capacity-building efforts 4.Turnaround of LEAs and schools Low-achieving LEAs and schools, STEM schools 5.Local-level implementation and spending on RttT Cloud computing, allocation of RttT funds, cost savings 6.Overall impact of RttT on students, teachers, and school leaders 4

CONSORTIUM FOR EDUCATIONAL RESEARCH AND EVALUATION – NORTH CAROLINA 1. Teacher and Leader Effectiveness Purpose of this Evaluation Project To ensure quality, consistency, and fairness of new and ongoing teacher and principal evaluation processes through examination of validity and reliability across multiple observational perspectives To examine educators’ perspectives on new evaluation standards and the effect of these standards on educators’ practices 5

CONSORTIUM FOR EDUCATIONAL RESEARCH AND EVALUATION – NORTH CAROLINA 1. Teacher and Leader Effectiveness Evaluation Questions Have valid and reliable measures of student growth been identified for inclusion in the teacher and administrator/principal evaluation process? Does the revised evaluation process allow for/make meaningful distinctions between teachers’ and administrators’ effective and ineffective performance? How do educators view the implementation/rollout of the evaluation process? Does the new evaluation process change educators’ attitudes? Does it change educators’ practices? Do performance incentives for teachers in low-performing schools have positive effects on student and teacher outcomes? 6

CONSORTIUM FOR EDUCATIONAL RESEARCH AND EVALUATION – NORTH CAROLINA Purpose of this Evaluation Project To provide both summative and formative information about RttT efforts to increase the overall supply and to ensure the equitable distribution of effective educators statewide Evaluation Strands Baseline Regional Leadership Academies NCTC & TFA Expansion Strategic Staffing New Teacher Induction NCVPS Blended STEM Courses 7 2. Equitable Supply and Distribution of Teachers and Leaders

CONSORTIUM FOR EDUCATIONAL RESEARCH AND EVALUATION – NORTH CAROLINA Overall Evaluation Questions What is the nature and quality of the experiences provided by each of the 6 programs? Are students affected by each of these programs better off than students in schools and districts not served by these programs? Are these initiatives cost-effective and sustainable? To what extent did the initiatives further the goal of having an effective teacher in every classroom and an effective principal in every school? 8 2. Equitable Supply and Distribution of Teachers and Leaders

CONSORTIUM FOR EDUCATIONAL RESEARCH AND EVALUATION – NORTH CAROLINA 3. Professional Development Purpose of this Evaluation Project To conduct ongoing analysis of the delivery and quality of state- and local-level professional development, with the goal of analyzing the impact of the PDI on local capacity, teacher practices, and student achievement. We will examine longitudinal education data combined with data collected using a sample of schools approach. 9

CONSORTIUM FOR EDUCATIONAL RESEARCH AND EVALUATION – NORTH CAROLINA 3. Professional Development Major Evaluation Questions State Strategies: To what extent did the state implement and support proposed RttT PD efforts? Short-Term Outcomes: What were direct outcomes of State-level RttT PD Efforts? Intermediate Outcome: To what extent did RttT PD efforts successfully update the NC Education Workforce? Impacts on student performance: To what extent are gains in student performance outcomes associated with RttT PD ? 10

CONSORTIUM FOR EDUCATIONAL RESEARCH AND EVALUATION – NORTH CAROLINA 4. Turnaround of LEAs and Schools Purpose of this Evaluation Project To understand the extent to which and the ways in which interventions by the District and School Transformation division (DST) improve outcomes for students in the state’s lowest-performing schools and districts To explore the fidelity of implementation of the STEM Schools initiative and examine its impacts on students, teachers, principals, schools, and school networks. 11

CONSORTIUM FOR EDUCATIONAL RESEARCH AND EVALUATION – NORTH CAROLINA 4. Turnaround of LEAs and Schools Turnaround of LEAs and Schools Evaluation Questions What problems are identified in the low-performing schools and districts? What are the main intervention strategies that the District and School Transformation unit employs to improve low-performing schools? What are the intended mechanisms of improvement? How do the strategies work? Do the strategies and mechanisms play out as intended? What is the impact of the intervention strategies on intermediate outcomes as well as student achievement and graduation rates? 12

CONSORTIUM FOR EDUCATIONAL RESEARCH AND EVALUATION – NORTH CAROLINA 4. Turnaround of LEAs and Schools STEM Anchor Evaluation Questions To what extent have the network of STEM anchor and cluster schools been implemented as intended? What are the impacts of the network of STEM anchor and cluster schools on student and on school-level outcomes and how these impacts compare with the impacts of other transformation models? Can the impacts on student performance be disaggregated by student and school characteristics? What mechanisms are put in place for the sustainability and scaling up of the model, or its most successful elements? 13

CONSORTIUM FOR EDUCATIONAL RESEARCH AND EVALUATION – NORTH CAROLINA 5. Local-Level Implementation & Spending Purpose of this Evaluation Project To determine how Race to the Top funding is being allocated and used across districts and schools across NC 14

CONSORTIUM FOR EDUCATIONAL RESEARCH AND EVALUATION – NORTH CAROLINA 5. Local-Level Implementation & Spending Cloud Questions To what extent does the Cloud reduce state & local expenditures for technology? To what extent does the Cloud provide reliable, secure, accessible, and efficient service? How satisfied are LEAs with the Cloud Computing infrastructure? Local Spending Questions How do local districts spend RttT funds? Are some local RttT spending patterns associated with higher student performance in schools and districts? Local Efficiencies and Savings Questions Do RttT funds alter costs incurred by the state and districts? 15

CONSORTIUM FOR EDUCATIONAL RESEARCH AND EVALUATION – NORTH CAROLINA 6. Overall Impact Purposes of this Evaluation Project To provide estimations of the overall impact of RttT- funded initiatives To explore under what conditions and circumstances the initiatives collectively and in various combinations appeared to be most effective, and for whom To consider sustainability options beyond the life of the grant To track and compare the metrics/goals defined in the proposal 16

CONSORTIUM FOR EDUCATIONAL RESEARCH AND EVALUATION – NORTH CAROLINA 6. Overall Impact Evaluation Questions Was each RttT initiative implemented as intended? What are the overall impacts of RttT on increasing student performance, such as achievement, engagement, attendance, graduation? Are the impacts of RttT on student performance larger in some schools/districts than others (for example, high-poverty or low- performing schools)? Are some RttT initiatives more effective in increasing student performance than others? How can the successful RttT initiatives be sustained after 2014? 17

CONSORTIUM FOR EDUCATIONAL RESEARCH AND EVALUATION – NORTH CAROLINA Omnibus Teacher and Leader Survey Administered to probability sample of 358 schools across the state Assessing “Instructional Climate”  23 domains on leadership and organizational conditions affecting instruction Baseline Fall 2011, bi-annually over course of RttT Next administration Feb/Mar

CONSORTIUM FOR EDUCATIONAL RESEARCH AND EVALUATION – NORTH CAROLINA School Level Activities Report 19

CONSORTIUM FOR EDUCATIONAL RESEARCH AND EVALUATION – NORTH CAROLINA Questions? (919)