Sustaining School Reform A guide to support evaluation policy Presented at Evaluation 2010, San Antonio Texas 11/12/10 Zoe Barley & Lou Cicchinelli McREL Denver, Colorado

Presentation Outline School reform evaluation policy - under earlier federal initiatives under current initiatives Revisions to a guide for school reform evaluation Concerns about evaluation policy

School Reform Evaluation - then Evaluation required CSRD Program Criteria # 8 Have an evaluation plan

Additional metrics for SIGs And now Reporting metrics (not an evaluation) Existing metrics Additional metrics for SIGs Existing metrics: AYP status by targets, % proficient x grade x subgroup, grad rate, dropout rate, attendance, discipline, truancy New metrics for SIGs: type of intervention, minutes in the school year, scale scores, advanced coursework, college enrollment, teachers x performance, teacher attendance

Creating a Guide for Schools Evaluating for Success - 1999 Sustaining Coherent Reform - 2010 Guide for schools in reform, especially CSRD grantees Guide for schools in reform, especially SIG grantees

Structure of the Guide 5 stages with steps Narrative 8 worksheet examples 8 blank worksheets Tips and Figures

This guide will assist users to - - - • develop a detailed evaluation design; • conduct an evaluation of school-reform implementation and impact; • report key findings useful to key audiences; and • encourage use of evaluation findings

The purpose . . .is to provide practical information, tips, and tools, which collectively serve as a resource for novice evaluators as well as a reference for expert evaluators.

All kinds of schools well almost all. .

5 stages of evaluation process Planning for the evaluation Designing the evaluation Conducting the evaluation Reporting the findings Encouraging use of the findings

Evaluation preliminaries Examine process and impacts Create a flexible design Keep the design realistic Establish an evaluation team

Stage 1: Planning for the evaluation Step 1 – Define the purpose(s) Step 2 – Identify requirements Step 3 – Understand the context Step 4 – Consider a consultant Not going to cover the guide in detail – there are 2 worksheets in this stage. One is to gather the requirements – come back to the federal ones for sig grants later – others would be related to a reform initiative and/or any local requirements (e.g. School Board wants to know how teachers feel about the reform.)

Second worksheet. Helps consider and organize the relevant local background.

Blank from back or guide

EXAMPLE Example of a Tips piece

Stage 2: Designing the evaluation Step 1 – Link the evaluation to the program Step 2 – Identify the evaluation questions Step 3 –Choose data sources, variables measures Step 4 – Move from planning to action Four Worksheets for this stage – one for each Step.

Always a bit queasy at this stage – asking local school or district folk to create a logic model – to create it users are asked to pull the context material from Worksheet 2 for the first two columns, then consider the more immediate outcomes the program intends to accomplish (Column 4) – and finally what are the assumptions about what will happen to accomplish those outcomes.

In Step 3 of this stage they are developing the evaluation questions In Step 3 of this stage they are developing the evaluation questions. Here’s a Figure to organize the questions as context, implementation, or outcome. There is a worksheet for recording the questions.

Worksheet 5: Data Sources & Variables Evaluation Questions Data Sources Variables Measures CONTEXT IMPLEMENTATION OUTCOME Simplification of worksheet 5. First they bring the evaluation questions into the first column, then complete the additional columns for each question. Worksheet 6 is a design matrix – organizing the work into tasks, making an assignment of a lead, and setting up a schedule to accopmlish.

Stage 3 – Conducting the evaluation Step 1 – Data collection instruments Step 2 – Collecting data Step 3 – Getting ready for analysis Step 4 – Analysis Four steps in Stage 3 - There’s a good table about data collection instruments – pros and cons on page 26.

EXAMPLE ETC.

There is only one worksheet for Stage 3 – the data collection plan There is only one worksheet for Stage 3 – the data collection plan. This takes only the data collection aspects of the matrix and gets very concrete about how, who, and when.

Stage 4 Stage 5 Organize findings Create opportunities to discuss Reporting findings Encouraging use Organize findings Select reporting methods Frame recommendations Deliver findings Create opportunities to discuss Follow up Revise evaluation plan

Concerns about evaluation policy School/District Level Staff time/capacity Ability to find/hire a consultant Energy to sustain Fractious school politics The Broader Context De-emphasis on evaluation ‘Evaluation’ as RCT SIG data requirements Focus on SLDS requirements Local - Presented to group of rural educators – aware of how difficult evaluation is for them – staff time and capacity, proximity of consultants Broader – SIG grant does not include clear evaluation policy, the term evaluation has been associated strongly with RCTs placing it in a summative role, the concern for data burden seems to have diminished data collection and use for SIG grantees, & seeing the data stream as moving to state and national has diminished the perspective of local use

Questions? Sustaining Coherent Reform http://www.mcrel.org/topics/products/418 Louis F. Cicchinelli, Ph.D. lcicchinelli@mcrel.org Zoe A. Barley, Ph.D. zbarley@mcrel.org Doing some revisions to the Introduction and first worksheet.