1. Sustaining School Reform A guide to support evaluation policy
Presented at Evaluation 2010, San Antonio Texas
11/12/10
Zoe Barley & Lou Cicchinelli
McREL Denver, Colorado

2. Presentation Outline School reform evaluation policy -
under earlier federal initiatives
under current initiatives
Revisions to a guide for school reform evaluation
Concerns about evaluation policy

3. School Reform Evaluation - then
Evaluation required
CSRD Program Criteria # 8
Have an evaluation plan

4. Additional metrics for SIGs
And now
Reporting metrics (not an evaluation)
Existing metrics
Additional metrics for SIGs
Existing metrics:
AYP status by targets, % proficient x grade x subgroup, grad rate, dropout rate, attendance, discipline, truancy
New metrics for SIGs:
type of intervention, minutes in the school year, scale scores, advanced coursework, college enrollment, teachers x performance, teacher attendance

5. Creating a Guide for Schools
Evaluating for Success
Sustaining Coherent Reform
Guide for schools in reform, especially CSRD grantees
Guide for schools in reform, especially SIG grantees

6. Structure of the Guide
5 stages with steps
Narrative
8 worksheet examples
8 blank worksheets
Tips and Figures

7. This guide will assist users to - - -
• develop a detailed evaluation design;
• conduct an evaluation of school-reform implementation and impact;
• report key findings useful to key audiences; and
• encourage use of evaluation findings

8. The purpose . . .is to provide practical information, tips, and tools, which collectively serve as a resource for novice evaluators as well as a reference for expert evaluators.

9. All kinds of schools
well
almost all. .

10. 5 stages of evaluation process
Planning for the evaluation
Designing the evaluation
Conducting the evaluation
Reporting the findings
Encouraging use of the findings

11. Evaluation preliminaries
Examine process and impacts
Create a flexible design
Keep the design realistic
Establish an evaluation team

12. Stage 1: Planning for the evaluation
Step 1 – Define the purpose(s)
Step 2 – Identify requirements
Step 3 – Understand the context
Step 4 – Consider a consultant
Not going to cover the guide in detail – there are 2 worksheets in this stage. One is to gather the requirements – come back to the federal ones for sig grants later – others would be related to a reform initiative and/or any local requirements (e.g. School Board wants to know how teachers feel about the reform.)

13. Second worksheet. Helps consider and organize the relevant local background.

14. Blank from back or guide

15. EXAMPLE
Example of a Tips piece

16. Stage 2: Designing the evaluation
Step 1 – Link the evaluation to the program
Step 2 – Identify the evaluation questions
Step 3 –Choose data sources, variables measures
Step 4 – Move from planning to action
Four Worksheets for this stage – one for each Step.

17. Always a bit queasy at this stage – asking local school or district folk to create a logic model – to create it users are asked to pull the context material from Worksheet 2 for the first two columns, then consider the more immediate outcomes the program intends to accomplish (Column 4) – and finally what are the assumptions about what will happen to accomplish those outcomes.

18. In Step 3 of this stage they are developing the evaluation questions
In Step 3 of this stage they are developing the evaluation questions. Here’s a Figure to organize the questions as context, implementation, or outcome. There is a worksheet for recording the questions.

19. Worksheet 5: Data Sources & Variables
Evaluation Questions
Data Sources
Variables
Measures
CONTEXT
IMPLEMENTATION
OUTCOME
Simplification of worksheet 5. First they bring the evaluation questions into the first column, then complete the additional columns for each question. Worksheet 6 is a design matrix – organizing the work into tasks, making an assignment of a lead, and setting up a schedule to accopmlish.

20. Stage 3 – Conducting the evaluation
Step 1 – Data collection instruments
Step 2 – Collecting data
Step 3 – Getting ready for analysis
Step 4 – Analysis
Four steps in Stage 3 - There’s a good table about data collection instruments – pros and cons on page 26.

21. EXAMPLE
ETC.

22. There is only one worksheet for Stage 3 – the data collection plan
There is only one worksheet for Stage 3 – the data collection plan. This takes only the data collection aspects of the matrix and gets very concrete about how, who, and when.

23. Stage 4 Stage 5 Organize findings Create opportunities to discuss
Reporting findings
Encouraging use
Organize findings
Select reporting methods
Frame recommendations
Deliver findings
Create opportunities to discuss
Follow up
Revise evaluation plan

24. Concerns about evaluation policy
School/District Level
Staff time/capacity
Ability to find/hire a consultant
Energy to sustain
Fractious school politics
The Broader Context
De-emphasis on evaluation
‘Evaluation’ as RCT
SIG data requirements
Focus on SLDS requirements
Local - Presented to group of rural educators – aware of how difficult evaluation is for them – staff time and capacity, proximity of consultants
Broader – SIG grant does not include clear evaluation policy, the term evaluation has been associated strongly with RCTs placing it in a summative role, the concern for data burden seems to have diminished data collection and use for SIG grantees, & seeing the data stream as moving to state and national has diminished the perspective of local use

25. Questions? Sustaining Coherent Reform
Louis F. Cicchinelli, Ph.D.
Zoe A. Barley, Ph.D.
Doing some revisions to the Introduction and first worksheet.