Program Evaluation Strategies to Improve Teaching for Learning Rossi Ray-Taylor and Nora Martin Ray.Taylor and Associates MDE/NCA Spring School Improvement Conference 2008

Introductions Overview for the session

Evaluation comfort levels

Research driven school design Rigor, relevance, relationships

Evaluation can provide information about Definition and evidence of the problem (needs assessment) Contextual information about the factors related to the problem and solutions The input factors and resources The interventions, processes and strategies employed Outcomes, results and effectiveness

Program evaluation can be a form of action research

Elements of a sound evaluation Clear statement of what the project is intended to do and why –Needs assessment –Theory of action –Clear, measurable statement of goal attainment Appropriate evaluation methods and tools Transparency

Evaluation can lay the ground work for accountability

Key concepts Identify the total system impact of program and financial decisions Institutions led by data and evidence of results

Types of Data Achievement Demographic Program Perception Process Costs

Qualities of Data & Information Time series, repeated data collection– how does the effect change over time? Cohort analysis – overall effect on one group over time Benchmarking & standards – makes the data relative, establishes context and comparables

Triangulation – looking for information in multiple indicators Pattern & trend analysis Leading & lagging indicators

Practical matters Accuracy Reliability Validity Accessibility

Evaluation Employ evaluation strategies from the very beginning of a project and assemble and review effectiveness data. State measurable outcomes, document processes and review progress throughout the life of the project.

“Where outcomes are evaluated without knowledge of implementation, the results seldom provide a direction for action because the decision maker lacks information about what produced the outcomes (or lack of outcomes).” Michael Quinn Patten Quoted in “Data Analysis for Comprehensive Schoolwide Improvement”, Victoria L. Bernhardt, 1998

Central Evaluation Questions Did the program / project do what was intended? –Did the project stick to the plan? There may be valid reasons for varying from the plan – if so what are they?

What is the theory of change for the project? Why is this project being carried out this way and why is it judged to be the most appropriate way? –These questions are important because they help when judging the impact of changes in the plan.

What is the context for the project? –Issues include past trends, local politics, resource distribution, threats, opportunities, strengths and weaknesses

What actually happened during the course of the project? Who was served by the project? Why? Who was not served by the project? Why? Was the project valued by the intended audience?

What were the inputs, resources – both real e.g., financial and material – and intellectual? What was the “cost” of the project?

What were the results /outcomes? Were goals and objectives met? What were the intended and unintended consequences? What was the impact on the overall system? Was there a process impact – did the project result in a change in the way that business is done?

A successful project begins and ends with a good evaluation design

How do these questions improve teaching for learning?

The best and most “sticky”, lasting interventions have the following components They are based in research and evidence They are locally constructed and customized

They are targeted to a clear view of the problem to be solved They are built for sustainability They are “owned” – not just a matter of compliance They are designed to build local capacity

Understand change and apply research about change to improve teaching for learning Change takes trust Change takes building relationships Change takes endurance (time) Change takes knowledge of research

Examine policies and practices that serve as barriers and those that serve as catalysts to achievement

Evaluate Audit the system Measure results Change to goal not just awareness or implementation

Meta Evaluation Use meta evaluation strategies to look for results across projects and interventions

Evaluation readiness Identify and clearly state program goals and outcomes Transform these goals into measurable objectives Define program theory and supporting research

Develop the formative and summative evaluation plan Develop the plan to gather data, deploy evaluation resources and gather information from and report to stakeholders

Design for continuous feedback and transparency

Consider system evaluation policies and expectations At the time of proposal initiatives, programs and projects should be designed to include program evaluation

“It’s easy to make judgments – that’s evaluation. It’s easy to ask questions about impact – that’s evaluation. It’s easy to disseminate reports – that’s evaluation. What’s hard is to put all those pieces together in a meaningful whole which tells people something they want to know and can use about a matter of importance. That’s evaluation.” Halcolm Quoted in “Data Analysis for Comprehensive Schoolwide Improvement”, Victoria L. Bernhardt, 1998

Program evaluation planning document Goal: What does this project intend to achieve? State goals in observable terms. Limit the project to 3-5 overall goals. Objective State project objectives in measurable terms Measure/ Criteria How will the objective be measured? What tools or methods will be used? What are the criteria for success? Evaluation activities & questions What activities and tasks need to be accomplished to carry out evaluation of this objective? State the evaluation questions Time line When will data be collected and reported? Person responsible Who is responsible for this task? Resources and budget What is the budget and what other resources will be needed?

In the beginning you think. In the end you act. In between you negotiate the possibilities. Some people move from complexity to simplicity and on into catastrophe. Others move from simplicity to complexity and onward into full scale confusion. Simplification makes action possible in the face of overwhelming complexity. It also increases the odds of being wrong. The trick is to let a sense of simplicity inform our thinking, a sense of complexity inform our actions, and a sense of humility inform our judgments…” Michael Quinn Patten (p. 143 Bernhardt) Quoted in “Data Analysis for Comprehensive Schoolwide Improvement”, Victoria L. Bernhardt, 1998

Resources Center for Evaluation & Education Policy The Evaluation Center; Western Michigan University

More Resources What Works in Schools Translating Research into Action by Robert J. Marzano Data Analysis for Comprehensive Schoolwide Improvement; Victoria L. Bernhardt

More Resources The “Data Wise” Improvement Process: Eight steps for using test data to improve teaching and learning, by Kathryn Parker Boudett et al in Harvard Education Letter, January/February 2006 Volume 22, Number 1.

Rossi Ray-Taylor, PhD Nora Martin, PhD Ray.Taylor and Associates 2160 S. Huron Parkway, Suite 3 Ann Arbor, Michigan