IMPLEMENTING LEAPS IN CENTRAL AND EASTERN EUROPE: TRAINERS’ HANDBOOK Monitoring and Evaluating Results

PHASE 2: ASSESSING ENVIRONMENTAL ISSUES ND SETTING PRIORITIES MONITORING AND EVALUATING RESULTS IMPLEMENTING LEAPS IN CENTRAL AND EASTERN EUROPE: TRAINERS’ HANDBOOK 2 Monitoring and Evaluating Results 1)Review environmental targets and indicators 2)Establish reporting system 3)Collect data on baseline conditions and project results 4)Evaluate results 5)Utilise evaluation results 6)Communicate results to the community “There is nothing so easy to learn as experience and nothing so hard to apply.” Josh Billings

PHASE 2: ASSESSING ENVIRONMENTAL ISSUES ND SETTING PRIORITIES MONITORING AND EVALUATING RESULTS IMPLEMENTING LEAPS IN CENTRAL AND EASTERN EUROPE: TRAINERS’ HANDBOOK 3 The monitoring and evaluation process provides an opportunity to: Why Monitor and Evaluate? Effective monitoring and evaluation provide a steady, systematic flow of information that strengthens project implementation. Compare implementation efforts with original goals and targets Determine whether sufficient progress is being made toward achieving expected results Determine whether you are adhering to the project time schedule

PHASE 2: ASSESSING ENVIRONMENTAL ISSUES ND SETTING PRIORITIES MONITORING AND EVALUATING RESULTS IMPLEMENTING LEAPS IN CENTRAL AND EASTERN EUROPE: TRAINERS’ HANDBOOK 4 An effective monitoring and evaluation programme requires: Consider forming a monitoring and evaluation team to help design the evaluation approach, collect data and evaluate project results. Collecting and analysing data throughout the life of a project Reporting on progress toward environmental improvements Making connections between actions and intended outcomes Engaging stakeholders in the process

PHASE 2: ASSESSING ENVIRONMENTAL ISSUES ND SETTING PRIORITIES MONITORING AND EVALUATING RESULTS IMPLEMENTING LEAPS IN CENTRAL AND EASTERN EUROPE: TRAINERS’ HANDBOOK 5 1. Review Environmental Targets and Indicators Targets include: Measurable commitments, e.g. 20% reduction in pollution levels Specified time frames, e.g. within five years Indicators: Measure whether environmental targets are being achieved Targets and indicators provide a framework for measuring progress in implementing actions identified in environmental action plan.

PHASE 2: ASSESSING ENVIRONMENTAL ISSUES ND SETTING PRIORITIES MONITORING AND EVALUATING RESULTS IMPLEMENTING LEAPS IN CENTRAL AND EASTERN EUROPE: TRAINERS’ HANDBOOK 6 Be sure to consult the stakeholder group and/or local government before changing targets. Review targets and indicators and consider: Since the action plan was prepared, has the stakeholder group collected new information about the expected impacts of selected actions? Are the targets realistic and the proposed timeframes reasonable? Are the indicators valid measures of selected targets?

PHASE 2: ASSESSING ENVIRONMENTAL ISSUES ND SETTING PRIORITIES MONITORING AND EVALUATING RESULTS IMPLEMENTING LEAPS IN CENTRAL AND EASTERN EUROPE: TRAINERS’ HANDBOOK 7 2. Establish Reporting System An effective reporting system: Records the performance of all institutions with implementation responsibilities Provides a system of accountability for all responsible parties on how well they are achieving the targets Consider whether the proposed reporting system includes: Articulated environmental targets and a set of indicators A schedule and consistent reporting format and framework Opportunity for responsible parties to coordinate actions and review performance A link between the evaluation process and statutory planning cycles, e.g. annual budgeting and capital planning

PHASE 2: ASSESSING ENVIRONMENTAL ISSUES ND SETTING PRIORITIES MONITORING AND EVALUATING RESULTS IMPLEMENTING LEAPS IN CENTRAL AND EASTERN EUROPE: TRAINERS’ HANDBOOK 8 3. Collect Data on Baseline Conditions and Project Results Quantitative data: Information that can be measured Qualitative data: Information that is hard to measure, such as the difficulty of problems encountered, consumer satisfaction and unanticipated benefits Most evaluations require collecting two kinds of data: As you collect data, considering the following questions: To which indicators does the data apply? Who are key sources? Are they represented on the monitoring/ evaluation team? How valid and accurate is the data? Is the data readily available? Are there any costs associated with acquiring the data? For indicators where no data exists, how can new data be acquired?

PHASE 2: ASSESSING ENVIRONMENTAL ISSUES ND SETTING PRIORITIES MONITORING AND EVALUATING RESULTS IMPLEMENTING LEAPS IN CENTRAL AND EASTERN EUROPE: TRAINERS’ HANDBOOK 9 4. Evaluate Results When evaluating results, it is important to collect data to monitor both compliance and effectiveness. Evaluation involves comparing actual results with the targets and whether results were achieved within the designated timeframe and budget. Compliance monitoring measures whether the implementing institution did what it said it was going to do, e.g., install 5,000 low-flow showerheads. Effectiveness monitoring measures whether the actions achieved their intended result, e.g., reduced water usage by 20%.

PHASE 2: ASSESSING ENVIRONMENTAL ISSUES ND SETTING PRIORITIES MONITORING AND EVALUATING RESULTS IMPLEMENTING LEAPS IN CENTRAL AND EASTERN EUROPE: TRAINERS’ HANDBOOK 10 Questions to Ask in Preparing an Evaluation Did the action achieve its intended effect? Were responsibilities and deadlines clear and reasonable? Were the actions consistent with the implementation plan? Were the costs consistent with the budget? What external and internal factors affected the results? What are the recommendations for improvements?

PHASE 2: ASSESSING ENVIRONMENTAL ISSUES ND SETTING PRIORITIES MONITORING AND EVALUATING RESULTS IMPLEMENTING LEAPS IN CENTRAL AND EASTERN EUROPE: TRAINERS’ HANDBOOK 11 Proposed Format for Evaluation Report Summary of results Project description: project history, participating institutions, evaluation team, goals and targets Data collection information: selected indicators, reporting requirements and data collection methods Qualitative impacts Satisfaction level of beneficiaries Unforeseen benefits Educational impacts Knowledge acquired Skills developed Attitudes altered or reinforced Results: Quantitative impacts Comparison to targets Environmental gains Targets achieved within timeframe Cost consistent with budget Difficulties encountered: Problems related to both internal factors and external factors Lessons learned: Analysis of what knowledge has been gained in the project and recommendations for future implementation efforts

PHASE 2: ASSESSING ENVIRONMENTAL ISSUES ND SETTING PRIORITIES MONITORING AND EVALUATING RESULTS IMPLEMENTING LEAPS IN CENTRAL AND EASTERN EUROPE: TRAINERS’ HANDBOOK Utilise Evaluation Results Evaluation process should provide information that can be utilised by project managers and staff to improve results. Evaluation results inform decisions and provide information on how to improve project performance. Implementing institutions are more likely to use information generated from an evaluation if they understand and participate in the evaluation process. Who will decide whether to make these changes? Who will hold implementing institutions accountable for making those changes? How often will changes be made? Some key questions to consider in utilising evaluation results: At what point will changes to policies or programmes be made based upon evaluation results?

PHASE 2: ASSESSING ENVIRONMENTAL ISSUES ND SETTING PRIORITIES MONITORING AND EVALUATING RESULTS IMPLEMENTING LEAPS IN CENTRAL AND EASTERN EUROPE: TRAINERS’ HANDBOOK Communicate Results to the Community Effective communication with the community involves: It is critical to keep community members informed about environmental conditions, improvements made and actions they can take. Providing regular information to community members Ensuring that implementing institutions hear opinions and concerns of community Using a variety of techniques such as visual displays, oral presentations, summary statements, interim reports and informal conversations

PHASE 2: ASSESSING ENVIRONMENTAL ISSUES ND SETTING PRIORITIES MONITORING AND EVALUATING RESULTS IMPLEMENTING LEAPS IN CENTRAL AND EASTERN EUROPE: TRAINERS’ HANDBOOK 14 Ways to Communicate Evaluation Results to the Public Public-friendly executive summary and popular articles Press conferences featuring public figures such as the mayor Verbal presentations to selected groups Professionally designed graphics, charts and displays Video presentations Debate on controversial issues