1. © 2009 Berman Group. Evidence-based evaluation RNDr. Jan Vozáb, PhD partner, principal consultant Berman Group

2. © 2009 Berman Group. Objectives of the evidence-based evaluation To increase impact and value added of the Cohesion Policy interventions To increase effectiveness and efficiency of interventions in terms of achieving results that make real (and desirable) changes To provide better and more reliable information for strategic management of OPs

3. © 2009 Berman Group. Key issues and dilemmas Evidence is always part of its context – explanation of findings is part of the evidence-based evaluation The more complex, complicated or integrated is the intervention, the more difficult is to obtain reliable, strong evidence of its effects The better is the evidence for measuring of effects, the more difficult is to cause them by the intervention itself

4. © 2009 Berman Group. Evaluators´ position Real issues MAs intentions Interventions / Operations Outcomes of interventions Real effects Evaluation

5. © 2009 Berman Group. Programming issues related to evidence-based evaluation Problem and needs identification Definition of objectives Planning of operations Set up of targets and indicators structure Implications for evaluations: Difficulties to link outcomes to the objectives and needs Difficulties to find evidence of real effects of operations (although they exist) Intervention logic

6. © 2009 Berman Group. Implementation issues related to evidence-based evaluation Random or non-targeted selection of projects Week coordination of operations within OP Feedback on outcomes for management purposes Costs to track evidence on results/impacts Implications for evaluation: Complicated establishing of evidence on outcomes from available data sets Lack of suitable data for evaluation – monitoring data only

7. © 2009 Berman Group. Methodological issues related to evidence-based evaluation Formulation of evaluation questions Use of representative data Explanation of findings in proper context Deriving conclusions, providing recommendations Implications for evaluation: Evidence based on intervention logic Availability of data Combination of qualitative and quantitative evidence

8. © 2009 Berman Group. Recommendations for improving evidence-based evaluation To make sure that programming is sufficiently evidence-based as a prerequisite for evaluation To strengthen vertical and horizontal intervention logic within and among OPs and partnership contract To make sure that operations aim at causes of problems and are sufficiently coherent and focused To strengthen management capacity of MAs: –To manage operations and deliver interventions –To take more responsibility for results and wider effects of interventions (not only for indicators/targets)

9. © 2009 Berman Group. Recommendations for improving evidence-based evaluation To strengthen analytical capacity and skills of evaluators To use evaluation in a better way to support decision-making: –Small, targeted, ad hoc evaluations –To adjust monitoring systems for evaluation purposes –To strengthen links between evaluators and professionals in thematic fields To establish or strengthen feedback on incorporation of evaluators´ findings into OP management

10. © 2009 Berman Group. Thank you for your attention Jan Vozáb vozab@bermangroup.cz