1 International Conference Evaluation: Evidence-based Tools for Decision-making Future Cohesion Policy: Implications for Monitoring and Evaluation Budapest May 24-25, 2011 Dr Jim Fitzpatrick Managing Director Fitzpatrick Associates 122 Ranelagh Village Dublin 6 Tel: Fax:

2  issues in future Cohesion Policy affecting Monitoring and Evaluation (M+E)  emerging parameters of period  the “new” Logical Framework  challenges in implementing new approaches  some practical suggestions TOPICS

3 EU STRUCTURAL AND COHESION FUNDING  20 years + of multi-annual programmes  M+E central to the programming package  Objectives, priorities, targets/indicators  Agreed strategies, programmes  Regular monitoring  Formal evaluation  Gradual development of M+E practice  will involve significant changes CEE entry

: INCREASING THE EFFECTIVENESS OF COHESION POLICY  the main focus of evaluations “tended to be on processes and financial implementation rather than on the actual results of programmes”, ( Ex Post Evaluation of Cohesion Policy Programme , Synthesis Report April 2010)  a Member State-Commission National Strategic contract where Member States (or Regions) “commit themselves to quantified and verifiable objectives”. ( An Agenda for a Reformed Cohesion Policy, Barca Report, April 2009)  “specific binding conditionality in the areas directly linked to cohesion policy would be agreed with each Member State and/or Region – depending on the institutional context…” ( Investing In Europe’s Future: Fifth Report on Economic, Social and Territorial Cohesion, Nov. 2010) Work of High Level Group on Future Cohesion Policy, Conditionality Task Force (Feb.-April 2011), Evaluation Unit/Network, Hungarian Presidency High Level Conference (March 31/April 1).

5 A CONFLUENCE OF RELATED STRANDS Monitoring And Evaluation Thematic Concentration Binding Contracts Conditionality Better Information Better Evaluation/ CFA Performance/ Results measuring outcomes

6 provides goals provides themes regulatory requirements evaluation will inform content/progress ex ante, ongoing, ex post EMERGING NEW ARCHITECTURE * Possibly for Cohesion Fund, ERDF, ESF, EAFRD, EEF Europe 2020 Goals Common Strategic Framework (CSF) ?* Regulations Partnership and Investment Contracts* Operational Programmes Themes/Axes (from menu)

7 STRENGTHENED CONDITIONALITY?  already is various conditionality: regulatory (inc. M+E); strategic. Infrastructure planning; institutional  types of conditionality:  “macroeconomic” – part of Stability/Growth Pact  “ex ante” – preconditions (see existing categories)  “structural reform” – structural/admin reform milestones  “performance” – Programme/EC objectives  incorporated in Partnership Contracts?  possible financial incentives/sanctions?  if implemented, could have implications for Monitoring/Evaluation

8 Other Factors Monitoring and Evaluation Programming Strategy Needs Thematic Objective Intended Result Contribution - Impact Policy Allocated INPUTS Targeted OUTPUTS Actual INPUTS Achieved OUTPUTS Actual Result NEW LOGICAL FRAMEWORK  outcomes monitored/impacts evaluated  inputs/outputs “contribute” to outcomes  outcome indicators should be “responsive” to intervention Source: V. Gaffey, Acting Director, Policy Development, DG Regio, Intervention Evaluation Conference, Budapest, May 24-25, 2011.

9 RE-DEFINING THE LOGIC Source: based on DG Regio, Concepts and Ideas: Monitoring and Evaluation in Practice of European Cohesion Policy 2014+, Draft, 30 March 2011 Old: InputsOutputsResults (Short/medium) IMPACTS (Long term) MonitoringEvaluation Monitoring New: InputsOutputs Results/ outcomes Evaluation Impact

10 CRITERIA FOR A GOOD OUTCOME INDICATOR Reasonable: capturing the essence of an outcome according to a reasonable argument about which features of the outcome they can and cannot represent Robust: reliable, statistically and analytically validated, and, as far as practicable, complying with internationally recognised standards and methodologies; Responsive to policy: linked in as direct way as possible to the policy interventions for whose assessment they are used, while not being subject to manipulation; Normative: having a clear and accepted normative interpretation (i.e. there must be agreement that a movement in a particular direction or within a certain range is a favourable or an unfavourable result); Feasible: built, as far as practicable, on available underlying data, their measurement not imposing too large a burden on Member States, on enterprises, nor on the citizens; Debatable: timely available to a wide public, with room being built for public debate and for their own revision when needed and motivated. Source: F. Barca, P. McCann, Outcome Indicators and Targets – Towards a Performance Oriented EC Cohesion Policy, High-level Group on Future Cohesion Policy, Meeting No. 8, 15 Feb RRR-NFD instead of SMART! The challenge is in the “Third R”

11 THE RESPONSIVENESS ISSUE: SUB-CRITERIA FOR OUTCOME INDICATORS?  needs to be specifically related to the output?  easier if OP is the only source of funding (national + EU)?  needs to be about the beneficiaries?  easier if there is a clear physical link, e.g. infrastructure  needs a typology of intervention types* *i.e. infrastructure, subsidies, service provision. Use Pilots to develop this

12 CHALLENGES REGARDING OUTCOME/RESULT INDICATORS Conceptual:  the causation problem remains?  a lot of judgement? Communication:  can we be confident this will address the issue?  trade-off between communication and robustness Capacity:  who is going to define the indicators? A specialist task. Will need forensic precision Time-lags:  when will attributable outcomes arise? Conditionality:  M+E insufficiently robust to support binding conditionality

13 SOME SUGGESTIONS FOR COMMISSION/MEMBER STATES  skills, capacity, technical assistance, technical support  M+E as communication tools? Simplify language? Consistent terminology, e.g. “outcomes” v “results”  revisit institutional context, inc. Monitoring Committees, Managing Authority  Nature of Commission Guidelines and technical support  use of regular Peer Review Groups, (“boots on the ground”) ongoing Evaluators  merge Monitoring and Evaluation function in Member States  greater role for Eurostat/National Statistical Offices (but avoid pure context indicators)  nature of Multi-annual Evaluation plans

14 And finally…  keep the Regulations general and Guidelines detailed, specific (not vice versa)  don’t rush the Guidelines THANK YOU.