1. 1 International Conference Evaluation: Evidence-based Tools for Decision-making Future Cohesion Policy: Implications for Monitoring and Evaluation Budapest May 24-25, 2011 Dr Jim Fitzpatrick Managing Director Fitzpatrick Associates 122 Ranelagh Village Dublin 6 Tel: +353 1 6280084 Fax: + 353 1 6219771

2. 2  issues in future Cohesion Policy affecting Monitoring and Evaluation (M+E)  emerging parameters of 2014-20 period  the “new” Logical Framework  challenges in implementing new approaches  some practical suggestions TOPICS

3. 3 EU STRUCTURAL AND COHESION FUNDING  20 years + of multi-annual programmes  M+E central to the programming package  Objectives, priorities, targets/indicators  Agreed strategies, programmes  Regular monitoring  Formal evaluation  Gradual development of M+E practice  2014-20 will involve significant changes 1989 1990 1991 1992 1993 1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 CEE entry

4. 4 2014-20: INCREASING THE EFFECTIVENESS OF COHESION POLICY  the main focus of evaluations “tended to be on processes and financial implementation rather than on the actual results of programmes”, ( Ex Post Evaluation of Cohesion Policy Programme 2000-06, Synthesis Report April 2010)  a Member State-Commission National Strategic contract where Member States (or Regions) “commit themselves to quantified and verifiable objectives”. ( An Agenda for a Reformed Cohesion Policy, Barca Report, April 2009)  “specific binding conditionality in the areas directly linked to cohesion policy would be agreed with each Member State and/or Region – depending on the institutional context…” ( Investing In Europe’s Future: Fifth Report on Economic, Social and Territorial Cohesion, Nov. 2010) Work of High Level Group on Future Cohesion Policy, Conditionality Task Force (Feb.-April 2011), Evaluation Unit/Network, Hungarian Presidency High Level Conference (March 31/April 1).

5. 5 A CONFLUENCE OF RELATED STRANDS Monitoring And Evaluation Thematic Concentration Binding Contracts Conditionality Better Information Better Evaluation/ CFA Performance/ Results measuring outcomes

6. 6 provides goals provides themes regulatory requirements evaluation will inform content/progress ex ante, ongoing, ex post EMERGING NEW ARCHITECTURE 2014-20 * Possibly for Cohesion Fund, ERDF, ESF, EAFRD, EEF Europe 2020 Goals Common Strategic Framework (CSF) ?* Regulations Partnership and Investment Contracts* Operational Programmes Themes/Axes (from menu)

7. 7 STRENGTHENED CONDITIONALITY?  already is various conditionality: regulatory (inc. M+E); strategic. Infrastructure planning; institutional  types of conditionality:  “macroeconomic” – part of Stability/Growth Pact  “ex ante” – preconditions (see existing categories)  “structural reform” – structural/admin reform milestones  “performance” – Programme/EC objectives  incorporated in Partnership Contracts?  possible financial incentives/sanctions?  if implemented, could have implications for Monitoring/Evaluation

8. 8 Other Factors Monitoring and Evaluation Programming Strategy Needs Thematic Objective Intended Result Contribution - Impact Policy Allocated INPUTS Targeted OUTPUTS Actual INPUTS Achieved OUTPUTS Actual Result NEW LOGICAL FRAMEWORK  outcomes monitored/impacts evaluated  inputs/outputs “contribute” to outcomes  outcome indicators should be “responsive” to intervention Source: V. Gaffey, Acting Director, Policy Development, DG Regio, Intervention Evaluation Conference, Budapest, May 24-25, 2011.

9. 9 RE-DEFINING THE LOGIC Source: based on DG Regio, Concepts and Ideas: Monitoring and Evaluation in Practice of European Cohesion Policy 2014+, Draft, 30 March 2011 Old: InputsOutputsResults (Short/medium) IMPACTS (Long term) MonitoringEvaluation Monitoring New: InputsOutputs Results/ outcomes Evaluation Impact

10. 10 CRITERIA FOR A GOOD OUTCOME INDICATOR Reasonable: capturing the essence of an outcome according to a reasonable argument about which features of the outcome they can and cannot represent Robust: reliable, statistically and analytically validated, and, as far as practicable, complying with internationally recognised standards and methodologies; Responsive to policy: linked in as direct way as possible to the policy interventions for whose assessment they are used, while not being subject to manipulation; Normative: having a clear and accepted normative interpretation (i.e. there must be agreement that a movement in a particular direction or within a certain range is a favourable or an unfavourable result); Feasible: built, as far as practicable, on available underlying data, their measurement not imposing too large a burden on Member States, on enterprises, nor on the citizens; Debatable: timely available to a wide public, with room being built for public debate and for their own revision when needed and motivated. Source: F. Barca, P. McCann, Outcome Indicators and Targets – Towards a Performance Oriented EC Cohesion Policy, High-level Group on Future Cohesion Policy, Meeting No. 8, 15 Feb. 2011 RRR-NFD instead of SMART! The challenge is in the “Third R”

11. 11 THE RESPONSIVENESS ISSUE: SUB-CRITERIA FOR OUTCOME INDICATORS?  needs to be specifically related to the output?  easier if OP is the only source of funding (national + EU)?  needs to be about the beneficiaries?  easier if there is a clear physical link, e.g. infrastructure  needs a typology of intervention types* *i.e. infrastructure, subsidies, service provision. Use Pilots to develop this

12. 12 CHALLENGES REGARDING OUTCOME/RESULT INDICATORS Conceptual:  the causation problem remains?  a lot of judgement? Communication:  can we be confident this will address the issue?  trade-off between communication and robustness Capacity:  who is going to define the indicators? A specialist task. Will need forensic precision Time-lags:  when will attributable outcomes arise? Conditionality:  M+E insufficiently robust to support binding conditionality

13. 13 SOME SUGGESTIONS FOR COMMISSION/MEMBER STATES  skills, capacity, technical assistance, technical support  M+E as communication tools? Simplify language? Consistent terminology, e.g. “outcomes” v “results”  revisit institutional context, inc. Monitoring Committees, Managing Authority  Nature of Commission Guidelines and technical support  use of regular Peer Review Groups, (“boots on the ground”) ongoing Evaluators  merge Monitoring and Evaluation function in Member States  greater role for Eurostat/National Statistical Offices (but avoid pure context indicators)  nature of Multi-annual Evaluation plans

14. 14 And finally…  keep the Regulations general and Guidelines detailed, specific (not vice versa)  don’t rush the Guidelines THANK YOU.