Ex-ante Evaluation of the EU Structural Funds: the New Challenges of 2014–2020 Haroldas Brožaitis, PPMI (Lithuania) International Evaluation Conference.

Slides:



Advertisements
Similar presentations
Theory-Based Evaluation:
Advertisements

Performance Framework
Planning and Timely Implementation of Structural Funds Interventions Katarína Mathernová Director, DG Regional Policy European Commission 24 November 2005.
Intelligence Step 5 - Capacity Analysis Capacity Analysis Without capacity, the most innovative and brilliant interventions will not be implemented, wont.
PROJECT CYCLE MANAGEMENT (PCM)
Good Evaluation Planning – and why this matters Presentation by Elliot Stern to Evaluation Network Meeting January 16 th 2015.
Program Evaluation. Lecture Overview  Program evaluation and program development  Logic of program evaluation (Program theory)  Four-Step Model  Comprehensive.
1 Jela Tvrdonova,  Strategic approach to rural development  Common approach to evaluation: legal background and CMEF  Monitoring and evaluation.
Ray C. Rist The World Bank Washington, D.C.
Results-Based Management: Logical Framework Approach
Health Aspect of Disaster Risk Assessment Dr AA Abubakar Department of Community Medicine Ahmadu Bello University Zaria Nigeria.
Association for the Education of Adults EAEA European AE Research – Look towards the future ERDI General Assembly, 2004.
Evidence Based Cohesion Policy Focus on performance incentives Thomas Tandskov Dissing Senior Adviser Ministry of Economics and Business Affairs Danish.
Health Systems and the Cycle of Health System Reform
Opportunities for interregional cooperation in Regional Operational Programmes: Article 96.3.d) of the Common Provisions Regulation INTERACT Future Pilot.
Results-Based Management
Evaluation methods and tools (Focus on delivery mechanism) Jela Tvrdonova, 2014.
 Summary Presentation of Haiti  Norway’s Evaluation: Basic Information  Challenges Leading to Policy Level Findings  Lessons from the Norwegian Portfolio.
Project “Ex-ante evaluation of programming documents and strengthening evaluation capacity for EU funds post-accession” (EUROPAID/130401/D/SER/HR) Project.
Project design & Planning The Logical Framework Approach An Over View Icelandic International Development Agency (ICEIDA) Iceland United Nations University.
KEYWORDS REFRESHMENT. Activities: in the context of the Logframe Matrix, these are the actions (tasks) that have to be taken to produce results Analysis.
ESPON Seminar 15 November 2006 in Espoo, Finland Review of the ESPON 2006 and lessons learned for the ESPON 2013 Programme Thiemo W. Eser, ESPON Managing.
Logic Models and Theory of Change Models: Defining and Telling Apart
1 European Union Regional Policy – Employment, Social Affairs and Inclusion The new architecture for cohesion policy post-2013 High-Level Meeting on the.
Better Evidence for a Better Start the social research unit at dartington.
TEN-T Experts Briefing, March Annual Call Award Criteria.
Regional Policy EU Cohesion Policy 2014 – 2020 Proposals from the European Commission.
4/5 June 2009Challenges of the CMEF & Ongoing Evaluation 1 Common monitoring and evaluation framework Jela Tvrdonova, 2010.
Project Management Learning Program 7-18 May 2012, Mekong Institute, Khon Kaen, Thailand Writing Project Report Multi-Purpose Reporting.
Approach to GEF IW SCS Impact Evaluation Aaron Zazueta Reference Group Meeting Bangkok, Thailand September 27, 2010.
Regional Policy Veronica Gaffey Evaluation Unit DG Regional Policy International Monitoring Conference Budapest 11 th November 2011 Budapest 26 th September2013.
Screen 1 of 22 Food Security Policies – Formulation and Implementation Policy Monitoring and Evaluation LEARNING OBJECTIVES Define the purpose of a monitoring.
1 Monitoring & evaluation 2013+: concepts and ideas (ERDF & CF) CMEF meeting, 17 th June 2011, Kai Stryczynski, DG REGIO Evaluation Unit.
Regional Policy Result Orientation of future ETC Programes Veronica Gaffey Head of Evaluation & European Semester 23 April 2013.
MONITORING SYSTEM OF EU STRUCTURAL FUNDS: PHYSICAL INDICATORS International Conference for New Member States February 1-2, 2012, Vilnius (Lithuania) European.
Regional Policy How are evaluations used in the EU? How to make them more usable? Stockholm, 8 October 2015 Kai Stryczynski, DG Regional and Urban Policy.
Project Management Learning Program 23 Mar – 3 Aprl 2009, Mekong Institute, Khon Kaen, Thailand Managing for Development Results Results-Oriented Monitoring.
Rome, july 5, 2006 Observing project implementation and conducting project analysis (UVER) Presentation by Luigi Guerci.
Community Planning 101 Disability Preparedness Summit Nebraska Volunteer Service Commission Laurie Barger Sutter November 5, 2007.
Consultant Advance Research Team. Outline UNDERSTANDING M&E DATA NEEDS PEOPLE, PARTNERSHIP AND PLANNING 1.Organizational structures with HIV M&E functions.
Developing a Framework In Support of a Community of Practice in ABI Jason Newberry, Research Director Tanya Darisi, Senior Researcher
PREPARING FOR MONITORING AND EVALUATION 27 – 31 May 2013 Bangkok Bangkok Office Asia and Pacific Regional Bureau for Education United Nations Educational,
Advanced Engineering Projects Management Dr. Nabil I El Sawalhi Associate Professor of Construction Management 1AEPM 4.
Logical Framework Slide 1 Mekong Institute & UNESCO Regional Office-Bangkok 23 February – 6 March 2009; Khon Kaen, Thailand Prepared by the Education Policy.
Evaluation of NRNs Andreas Resch, Evaluation Advisor.
REGIONAL POLICY EUROPEAN COMMISSION The contribution of EU Regional/Cohesion programmes Corinne Hermant-de Callataÿ European Commission,
Grant Application Form (Annex A) Grant Application Form (Annex A) 2nd Call for Proposals.
Proposal Development Guidelines for Signature Grantee Semi-Finalists The Covenant Foundation.
"The challenge for Territorial Cohesion 2014 – 2020: delivering results for EU citizens" Veronica Gaffey Acting Director EUROPEAN COMMISSION, DG for Regional.
Project Evaluation for MSP Targeted Partnerships Eric R. Banilower Horizon Research, Inc.
4/5 June 2009Challenges of the CMEF & Ongoing Evaluation Common monitoring and evaluation framework for evaluation of rural development programs.
Development of Gender Sensitive M&E: Tools and Strategies.
Croatia: Result orientation within the process of preparation of programming documents V4+ Croatia and Slovenia Expert Level Conference Budapest,
Interreg Programmes Preliminary Conclusions May 2016.
Building an ENI CBC project
Monitoring and Evaluating Rural Advisory Services
PRIORITIES FOR THE FUTURE
Evaluation : goals and principles
Veronica Gaffey & Antonella Schulte-Braucks
HUMAN RESOURCE GOVERNANCE, RISK MANAGEMENT AND COMPLIANCE
Tracking development results at the EIB
Presentation ESF performance report AIR 2016 ESF Technical Working Group 9 February 2018 Brussels Costanza Pagnini.
Logic Models and Theory of Change Models: Defining and Telling Apart
Enhancing institutional capacity and an efficient public administration (TO 11) - state of play in the framework of the negotiations     Florian HAUSER,
CATHCA National Conference 2018
Kamil Valica Unit A.3 Impact Assessment and Evaluation
Regulated Health Professions Network Evaluation Framework
ESF monitoring and evaluation in Draft guidance
Project intervention logic
Notes: Rapid assessments.
Presentation transcript:

Ex-ante Evaluation of the EU Structural Funds: the New Challenges of 2014–2020 Haroldas Brožaitis, PPMI (Lithuania) International Evaluation Conference Cohesion Policy : Towards Evidence Based Programming and Evaluation 4-5 July 2013, Vilnius

Outline Role of the theory of change in programming for Our approach to it in ex ante evaluation in Lithuania Challenges experienced in applying this approach in practice Expected and actual benefits of doing that

Previous experience in programming OPs often very extensive in descriptions, but not built on a clear intervention logic: broad socio-economic needs and objectives are mentioned, but not clear indication as to what change is expected, even more so how it will be achieved and in what timeframe. Variety of reasons: some European or national priorities are so overriding that investment in them is perceived as self-explanatory, no need for further evidence search for flexibility, avoiding reprogramming path dependency and failure to question the past experience incorrect balance with political priorities ?

EC guidance for programming in identify the key problems to be addressed (multitude of real or perceived needs); define the direction of the desired change, the desired situation that should be achieved (target); analyse different factors that can drive the intended result towards or away from the desired change (evidence); propose how to measure the achievements (monitoring and evaluation) = develop theory of change for OP and its key interventions

From logic model to the theory of change Logic model Theory of change Logframe matrix Descriptory (what) Weak causal relations No assumptions and risks Descriptory (what) Weak causal relations No assumptions and risks Explanatory (how, why) Clearer rationale for intervention Includes assumptions, external factors, unexpected effects Explanatory (how, why) Clearer rationale for intervention Includes assumptions, external factors, unexpected effects

Logic model

Key components of Theory of Change Resources and actions Unexpected effects Outputs Rationales: why outputs/outcomes are needed to bring about the change Results / Immediate outcomes Longer-term outcomes Ex-ante conditionalities Assumptions External factors (socio- economic context, regulation, other interventions)

Key components of Theory of Change (2) Sequence: start with long-term outcomes, work backward toward initial activities, and then map required/existing resources Two levels of theory: OP explains – programme theory: how the expected change will contribute to both the domestic and European objectives; – implementation theory: how the planned activities will lead to the expected results.

Three attributes of a good theory of change Plausible: evidence and common sense suggest that the activities, if implemented, will lead to desired outcomes Doable: the economic, technical, political, institutional, and human resources will be available to carry out the initiative Testable: the theory of change is specific and complete enough for an evaluator to assess it and/or track its progress in credible and useful ways

Role of ex ante evaluators Evidence-based, participatory (stakeholder involvement) assessment of 1. Validity of the addressed development issue (perceived v. real): evidence 2. Dimension(s) of the development issue is/are being targeted: disaggregation 3. External factors that may influence the outcomes: alternative interventions? 4. Key assumptions (political, economic, social) for the impact to occur; the risks of the assumptions not being met; and 5. Financial and human resources, admin capacity: likelihood of success 6. Risks of negative or unexpected outcomes, their mitigation: wider view, form of assistance 7. Time horizon for the expected outputs and outcomes 8. SMARTness of indicators

Key challenges in schools: Costs of education per pupil varies up to 5 times depending on type of school Average number of pupils per school: 2005 m , 2012 m Due to demographic change, the number of pupils until 2015 will further decrease by 8% Total Secondary schools Primary schools

Achievements in math Achievements in science Achievements in reading

 Early school leaving is 3 times more intensive in rural areas, and  2 times more likely among boys  Lack of preventive measures, individualised approach to education and assistance Key challenges in schools: Early school leavers in EU, 2011

Example of our approach

Regulatory (EU) challenges Strategic cap: thematic objectives, investment priorities + focus on attribution and change - fragmentation ERDF v. ESF: requirement to separate but also different approach to measuring outcomes Limitation of space in OP template [500 symbols to explain selection of certain investment priority, 2000 – to describe baseline situation and results to be achieved with ESI support, 7000 – to explain actions to be supported, including their link to specific objective, target groups, territories and beneficiaries] Limited number of specific objectives and (especially) result indicators (1- 2) [while (in theory) the measurement in a theory of change should be quite extensive with multiple measures of outcomes and activities multiple levels]

Contextual (domestic) challenges Time pressure (sufficient time is needed to analyse the programme theory and to refine it in interaction with programme developers) due to – delayed negotiations at EU level – time spent to develop other national strategic documents (National Development Programme ) – EU presidency Lack of evidence (e.g., evaluations of variety of pilot interventions in )

Implementation challenges How many theories of change? (one per OP, TO or IP) Reluctance to commit to socioeconomic outcomes due to – smaller relative importance of structural funds in driving change and the probability that they will make no measurable difference (intervention v. regulation); – technical difficulty to follow the effects of the programme because of insufficient statistical systems; – varying capacity of stakeholders (and evaluators) to identify, prioritise, and then assess the key activities and contextual factors in advance.

Implementation challenges (2) When developing theory of change, identifying and agreeing upon long-term outcomes is easy (broad -> uncontroversial); identifying (possible) actions is relatively straightforward (delaying issues for further phases); but intermediate outcomes are difficult to specify and link to longer-term outcomes because – scientific knowledge/other evidence about links between actions, intermediate outcomes, and long-term outcomes is not well developed (e.g., what improves education attainment levels in schools); – this can be a politically sensitive process, especially if those outcomes might imply major resource reallocation or power shifts (e.g., optimisation of schools network or deinstitutionalisation of social services provision). Also, reconciling multiple theories change among the stakeholders

Expected and actual benefits Planning and implementation - Sharpened and refined planning with clearer and more explicit programmes theories through: constructive critique of common sense assumptions such as “training increases qualification“ or “grants increase innovation”; challenging them by proposing alternative theories based on past evaluations or research; "triangulating" the estimation of possible outcomes through different approaches (e.g., econometric modelling) E.g.: March 2013: 49 IPs and 174 SOs  June 2013: 36 IPs and 55 SOs - Managing expectations amongst stakeholders because the resource required to achieve goals is more transparent

Expected and actual benefits Monitoring and evaluation Arguing that the intervention is likely to make a difference identifies weaknesses in the argument (= identification where evidence for strengthening such claims is most needed) Monitoring: facilitating the measurement and data collection – ONGOING Evaluation: improved focus on assessing outcomes and guiding choices as to when and how to evaluate – EXTENSIVE EVALUATION PLAN (with some 63 initiatives) Research: exposing gaps in existing knowledge and laying out a research agenda in key related fields – IDEAS FOR APPLIED RESEARCH