Ex-ante Evaluation of the EU Structural Funds: the New Challenges of 2014–2020 Haroldas Brožaitis, PPMI (Lithuania) International Evaluation Conference Cohesion Policy : Towards Evidence Based Programming and Evaluation 4-5 July 2013, Vilnius

Outline Role of the theory of change in programming for Our approach to it in ex ante evaluation in Lithuania Challenges experienced in applying this approach in practice Expected and actual benefits of doing that

Previous experience in programming OPs often very extensive in descriptions, but not built on a clear intervention logic: broad socio-economic needs and objectives are mentioned, but not clear indication as to what change is expected, even more so how it will be achieved and in what timeframe. Variety of reasons: some European or national priorities are so overriding that investment in them is perceived as self-explanatory, no need for further evidence search for flexibility, avoiding reprogramming path dependency and failure to question the past experience incorrect balance with political priorities ?

EC guidance for programming in identify the key problems to be addressed (multitude of real or perceived needs); define the direction of the desired change, the desired situation that should be achieved (target); analyse different factors that can drive the intended result towards or away from the desired change (evidence); propose how to measure the achievements (monitoring and evaluation) = develop theory of change for OP and its key interventions

From logic model to the theory of change Logic model Theory of change Logframe matrix Descriptory (what) Weak causal relations No assumptions and risks Descriptory (what) Weak causal relations No assumptions and risks Explanatory (how, why) Clearer rationale for intervention Includes assumptions, external factors, unexpected effects Explanatory (how, why) Clearer rationale for intervention Includes assumptions, external factors, unexpected effects

Logic model

Key components of Theory of Change Resources and actions Unexpected effects Outputs Rationales: why outputs/outcomes are needed to bring about the change Results / Immediate outcomes Longer-term outcomes Ex-ante conditionalities Assumptions External factors (socio- economic context, regulation, other interventions)

Key components of Theory of Change (2) Sequence: start with long-term outcomes, work backward toward initial activities, and then map required/existing resources Two levels of theory: OP explains – programme theory: how the expected change will contribute to both the domestic and European objectives; – implementation theory: how the planned activities will lead to the expected results.

Three attributes of a good theory of change Plausible: evidence and common sense suggest that the activities, if implemented, will lead to desired outcomes Doable: the economic, technical, political, institutional, and human resources will be available to carry out the initiative Testable: the theory of change is specific and complete enough for an evaluator to assess it and/or track its progress in credible and useful ways

Role of ex ante evaluators Evidence-based, participatory (stakeholder involvement) assessment of 1. Validity of the addressed development issue (perceived v. real): evidence 2. Dimension(s) of the development issue is/are being targeted: disaggregation 3. External factors that may influence the outcomes: alternative interventions? 4. Key assumptions (political, economic, social) for the impact to occur; the risks of the assumptions not being met; and 5. Financial and human resources, admin capacity: likelihood of success 6. Risks of negative or unexpected outcomes, their mitigation: wider view, form of assistance 7. Time horizon for the expected outputs and outcomes 8. SMARTness of indicators

Key challenges in schools: Costs of education per pupil varies up to 5 times depending on type of school Average number of pupils per school: 2005 m , 2012 m Due to demographic change, the number of pupils until 2015 will further decrease by 8% Total Secondary schools Primary schools

Achievements in math Achievements in science Achievements in reading

 Early school leaving is 3 times more intensive in rural areas, and  2 times more likely among boys  Lack of preventive measures, individualised approach to education and assistance Key challenges in schools: Early school leavers in EU, 2011

Example of our approach

Regulatory (EU) challenges Strategic cap: thematic objectives, investment priorities + focus on attribution and change - fragmentation ERDF v. ESF: requirement to separate but also different approach to measuring outcomes Limitation of space in OP template [500 symbols to explain selection of certain investment priority, 2000 – to describe baseline situation and results to be achieved with ESI support, 7000 – to explain actions to be supported, including their link to specific objective, target groups, territories and beneficiaries] Limited number of specific objectives and (especially) result indicators (1- 2) [while (in theory) the measurement in a theory of change should be quite extensive with multiple measures of outcomes and activities multiple levels]

Contextual (domestic) challenges Time pressure (sufficient time is needed to analyse the programme theory and to refine it in interaction with programme developers) due to – delayed negotiations at EU level – time spent to develop other national strategic documents (National Development Programme ) – EU presidency Lack of evidence (e.g., evaluations of variety of pilot interventions in )

Implementation challenges How many theories of change? (one per OP, TO or IP) Reluctance to commit to socioeconomic outcomes due to – smaller relative importance of structural funds in driving change and the probability that they will make no measurable difference (intervention v. regulation); – technical difficulty to follow the effects of the programme because of insufficient statistical systems; – varying capacity of stakeholders (and evaluators) to identify, prioritise, and then assess the key activities and contextual factors in advance.

Implementation challenges (2) When developing theory of change, identifying and agreeing upon long-term outcomes is easy (broad -> uncontroversial); identifying (possible) actions is relatively straightforward (delaying issues for further phases); but intermediate outcomes are difficult to specify and link to longer-term outcomes because – scientific knowledge/other evidence about links between actions, intermediate outcomes, and long-term outcomes is not well developed (e.g., what improves education attainment levels in schools); – this can be a politically sensitive process, especially if those outcomes might imply major resource reallocation or power shifts (e.g., optimisation of schools network or deinstitutionalisation of social services provision). Also, reconciling multiple theories change among the stakeholders

Expected and actual benefits Planning and implementation - Sharpened and refined planning with clearer and more explicit programmes theories through: constructive critique of common sense assumptions such as “training increases qualification“ or “grants increase innovation”; challenging them by proposing alternative theories based on past evaluations or research; "triangulating" the estimation of possible outcomes through different approaches (e.g., econometric modelling) E.g.: March 2013: 49 IPs and 174 SOs  June 2013: 36 IPs and 55 SOs - Managing expectations amongst stakeholders because the resource required to achieve goals is more transparent

Expected and actual benefits Monitoring and evaluation Arguing that the intervention is likely to make a difference identifies weaknesses in the argument (= identification where evidence for strengthening such claims is most needed) Monitoring: facilitating the measurement and data collection – ONGOING Evaluation: improved focus on assessing outcomes and guiding choices as to when and how to evaluate – EXTENSIVE EVALUATION PLAN (with some 63 initiatives) Research: exposing gaps in existing knowledge and laying out a research agenda in key related fields – IDEAS FOR APPLIED RESEARCH