Stimulating Sustainable Growth Using Cohesion Policy: Realistic or a Holy Grail? John Bradley International Evaluation Conference Cohesion Policy : Towards Evidence-Based Programming and Evaluation Vilnius, July 4-5, 2013

Main issues to be explored Is the process of monitoring and evaluation losing its way in a web of complexity? Monitoring and evaluation in a world of bounded rationality Two challenging examples of evaluation A plea to move to simpler, integrated, research-based, robust evaluation approaches

Main goals of Guidance Document on Monitoring & Evaluation Emphasise a clearer understanding and organisation of monitoring and evaluation Specifically, a clearer articulation of policy objectives A better specification of differences in tasks between monitoring and evaluation

Tools of evaluation Result indicators: Variables that provide information on specific aspects of results that lend themselves to be measured Change in result indicator = Contribution of intervention + Contribution of other factors Impact: Change that can credibly be attributed to an intervention

Identifying effects of an intervention Counterfactual impact evaluation: Did the intervention have an effect at all and if yes, how big – positive or negative – was this effect. Theory-based impact evaluation: Why an intervention produces intended (and unintended) effects

What’s wrong with this? Policy Monitoring/Evaluation has become detached from Design/Implementation Use of “performance indicators” is known to distort execution of core tasks Counterfactual and Theory-based impact evaluation need to be treated as a single, integrated process CP design and impact analysis needs research- based knowledge of the structure and performance of the regional economy and/or enterprise target

The two meanings of “macro” The macro view of any economy, national or regional, takes into account all relevant aspects (e.g., production, income generation, expenditure, labour market, external interactions, etc.) and their interactions Large Operational Programmes (OPs) and the complete CP programme are referred to as “macro policies” since they have macroeconomic consequences for all aspects of the target economy

The two meanings of “micro” Microeconomic analysis focuses on a specific element of the economy (skill levels; transport costs between location A and B; social disadvantage in the labour market) Individual CP projects or small groups of projects (measures) are often referred to as “micro policies” (e.g., small scale policy actions that can be assumed to have mainly specific, localised impacts)

Macro and micro impact analysis in practice It is essential to be aware of where micro impact analysis and macro impact analysis overlap. They should not be treated separately The overlap challenge is illustrated by two recent ex-post studies a.M1 motorway project in Ireland b.International transhipment port of Gioia Tauro in Southern Italy

The Irish M1 motorway project

The M1 motorway: wider context

The impact analysis challenge: M1 The M1 motorway was only one element of a national network of motorways, inside and outside the island of Ireland The wider network benefits dwarfed benefits specific to the M1 “catchment” area CBA technique did not quantify the major network benefits Wider macroeconomic benefits were understated or ignored

The Gioia Tauro transhipment port: Regional context

The impact analysis challenge: Gioia Tauro Micro analysis and CBA techniques pointed to a low rate of return on the investment An anticipated spin-off from Gioia Tauro was development in the regional hinterland of the port This never happened. Why? Answer needed a deeper microeconomic as well as macro-structural analysis of the regional economy, which was missing from the project design and evaluation

What to do??? Development Strategy Frameworks Macroeconomic analysis is often possible at a regional level, but requires considerable effort to make maximum use of available data. Good data permit development of regional models (Polish NUTS- 2 regions), but formal modelling often difficult Business strategy frameworks should also be used to understand the region’s enterprise sector (e.g., Porter, Best) Narrative analysis and case studies can also generate insights about the regional economy actors (e.g., talk to enterprises)

Research by walking about Officially published regional data tends to be patchy or not available Commercial databases can sometimes be used (e.g., FAME for many EU states) Targeted visits to selected enterprises and regional organisations can fill in gaps But you need to know what you are looking for! Strategy frameworks act as a guide

Snapshots from the Irish border project Clothing: Hunter Apparel Solutions, Derry, NI Metal Fabrication: Walter Watson Ltd, Castlewellan, Co Down, NI

Hunters: Hi-tech uniforms

Walter Watson: Metal fabrication

Towards more integrated policy design and analysis Macro, micro and enterprise perspectives are needed in policy design and impact evaluation Over the years the first two approaches have drifted apart and do not communicate with each other; the third is ignored! An new, integrating, holistic and clarifying approach is needed

ADIMES An integrated policy design and evaluation framework Analysis: Take very seriously Design: Drive using the analysis stage Implementation: Centralise at strategic level Monitoring: Yes, but keep it simple Evaluation: Integrate into implementation Synthesis: Accumulate lessons/feedback