How to engage policy makers in evaluation How to engage policy makers in evaluation. Production, dissemination and use of impact evaluation in Italy: the ESF experience from 2000-2006 to 2007- 2013 Paolo Severati Isfol – ESF Evaluation Unit ESF Evaluation Partnership Meeting Brussels, 19 November

Outline Introduction: evaluation context in Italy Production: building a favourable environment to outcome measurement and impact evaluation, many efforts and good results Dissemination: significant efforts, but Italy can and should do better Use: a highly critical point Conclusions

Introduction: evaluation context in Italy

The Italian evaluation context (1) Experiences in evaluation of regional policies and no experiences (or, better, no demand) in evaluation of national policies Evaluation is confined to the Structural Funds world

The Italian evaluation context (2) At regional level: In many cases there is no clear programme theory Sometimes outcomes are not declared and then are not measured; outputs and results are not recurrently and rigorously measured More generally: Great attention to inputs and to the formal respect of rules and laws on the ‘correct’ use of funds Accountability is not always and everywhere considered an inescapable obligation

Demand of evaluation as the result of external stimulus rather than of an endogenous commitment Scarce demand of evaluation Generally the bulk of the demand of evaluation is due to exogenous pressures Evaluation in too many cases, still today, is perceived as an obligation Policy makers could fear impact evaluation findings Scarce demand of evaluation coming from the Programme Managing Authority, both at national and regional level Generally the bulk of the demand of evaluation is due to exogenous pressures, stemming from the European Commission instead of an endogenous initiative Evaluation in too many cases, still today, is perceived as an obligation, a formal fulfilment Policy makers could fear impact evaluation findings. As a matter of fact, rigorous impact evaluation can go any direction: positive, zero or negative impact can emerge from the analysis

Italy: 2007-2013 ESF governance Italy is made up of 21 ESF Regional Managing Authorities (19 Regions and 2 Autonomous Provinces); 5 of them belong to Convergence Objective (Basilicata, phasing out) Local monitoring systems are very heterogeneous and evaluation culture is not evenly developed

Production: building a favourable environment to outcome measurement and impact evaluation, many efforts and good results

2000-2006 Programming Period. Beneficiary surveys and impact evaluations: where, when and by whom Where: Objective 3 (all the regions); Objective 1 (all the regions with the exception of Calabria) When: near the mid-term evaluation and the updating of the mid term evaluation (Objective 3); at the end of the programming period (2006 and 2008) in Objective 1 Regions By whom: in Objective 3, regional unit (regional research centres, regional agencies); in Objective 1, ISFOL (2006) and regional unit (Sicilia, Basilicata;2008) Where: Objective 3 When: mid-term evaluation and updating of mid-term evaluation By whom: ISFOL (national level); Emilia Romagna, Friuli Venezia Giulia (regional level) Outcomes measurement Impact evaluations

2007-2013 Programming Period: the present state At regional level, large and consistent delay in the production of data On going evaluation, no obligation (evaluation plans were defined at the beginning of the programming period) No impact evaluation realised

2007-2013 Programming Period: what is going on at a national level A national beneficiary survey conducted by ISFOL has started. The esteems will be significant (at least) at a regional level. Each region will dispose of the micro data referring to its interventions Internal control groups will be complemented by external control groups extracted from administrative datasets coming from PES (Public Employment Services) Impact evaluations will be realised by ISFOL where it will be possible More and better baseline data A scientific Committee will support the design of the national beneficiary survey A Steering Committee, composed by referents of all the managing authorities really interested to impact evaluation, will be instituted

The reinforcement of the National Placement Working Group A technical group coordinated by Isfol and participated by all regional authorities, devoted to the building up of common instruments, the exchange of methods and approaches to evaluation More and better data for impact evaluations

Dissemination: significant efforts, but Italy can and should do better

Information and communication on evaluation in the last 2000-2006 ESF programming period Evaluation reports were discussed in the Steering and Monitoring Committee and in the Evaluation Technical Groups Paper copies of the evaluation reports had a good circulation among the insiders. Electronic versions of the same reports were freely available on the web (regional sites and UVAL-DPS site). Anyway evidence coming from evaluation didn’t reach the citizens and the media, either at a national or local level

Information and communication on evaluation in the current 2007-2013 ESF programming period: options to be developed Instruments used in 2000-2006 ESF programming period still remain in the current programming period Anyway, more efforts have to be made to reach the public opinion: Press releases on the evaluation results and policy recommendations. Interviews and video-interviews to the evaluators. Short synthesis of the evaluations for citizens, public officers, project managers and social partners ... More publication in English: the potential reader is possibly someone who lives in other European countries Evaluators have few occasions to share knowledge with stakeholders (social partnerships) and policy makers . On this point the past programming period was better suited

Sharing data and methods at European level: opportunities to learn from each other A recent, important experience Italy was one of the four countries that cooperated with DG EMPL (ESF Coordination division) in order to realise a study on the return on the ESF investment in the human capital The data used for impact evaluations at regional and national level were used by an independent evaluator for new evaluations at European level This experience showed to be very useful As for Italy, the exercise produced impact very similar with the findings of the evaluation at a national level. Anyway differences emerged among different countries. This experience showed to be very useful to share methods, data and more generally information. It would interesting to repeat it in the next years. Immediate effects of cooperation As a result of Eureval – Ecorys – Rambol Management study case on Emilia Romagna, the region changes its evaluation strategy. There are comfortable signals of a strong commitment and ownership

Use: a highly critical point

The use of evaluation during the last programming period In the last programming period, generally evaluations were ignored. But there were some exceptions Two examples: one good example (Emilia Romagna) of effective use of the major evaluation findings unfortunately had short life one bad example: Isfol recommendations (distinction of preventive and curative approach measures) contained in the mid term evaluation of CSF Obj.3 were not respected: organisational needs expressed by some relevant regions prevailed

Rules of engagement for evaluation: stick and carrot In the last programming period too strict rules on evaluation (too much stick) were stated In the current programming period: too loose rules on evaluation (too much carrot) are into force Use of evaluation It requires a strong commitment, self-commitment, by policy makers Managing Authorities has a large discretionary power in this particular and crucial phase. Evaluators, Member States and the European Commission can use moral suasion only A well informed public opinion, citizens interested in how public money was spent and which are curious to know if public interventions works or not can exert a real pressure on reluctant policy makers

Useful indications coming from Barca report To promote a new contractual relationship, implementation and reporting aimed at exchange of results between Member States (Regions) and the Commission To adopt a prospective impact evaluation: designing impact evaluation in tandem with policy design. A mutual commitment join policy makers and evaluators Countermeasures envisaged for the remaining years of this current programming period Well focussed impact evaluations Strict connections between demand and supply of evaluation Clearing house Different objectives for different regions: aggregate regions having common interests in evaluation Data diffusion Incentives for cooperation at a European level Communicate evaluation findings to the political level (Conferenza Stato-Regioni, Parliament)

Conclusions

Principal conclusions A significant effort was made to design an ambitious evaluation infrastructure, usable at national and regional level Unfortunately, monitoring and evaluation culture is not evenly diffused among the Italian regions Large and consistent delay in the production of data at regional level Next year new micro-data will be usable for impact evaluations

A step behind for lagging behind regions Here the item is how to involve policy makers in monitoring (surveying). In fact, at the moment: some regions do not collect micro data on projects and beneficiaries some other do not use micro data to start surveys No additional rules are necessary to change this embarrassing situation. It is ‘only’ needed to enforce the current ones. Incentives and sanctions to convince policy makers to be more transparent in the use of public funds

Thank you very much For contacts e-mail to : p.severati@isfol.it