Paolo Severati Isfol – ESF Evaluation Unit ESF Evaluation Partnership Meeting Brussels, 19 November

 Introduction: evaluation context in Italy  Production: building a favourable environment to outcome measurement and impact evaluation, many efforts and good results  Dissemination: significant efforts, but Italy can and should do better  Use: a highly critical point  Conclusions

 Experiences in evaluation of regional policies and no experiences (or, better, no demand) in evaluation of national policies  Evaluation is confined to the Structural Funds world

At regional level:  In many cases there is no clear programme theory  Sometimes outcomes are not declared and then are not measured; outputs and results are not recurrently and rigorously measured More generally:  Great attention to inputs and to the formal respect of rules and laws on the ‘correct’ use of funds  Accountability is not always and everywhere considered an inescapable obligation

 Scarce demand of evaluation  Generally the bulk of the demand of evaluation is due to exogenous pressures  Evaluation in too many cases, still today, is perceived as an obligation  Policy maker fear impact evaluation findings

 Italy is made up of 21 ESF Regional Managing Authorities (19 Regions and 2 Autonomous Provinces); 5 of them belong to Convergence Objective (Basilicata, phasing out)  Local monitoring systems are very heterogeneous and evaluation culture is not evenly developed

Outcomes measurementImpact evaluations  Where : Objective 3 (all the regions); Objective 1 (all the regions with the exception of Calabria)  When : near the mid-term evaluation and the updating of the mid term evaluation (Objective 3); at the end of the programming period (2006 and 2008) in Objective 1 Regions  By whom : in Objective 3, regional unit (regional research centres, regional agencies); in Objective 1, ISFOL (2006) and regional unit (Sicilia, Basilicata;2008) Where: Objective 3 When: mid-term evaluation and updating of mid-term evaluation By whom: ISFOL (national level); Emilia Romagna, Friuli Venezia Giulia (regional level)

 At regional level, large and consistent delay in the production of data  On going evaluation, no obligation (evaluation plans were defined at the beginning of the programming period)  No impact evaluation realised

 A national beneficiary survey conducted by ISFOL has started. The esteems will be significant (at least) at a regional level. Each region will dispose of the micro data referring to its interventions  Internal control groups will be complemented by external control groups extracted from administrative datasets coming from PES (Public Employment Services)  Impact evaluations will be realised by ISFOL where it will be possible

 A technical group coordinated by Isfol and participated by all regional authorities, devoted to the building up of common instruments, the exchange of methods and approaches to evaluation  More and better data for impact evaluations

 Evaluation reports were discussed in the Steering and Monitoring Committee and in the Evaluation Technical Groups  Paper copies of the evaluation reports had a good circulation among the insiders. Electronic versions of the same reports were freely available on the web (regional sites and UVAL-DPS site).

 Instruments used in ESF programming period still remain in the current programming period  Anyway, more efforts have to be made to reach the public opinion: 1. Press releases on the evaluation results and policy recommendations. Interviews and video-interviews to the evaluators. Short synthesis of the evaluations for citizens, public officers, project managers and social partners More publication in English: the potential reader is possibly someone who lives in other European countries

A recent, important experience  Italy was one of the four countries that cooperated with the DG EMPL (ESF Coordination division) in order to realise a study on the return on the ESF investment in the human capital  The data used for impact evaluations at a regional and national level were used by an independent evaluator for new evaluations at a European level  This experience showed to be very useful

 In the last programming period, generally evaluations were ignored. But there were some exceptions  Two examples: 1. one good example (Emilia Romagna) of effective use of the major evaluation findings unfortunately had short life 2. one bad example: Isfol recommendations (distinction of preventive and curative approach measures) contained in the mid term evaluation of CSF Obj.3 were not respected: organisational needs expressed by some relevant regions prevailed

 In the last programming period too strict rules on evaluation (too much stick) were stated  In the current programming period: too loose rules on evaluation (too much carrot) are into force

 To promote a new contractual relationship, implementation and reporting aimed at exchange of results between Member States (Regions) and the Commission  To adopt a prospective impact evaluation: designing impact evaluation in tandem with policy design. A mutual commitment join policy makers and evaluators

 A significant effort was made to design an ambitious evaluation infrastructure, usable at national and regional level  Unfortunately, monitoring and evaluation culture is not evenly diffused among the Italian regions  Large and consistent delay in the production of data at regional level  Next year new micro-data will be usable for impact evaluations

Here the item is how to involve policy makers in monitoring (surveying). In fact, at the moment:  some regions do not collect micro data on projects and beneficiaries  some others do not use micro data to start surveys No additional rules are necessary to change this embarrassing situation. It is ‘only’ needed to enforce the current ones. Incentives and sanctions to convince policy makers to be more transparent in the use of public funds

Thank you very much For contacts to :