1. How to engage policy makers in evaluation
How to engage policy makers in evaluation. Production, dissemination and use of impact evaluation in Italy: the ESF experience from to
Paolo Severati
Isfol – ESF Evaluation Unit
ESF Evaluation Partnership Meeting
Brussels, 19 November

2. Outline
Introduction: evaluation context in Italy
Production: building a favourable environment to outcome measurement and impact evaluation, many efforts and good results
Dissemination: significant efforts, but Italy can and should do better
Use: a highly critical point
Conclusions

3. Introduction: evaluation context in Italy

4. The Italian evaluation context (1)
Experiences in evaluation of regional policies and no experiences (or, better, no demand) in evaluation of national policies
Evaluation is confined to the Structural Funds world

5. The Italian evaluation context (2)
At regional level:
In many cases there is no clear programme theory
Sometimes outcomes are not declared and then are not measured; outputs and results are not recurrently and rigorously measured
More generally:
Great attention to inputs and to the formal respect of rules and laws on the ‘correct’ use of funds
Accountability is not always and everywhere considered an inescapable obligation

6. Demand of evaluation as the result of external stimulus rather than of an endogenous commitment
Scarce demand of evaluation
Generally the bulk of the demand of evaluation is due to exogenous pressures
Evaluation in too many cases, still today, is perceived as an obligation
Policy makers could fear impact evaluation findings
Scarce demand of evaluation coming from the Programme Managing Authority, both at national and regional level
Generally the bulk of the demand of evaluation is due to exogenous pressures, stemming from the European Commission instead of an endogenous initiative
Evaluation in too many cases, still today, is perceived as an obligation, a formal fulfilment
Policy makers could fear impact evaluation findings. As a matter of fact, rigorous impact evaluation can go any direction: positive, zero or negative impact can emerge from the analysis

7. Italy: 2007-2013 ESF governance
Italy is made up of 21 ESF Regional Managing Authorities (19 Regions and 2 Autonomous Provinces); 5 of them belong to Convergence Objective (Basilicata, phasing out)
Local monitoring systems are very heterogeneous and evaluation culture is not evenly developed

8. Production: building a favourable environment to outcome measurement and impact evaluation, many efforts and good results

9. 2000-2006 Programming Period. Beneficiary surveys and impact evaluations: where, when and by whom
Where: Objective 3 (all the regions); Objective 1 (all the regions with the exception of Calabria)
When: near the mid-term evaluation and the updating of the mid term evaluation (Objective 3); at the end of the programming period (2006 and 2008) in Objective 1 Regions
By whom: in Objective 3, regional unit (regional research centres, regional agencies); in Objective 1, ISFOL (2006) and regional unit (Sicilia, Basilicata;2008)
Where: Objective 3 When: mid-term evaluation and updating of mid-term evaluation By whom: ISFOL (national level); Emilia Romagna, Friuli Venezia Giulia (regional level)
Outcomes measurement
Impact evaluations

10. 2007-2013 Programming Period: the present state
At regional level, large and consistent delay in the production of data
On going evaluation, no obligation (evaluation plans were defined at the beginning of the programming period)
No impact evaluation realised

11. 2007-2013 Programming Period: what is going on at a national level
A national beneficiary survey conducted by ISFOL has started. The esteems will be significant (at least) at a regional level. Each region will dispose of the micro data referring to its interventions
Internal control groups will be complemented by external control groups extracted from administrative datasets coming from PES (Public Employment Services)
Impact evaluations will be realised by ISFOL where it will be possible
More and better baseline data
A scientific Committee will support the design of the national beneficiary survey
A Steering Committee, composed by referents of all the managing authorities really interested to impact evaluation, will be instituted

12. The reinforcement of the National Placement Working Group
A technical group coordinated by Isfol and participated by all regional authorities, devoted to the building up of common instruments, the exchange of methods and approaches to evaluation
More and better data for impact evaluations

13. Dissemination: significant efforts, but Italy can and should do better

14. Information and communication on evaluation in the last 2000-2006 ESF programming period
Evaluation reports were discussed in the Steering and Monitoring Committee and in the Evaluation Technical Groups
Paper copies of the evaluation reports had a good circulation among the insiders. Electronic versions of the same reports were freely available on the web (regional sites and UVAL-DPS site).
Anyway evidence coming from evaluation didn’t reach the citizens and the media, either at a national or local level

15. Information and communication on evaluation in the current ESF programming period: options to be developed
Instruments used in ESF programming period still remain in the current programming period
Anyway, more efforts have to be made to reach the public opinion:
Press releases on the evaluation results and policy recommendations. Interviews and video-interviews to the evaluators. Short synthesis of the evaluations for citizens, public officers, project managers and social partners ...
More publication in English: the potential reader is possibly someone who lives in other European countries
Evaluators have few occasions to share knowledge with stakeholders (social partnerships) and policy makers . On this point the past programming period was better suited

16. Sharing data and methods at European level: opportunities to learn from each other
A recent, important experience
Italy was one of the four countries that cooperated with DG EMPL (ESF Coordination division) in order to realise a study on the return on the ESF investment in the human capital
The data used for impact evaluations at regional and national level were used by an independent evaluator for new evaluations at European level
This experience showed to be very useful
As for Italy, the exercise produced impact very similar with the findings of the evaluation at a national level. Anyway differences emerged among different countries.
This experience showed to be very useful to share methods, data and more generally information. It would interesting to repeat it in the next years.
Immediate effects of cooperation
As a result of Eureval – Ecorys – Rambol Management study case on Emilia Romagna, the region changes its evaluation strategy. There are comfortable signals of a strong commitment and ownership

17. Use: a highly critical point

18. The use of evaluation during the last programming period
In the last programming period, generally evaluations were ignored. But there were some exceptions
Two examples:
one good example (Emilia Romagna) of effective use of the major evaluation findings unfortunately had short life
one bad example: Isfol recommendations (distinction of preventive and curative approach measures) contained in the mid term evaluation of CSF Obj.3 were not respected: organisational needs expressed by some relevant regions prevailed

19. Rules of engagement for evaluation: stick and carrot
In the last programming period too strict rules on evaluation (too much stick) were stated
In the current programming period: too loose rules on evaluation (too much carrot) are into force
Use of evaluation It requires a strong commitment, self-commitment, by policy makers
Managing Authorities has a large discretionary power in this particular and crucial phase. Evaluators, Member States and the European Commission can use moral suasion only
A well informed public opinion, citizens interested in how public money was spent and which are curious to know if public interventions works or not can exert a real pressure on reluctant policy makers

20. Useful indications coming from Barca report
To promote a new contractual relationship, implementation and reporting aimed at exchange of results between Member States (Regions) and the Commission
To adopt a prospective impact evaluation: designing impact evaluation in tandem with policy design. A mutual commitment join policy makers and evaluators
Countermeasures envisaged for the remaining years of this current programming period
Well focussed impact evaluations
Strict connections between demand and supply of evaluation
Clearing house
Different objectives for different regions: aggregate regions having common interests in evaluation
Data diffusion
Incentives for cooperation at a European level
Communicate evaluation findings to the political level (Conferenza Stato-Regioni, Parliament)

21. Conclusions

22. Principal conclusions
A significant effort was made to design an ambitious evaluation infrastructure, usable at national and regional level
Unfortunately, monitoring and evaluation culture is not evenly diffused among the Italian regions
Large and consistent delay in the production of data at regional level
Next year new micro-data will be usable for impact evaluations

23. A step behind for lagging behind regions
Here the item is how to involve policy makers in monitoring (surveying). In fact, at the moment:
some regions do not collect micro data on projects and beneficiaries
some other do not use micro data to start surveys
No additional rules are necessary to change this embarrassing situation. It is ‘only’ needed to enforce the current ones. Incentives and sanctions to convince policy makers to be more transparent in the use of public funds

24. Thank you very much
For contacts to :