1 The best of both worlds: combining approaches Daniel Mouqué Evaluation Unit, DG REGIO

2

3 The magic bullet?

4 … More like a varied toolbox…

5 Tools of the trade include: « Quantitative » (numbers are the narrative): –Counterfactuals/comparison groups –Ex post CBA –Micro or macro models « Qualitative » (numbers feed the narrative): –Case studies –Surveys –Other « theory-based », eg rigorous observation

6 We need both approaches: Qualitative and quantitative Numbers and narrative I will illustrate with examples…

7 Examples from evaluations

8 Example 1: enterprise support in Eastern Germany (2010)

9 WP6c of the ex post evaluation R&D & modernisation grants in E. Germany 2 databases: IAB enterprise « panel », GEFRA survey of innovation in Thuringia Not just before/after but comparison with matched non-assisted enterprises (PSM, etc) Mainly quant., but results discussed with focus groups including project and prog managers

10 Results: R&D grants

11 Results: investment grants

12 Results: employment An estimated 27,000 jobs created Significantly lower than monitoring data for jobs created (107,000) and safeguarded (439,000) before/after monitoring not a guide to impact main effect of grants is investment (and productivity) change, not jobs This was not news to the focus group We would like more qualitative data to assess why this happens/applicability elsewhere

13 Example 2: enterprise support in Italy (forthcoming)

14 Grants to large firms, various support to SMEs Not just before/after but matched comparison enterprises (PSM, discontinuity design, etc) Early indication: SME support changes behaviour more. Eg 500k per job for large, 80k for SMEs In process of comparing different forms of support (grants, loans, VC, mentoring, combos) Beneficiary survey (1000 firms) shedding a lot of light on process

15 Example 3: Dutch innovation vouchers (2007)

16 The vouchers –credit note, worth max 7500 –Lottery: 100 winners, 944 losers (= control group) –for SMEs only –no match funding required –application-oriented research questions, placed with a defined group of institutes but no restrictions on level of question or technology –valid for half a year

17 Results Projects from 87% of winners, only 8% of losers, so 79% additional projects But no significant repeats/persistence For more qualitative methods: => Why no persistence? => Small effect on process innovation, no sig. effects on product inno

18 Example 4: GDP growth at regional level (Pellegrini, 2010)

19

20 Method shows clear impact But begs further questions: –What difference between regions, regional strategies? –Management efficiency and the delivery system? –Soft impact (beyond GDP, local development)? => Need for sectoral evaluations, case studies etc

21 E xample 5: ex post evaluation of URBAN 2 (2010)

22 URBAN 2? Ran from 2001 to 2006 (closed 2008) 70 programmes across EU14, 2.2 million people 754m ERDF, 1.6bn total Integrated approach to urban regeneration: physical, social, environmental Strong local development approach (local area, strong local partnership

23 Evaluation found lots of outputs 372 restoration projects (urban heritage) 2,314,000 m² of buildings converted and renovated (community centres, museums, libraries, creches) A further 557,115 m² developed for social, sports, education and health uses 3,238,000 m² of new green space

24 … and theres more 5,980 SMEs/micro/new entrepreneurs supported (incubation, business services, microfinance) 108,000 people trained, > ½ from vulnerable groups (helped to overcome illiteracy, continue education, enter labour market for 1st time) 247 projects to reduce local crime, delivered in collaboration with community groups : street wardens CCTV, landscaping and street lighting

25 Case studies, interviews etc found: Successful projects were: –« Owned » and initiated by local partners –Sustained by larger partners (and 60% of projects continue after URBAN II) In almost all cases, strong local perception of improvements due to URBAN 2 Excellent qualitative work and a great narrative, but…

26 Lack of good data blocked rigorous measurement of long term outcomes Notably in terms of: Employment/unemployment outcomes Enterprise performance Crime rates => No striking headline impact figures (And wanted to do a counterfactual, failed for lack of beneficiary – not comparison – data)

27 In conclusion…

28 The quantitative techniques bring a lot to the table… Clear and convincing headline results Ability to quantify and « stratify » Harnessing the power of statistics Provide a « reality check »

29 … but so do more qualitative techniques The power of narrative Numbers should be the starting point for discussion, not the end of it Not everything that counts can be measured, not everything that can be measured, counts (Albert Einstein)