1 Support to enterprise – a counterfactual approach Daniel Mouqué Evaluation Unit, DG REGIO Ex post evaluation – WP 6c.

Slides:

Advertisements

Similar presentations

Counterfactual impact evaluation

Advertisements

Theory-Based Evaluation:

1 The best of both worlds: combining approaches Daniel Mouqué Evaluation Unit, DG REGIO.

Counterfactual impact evaluation: what it can (and cannot) do for cohesion policy Alberto Martini Progetto Valutazione Torino, Italy

EN Regional Policy EUROPEAN COMMISSION Impact evaluation: some introductory words Daniel Mouqué Evaluation unit, DG REGIO Brussels, November 2008.

Impact analysis and counterfactuals in practise: the case of Structural Funds support for enterprise Gerhard Untiedt GEFRA-Münster,Germany Conference:

1 State of play Evaluations undertaken for DG REGIO.

1 Cohesion Policy JOSE LUIS CALVO DE CELIS DG REGIO EVALUATION UNIT Evaluation network meeting Brussels February 25th 2010 Ex post evaluation.

1 European Union Regional Policy – Employment, Social Affairs and Inclusion Draft guidance on monitoring and evaluation : Concepts and recommendations.

1 URBAN II – the Urban Community Initiative Daniel Mouqué Evaluation Unit, DG REGIO Ex post evaluation.

1 First steps in practice Daniel Mouqué Evaluation Unit DG REGIO.

Mywish K. Maredia Michigan State University

Improving the added value of EU Cohesion policy Professor John Bachtler European Policies Research Centre University of Strathclyde, Glasgow

Regional Policy Meta-Evaluation Some Reflections Evaluation Conference, Warsaw 12/13 November 2012 Veronica Gaffey Head of Evaluation DG for Regional.

Roma Inclusion: Monitoring and Evaluation for Results November 2010, Brussels, Belgium – DG REGIONAL POLICY Joost de Laat (PhD), Economist, Human Development.

Wider economic impacts from R&D investments Arild HervikResearch seminar

Research and Innovation Research and Innovation ODI / SME instrument ICT2013 – Vilnius Still subject to formal approval of H2020 work programmes Pierre.

VIII Evaluation Conference ‘Methodological Developments and Challenges in UK Policy Evaluation’ Daniel Fujiwara Senior Economist Cabinet Office & London.

Innovation Management Practice in Irish SME’s By: David Mulligan, Owen O’Leary, David O’Sullivan National University of Ireland, Galway.

UNECE Workshop on Confidentiality Manchester, December 2007 Comparing Fully and Partially Synthetic Data Sets for Statistical Disclosure Control.

1 Monitoring and evaluation after 2013 – some first ideas, mainly for ERDF / CF Evaluation network DG REGIO 14 th October 2010.

SAI’s role in development and use of key indicators for R&D evaluation: a quantitative example and some concluding remarks INTOSAI Working Group on Key.

Cross-national research: challenge, co-operation and compromise ESRC/NCRM TRAINING SEMINAR June 2006 Institute of Education London Susanne MacGregor.

Conference of the Upper Rhine The T rinational Metropolitan Region Upper Rhine - A laboratory for Territorial Cohesion in Europe - Dr. Joachim Beck, Euro-Institute.

EDA Tools for Your Region Mid-Columbia Economic Development District.

Beyond surveys: the research frontier moves to the use of administrative data to evaluate R&D grants Oliver Herrmann Ministry of Business, Innovation.

Restoring Rivers One Reach at a Time: Results from a Survey of U.S River Restoration Practitioners Ian McBride.

1 Cohesion Policy Ongoing Evaluation Budapest, 7 May 2010 Veronica Gaffey Head of Evaluation DG for Regional Policy European Commission.

1 Cohesion Policy Evaluation Network Meeting: Brussels, September 2009 Ex post evaluation of Cohesion policy programmes co-financed.

Regional Policy Ex post evaluation of ERDF and CF: State of play DG Regional and Urban Policy Brussels, 18 June Cohesion Policy.

Enterprise Ireland Mission Enterprise Ireland partners with entrepreneurs, Irish businesses, and the research and investment communities to develop Ireland’s.

1 Learning vs. accountability What is (are) the purpose(s) of evaluation? Alberto Martini.

Ifo Institute for Economic Research at the University of Munich Employment Effects of Innovation at the Firm Level Stefan Lachenmaier *, Horst Rottmann.

Feedback from Peer Review on 'Counterfactual Impact Evaluation' Tamara SMETANOVÁ Ministry of Labour and Social Affairs, Czech Republic Kamil VALICA DG.

Evaluation Designs Adrienne DiTommaso, MPA, CNCS Office of Research and Evaluation.

Briefing on 2009/10 report on the R&D tax incentive programme Briefing to the Portfolio Committee on Science and Technology Department of Science and Technology.

1 DG Regio work on counterfactuals Daniel Mouqué Evaluation Unit DG REGIO.

HORIZON 2020 W ORK PROGRAMME DG Research and Innovation.

Agenda: Quasi Experimental Design: Basics WSIPP drug court evaluation Outcomes and Indicators for your projects Next time: bring qualitative instrument.

Overview of evaluation of SME policy – Why and How.

Bilal Siddiqi Istanbul, May 12, 2015 Measuring Impact: Non-Experimental Methods.

The Starting Point Things to consider include Defining the overarching aim(s) of the event/activity Deciding the type of event you wish to undertake Defining.

1 Counterfactual impact evaluation: What is it, why do it? Daniel Mouqué Evaluation Unit DG REGIO.

Dr. Emmanuel Nkurunziza Director General, Rwanda Natural Resources Authority & Chief Registrar of Land Titles Rwanda’s experience in evidence-based policy.

Evaluating the benefits of using VAT data to improve the efficiency of editing in a multivariate annual business survey Daniel Lewis.

Regional Policy Ex post evaluation of ERDF and CF: State of play DG REGIO – Evaluation and European Semester Brussels, 5 November Cohesion.

Evvy Award for an excellent evaluation on socio-economic effects Terttu Luukkonen VTT Group for Technology Studies P.O.Box 1002 FIN VTT.

1 Counterfactual methods – summer school, future work Daniel Mouqué Evaluation Unit, DG REGIO.

Regression Discontinuity Design Case Study : National Evaluation of Early Reading First Peter Z. Schochet Decision Information Resources, Inc.

How do you know your product “works”? And what does it mean for a product to “work”?

Do European Social Fund labour market interventions work? Counterfactual evidence from the Czech Republic. Vladimir Kváča, Czech Ministry of Labour and.

1 Impact Evaluation in the European Commission Adam Abdulwahab Evaluation Unit, DG Regional Policy Budapest, 6 th May 2010.

The Vienna Institute for International Economic Studies ISMERI EUROPA Ex post evaluation of cohesion policy programmes Work Package 1: Coordination,

Tools and methods used to evaluate impact ICN Agency Effectiveness Workshop 11 March 2016 Gaborone, Botswana Liberty Mncube (Ph.D) Chief Economist Competition.

EVALUATIONS THAT CHALLENGE THE STATUS QUO: USE OF STATISTICAL TECHNIQUES IN MEASURING ADDITIONALITY Australasian Evaluation Society International Conference.

1 Vanderbilt University Name: Vanderbilt TAR Fellows Program Persons responsible: Thomas R. Harris, Derek Bruff, Jean Alley Time Commitment: Introductory.

TRACER STUDIES—Assessments and Evaluations

American Recovery and Reinvestment Act 2009 (787 billion) Impact on Research presented by: NJMS Research Office March 16th 2009.

STOA Workshop New Technologies and Regional Policy Marek Przeor Smart & Sustainable Growth DG Regional and Urban Policy European Commission.

Gender Equality Ex post evaluation of the ESF ( )

ESF Evaluations by MS Antonella Schulte-Braucks

Impact evaluation: Examples in the ex post

Public Support for Enterprise Innovation Policy: The Case of Poland

Counterfactual impact evaluation: Czech experience with control groups

Evaluation Network Meeting: Brussels, October 2010

ESF EVALUATION PARTNERSHIP MEETING 21 March 2014

Counterfactual impact evaluation State of play

Evaluating Impacts: An Overview of Quantitative Methods

Using evaluations to Inform recommendations for reform

Estimating net impacts of the European Social Fund in England

Presentation transcript:

1 Support to enterprise – a counterfactual approach Daniel Mouqué Evaluation Unit, DG REGIO Ex post evaluation – WP 6c

2 Enterprise & innovation support Some €79 billion in : the largest broad category of expenditure Key instrument: the grant (others include: loans/VC, advice, networking, incubators) Previously, evidence for success was mainly: –Monitoring data –Interviews on deadweight

3 The evaluation 2 grants (modernisation, R&D) in E. Germany 2 databases: IAB enterprise « panel », GEFRA survey of innovation in Thuringia Focus groups with project and prog managers 2 scientific experts

4 What is a counterfactual? Used in many methods (but often implicit or qualitative) New tool borrowed from medicine and science. Key feature: control group (think drug trial) Harnessing the power of statistics (pro: credible, con: data-heavy) Selecting matches: can range from very simple to very complex.

5 Main methods of this evaluation Controlled difference in difference (take a simple method, add regression) Propensity Score Matching (using statistics to find “twins”) Instrumental Variable (using grants to female entrepreneurs)

6 Results: investment grants

7 Results: R&D grants

8 Results: employment An estimated 27,000 jobs created Significantly lower than monitoring data for jobs created (107,000). Reconciliation: gross/net. Cannot reconcile with number of jobs safeguarded (439,000) => Strong conclusion: main effect of grants is investment (and productivity) change, not jobs

9 In conclusion Interesting results An interesting method (potentially more rigorous, gives clear headline figures) But data-heavy, so not appropriate in every case

10 Where to find the evaluation Inforegio, then the following steps: > The Policy > Impacts and results > Evaluation > Evaluations undertaken for the Commission >