1. Policy Research and Innovation Research and Innovation Evaluation of European Union Research Programmes Recent experiences and future challenges Evaluation 2012 Conference Minneapolis, 25 October 2012 Dr Peter Fisch European Commission - DG Research and Innovation

2. Policy Research and Innovation Research and Innovation Roadmap Introducing the EU Research Framework Programme (FP) FP Evaluation System Study on longer-term impact of the FP Horizon 2020 – the new Research and Innovation Programme 2014 – 2020 Challenges for the evaluation system

3. Policy Research and Innovation Research and Innovation Characteristics of EU Framework Programmes Project funding only True common put – no national quota Competitive process with annual calls and peer review proposal evaluation Main focus on collaborative projects with partners from several countries Open to academia and industry alike Open to the world, not restricted to Europe

4. Policy Research and Innovation Research and Innovation FP7 Structure + Ideas – Frontier Research Individual projects, bottom-up agenda Capacities – Research Capacity Sharing of infrastructures … People – Marie Curie Actions Mobility programmes for post-docs, Training courses, … Cooperation – Collaborative research Multi-partner projects, top-down agenda Other activities (EURATOM, JRC, …)

5. Policy Research and Innovation Research and Innovation FP7 Budget (in millions of €)

6. Policy Research and Innovation Research and Innovation FP7 in Key Figures 79.000 Applications 386.000 Participants in applications received 16.000 Projects funded 85.000 Participants in projects funded 38% Universities 28% Industry (17% SME) 26% Research Organisation 25.3 Billion € EU funding 169 Countries participating

7. Policy Research and Innovation Research and Innovation Global scientific collaborations

8. Policy Research and Innovation Research and Innovation FP7 Evaluation Basic Principles Systemic Approach Clear planning and timing; complete coverage Evidence based Quantitative and qualitative data Policy oriented Focus on recommendations for concrete improvements Decentralised Each activity “owns” its evaluation

9. Policy Research and Innovation Research and Innovation FP7 Evaluation Networking Across all services managing FP7 Exchange of experiences Planning Within all Commission services Methodologies Common administrative issues Network with Member States Interaction with experts and administrators Forum for national experiences Opportunity for “soft coordination”

10. Policy Research and Innovation Research and Innovation FP7 Evaluation Studies Around 20 to 30 evaluation studies per year in FP7 “Thematic Studies”, initiated by a specific activity to analyse a particular area, such as Transport Research Social Sciences & Humanities “Horizontal Studies”, initiated centrally to analyse overarching issues, such as Effects of simplification measures Long term impact of the FP

11. Policy Research and Innovation Research and Innovation FP7 Evaluation Full Programme Evaluations Political decision to carry out evaluations of the entire FP in regular intervals Kind of “Meta-Evaluation”, mainly based on available material from Monitoring and Studies Ex-Post Evaluation of FP6 (2008 – 2009) Interim Evaluation of FP7 (2010) Ex-Post Evaluation of FP7 (2015) http://ec.europa.eu/research/evaluations

12. Policy Research and Innovation Research and Innovation Long-Term Impact Study Study on longer term impact of FP Analysing impact of FP4, FP5 and FP6 "from the distance" Attempt to combine quantitative and qualitative methodologies Quantitative approach not as successful as hoped for … Carried out by Technopolis

13. Policy Research and Innovation Research and Innovation Long-Term Impact Study Combination of methods Definition Implementation Literature review Co-word analysis Interviews Intervention Logic 6 Case studies Citation network Breakthrough analysis Bibliometrics Interviews Methods Impact mechanisms Methodological Framework Conclusions Effects Typology Focus areas Findings on Impacts Critical reflections

14. Policy Research and Innovation Research and Innovation Long-Term Impact Study A Case-Based Approach to Impact 6 case studies: Quantum Information Processing and Computing Stratospheric Ozone Research Solar Photovoltaics (PV) Brain Research Automotive Industry Manufuture Technology Platform

15. Policy Research and Innovation Research and Innovation Long-Term Impact Study Key Findings Very difficult to access valid information 10 years is a long period … Long-term impact different from short- term impact Community building vs. Publications Long-term impact varies per theme / area Industry integration, discipline building, … Importance of referring back to programme objectives

16. Policy Research and Innovation Research and Innovation Long-term Impact Study Attempt for an Intervention Logic

17. Policy Research and Innovation Research and Innovation Long-Term Impact Study Conclusions Long-term perspective needed to complement usual short-term analysis Current monitoring and data tracking systems not suitable for capturing long-term effects Further efforts needed to develop methodologies further ("beyond case studies")

18. Policy Research and Innovation Research and Innovation Horizon 2020 Basics A single programme bringing together three separate programmes/initiatives (FP/CIP/EIT) Coupling research to innovation – from research to retail, all forms of innovation Focus on societal challenges facing EU society, e.g. health, clean energy and transport Simplified access, for all companies, universities, institutes in all EU countries and beyond

19. Policy Research and Innovation Research and Innovation Horizon 2020 Priority 1: Excellent science World class science is the foundation of tomorrow’s technologies, jobs and wellbeing Europe needs to develop, attract and retain research talent Researchers need access to the best infrastructures

20. Policy Research and Innovation Research and Innovation European Research Council Frontier research by the best individual teams 13 268 Future and Emerging Technologies Collaborative research to open new fields of innovation 3 100 Marie Curie actions Opportunities for training and career development 5 572 Research infrastructures (including e-infrastructure) Ensuring access to world-class facilities 2 478 Horizon 2020 Priority 1: Excellent science

21. Policy Research and Innovation Research and Innovation Horizon 2020 Priority 2: Industrial leadership Strategic investments in key technologies (e.g. advanced manufacturing, micro-electronics) underpin innovation across existing and emerging sectors Europe needs to attract more private investment in research and innovation Europe needs more innovative SMEs to create growth and jobs

22. Policy Research and Innovation Research and Innovation Leadership in enabling and industrial technologies (ICT, nanotechnologies, materials, biotechnology, manufacturing, space) 13 781 Access to risk finance Leveraging private finance and venture capital for research and innovation 3 538 Innovation in SMEs Fostering all forms of innovation in all types of SMEs 619 complemented by 6 829 (expected 15% of societal challenges + LEIT) and 'Access to risk finance' with strong SME focus Horizon 2020 Priority 2: Industrial leadership

23. Policy Research and Innovation Research and Innovation Horizon 2020 Priority 3: Societal challenges Concerns of citizens and society/EU policy objectives (climate, environment, energy, transport etc.) cannot be achieved without innovation Breakthrough solutions come from multi-disciplinary collaborations, including social sciences & humanities Promising solutions need to be tested, demonstrated and scaled up

24. Policy Research and Innovation Research and Innovation Health, demographic change and wellbeing8 033 Food security, sustainable agriculture, marine and maritime research & the bioeconomy 4 152 Secure, clean and efficient energy5 782 Smart, green and integrated transport6 802 Climate action, resource efficiency and raw materials3 160 Inclusive, innovative and secure societies3 819 Horizon 2020 Priority 3: Societal challenges

25. Policy Research and Innovation Research and Innovation Horizon 2020 Novelties Integration of the innovation dimension Additional emphasis on Public-Private-Partnerships and Public-Public-Partnerships New approaches to SME support Massive increase in financial instruments (loan guarantees for Mid-Caps) Even greater variety of governance modes (Knowledge and Innovation Communities for the European Institute for Technology and Innovation)

26. Policy Research and Innovation Research and Innovation Evaluation System Legal documents Explicit intervention logic Objectives specified  General objectives  Specific objectives  Operational objectives Key indicators  Specified with benchmark figures

27. Policy Research and Innovation Research and Innovation Evaluation System Hierarchy of objectives

28. Policy Research and Innovation Research and Innovation Evaluation System Work in progress Internal Working Group Community building Complementing objectives and indicators Gathering and handling of data Commission Staff Working Document Spring 2013 Key elements of Horizon 2020 evaluation strategy Timetable

29. Policy Research and Innovation Research and Innovation Evaluation Challenges Objectives and Indicators Objectives and indicators in the legal documents provide the "back bone" for the evaluation system Need to complement these by more specific objectives and indicators Old problem of the "missing middle" Preparing valid indicators in view of Horizon 2020 Ex- Post Evaluation in … 2023

30. Policy Research and Innovation Research and Innovation Evaluation Challenges Economic and societal impact High political emphasis on providing information on economic and societal impact Difficulties to trace back research and innovation effects Most commonly used indicators "stop too early" Little attention paid on human resource indicators (Training, Career tracking)

31. Policy Research and Innovation Research and Innovation Evaluation Challenges Longer term footprint Recent study highlighted difference in focus between short term and long term analysis Need to prepare the ground for adequate capturing of longer term effects Difficulty to define adequate proxies for phenomena like Community building New disciplines …

32. Policy Research and Innovation Research and Innovation Evaluation Challenges Governance and coherence Horizon 2020 implemented through (currently) 134 activity lines Managed by several DGs, executive agencies, and numerous other actors (Joint Undertakings, …) Huge variety of actions Small scale to global challenge Top-down vs. Bottom-up Projects, Networks, Careers, Risk Capital, Infrastructures, Mobility, …

33. Policy Research and Innovation Research and Innovation Contact Dr. Peter Fisch Head of Unit “Ex-post Evaluation and Reporting” European Commission – DG RTD A.6 SDME 2/41 1049 Brussels / BELGIUM peter.fisch@ec.europa.eu http://ec.europa.eu/research/evaluations