1 EuroCRIS Seminar Brussels, 13 September 2010 Dr. Peter Fisch European Commission Evaluation of the Framework Programme Tools and Challenges

2 13/09/2010 Roadmap European Research Evaluation Network Mandate, Composition, Activities Monitoring FP7 Third Monitoring Report Evaluation of FP7 Interim Evaluation of FP7

3 European Research Evaluation Network Discussion forum on RTD evaluation Established in 1997 Meetings twice a year, usually in the country of EU presidency Bottom-up agenda setting

4 European Research Evaluation Network Composition EU Member States, Candidate Countries, Associated Countries Two members per country “academia”/“administration” “supply”/”demand” Nominated by National Governments Renewal in regular intervals

5 European Research Evaluation Network Activities Exchange of information on European and National initiatives Presentations on “novel” approaches Examples: Contribution to FP evaluation work Long-term impact studies Sharing of “local” experiences

6 FP7 Monitoring System  Move from “ad hoc” campaigns using external experts (FP6) towards a systematic internal monitoring (FP7)  Annual analysis based on a core set of indicators  Flexible system to develop as FP7 will become more “mature” (outputs)  Important information source for FP7 evaluations (notably the ongoing interim evaluation)

7 FP7 Monitoring Structure of 2009 Report FP7 Implementation Overview Data / indicators on key aspects FP7 Implementation Special Focus Novelties (ERC, JTI, Art 185, RSFF) Selected fields (International, Sustainable Development, Marie Curie, EURATOM) Simplification Measures taken NCP Survey Achievements Very first findings

8 FP7 Monitoring Core Indicators (1) Promotion of FP7 1.1Number of attendees at launch days 1.2Number of information days 1.3Commission organised meetings of NCPs Performance of the calls 2.1Success rate (overall) by priority area and funding scheme 2.2Success rate for different types of organisation by priority area and funding scheme 2.3Success rate for different types of organisation by priority area and funding scheme & success rates per country Performance of the proposal evaluation and redress procedures 3.1Overall quality assessment of the proposal evaluators on the FP proposal evaluation process (evaluators survey) 3.2Assessment of quality by the evaluators between the FP evaluation process and other equivalent systems (evaluators survey) 3.3Time to contract/grant 3.4Percentage of experts reimbursed within the specified 45 days 3.5Redress cases upheld (i.e. leading to a re-evaluation) – numbers and percentages Quality of on-going research projects 4.1Average results of independent project review process by priority area 4.2Percentage of projects by priority area covered by reviews

9 FP7 Monitoring Core Indicators (2) Project performance by outputs 5.1Average number of project publications per project by priority area and funding scheme 5.2Average number of other forms of dissemination activities per project by priority area and funding scheme 5.3Average number of different types of intellectual property protection per project by priority area and funding scheme FP activity 6.1Total number of active projects by priority area 6.2Average financial size of projects by priority area and funding scheme 6.3Participation by types of organisation by priority area funding scheme 6.4Participation totals per country Achieving gender equality 7.1Number of male and female coordinators in proposals 7.2Number of male and female coordinators in projects 7.3Gender breakdown (by seniority) of project participants 7.4Percentage of male and female members in Advisory Groups and Programme Committees

10 FP7 Monitoring Core Indicators (3) Observing sound ethical principles in FP research 8.1Number of projects going through the review process/ % by area/ programme 8.2Number of ethical reviews where the result showed sufficient or insufficient attention had been given 8.3Number of projects stopped as a results of the ethical review 8.4Number of screenings by services Performance of International Cooperation activities 9.1Total numbers of participations of 3rd countries by priority area and funding scheme 9.2Success rates of 3rd countries in calls by priority area and funding scheme 9.3EC contribution to 3rd countries 9.4Number of international outgoing / incoming fellowships Simplification of the FP 10.1Do stakeholders perceive that the FP is getting simpler to use in terms of financial and administrative procedures? 10.2How do stakeholders find the ease of use of the FP compared to similar international research actions and large national schemes? 10.3Are there any aspects of FP procedures which are adversely affecting to a significant extent the quality of research carried out and the quality of participation in the FP?

11 FP7 Monitoring Key Data (1) Absolute figures ( ): proposals received applicants proposals retained participants 15 billion € EU contribution

12 FP7 Monitoring Key Data (2) Organisations: Universities 30% “Industry” 25% Research Organisations 23% Gender: 20.5% female “contact persons for scientific aspects” 36.1% female “fellows” in Marie Curie actions

13 Key Data (3) 0,0%5,0% 10,0% 15,0% 20,0% 25,0% 30,0% 35,0% 40,0% 45,0%50,0% Transport Space Energy SMEs Security Fusion JTIs Nanotech Infrastructures Fission Marie Curie ERC Environment Potential Food Regions Health INCO SSH Policies Society Gene ral WOMEN IN CONTACT PERSON ROLES Contact PersonContact Person for Scientific Aspects

14 FP Evaluation System Basics Embedded in the Commission Evaluation system Evaluations to be carried out by the services responsible for an activity as part of the management responsibilities Multi-layer system consisting of thematic evaluations at programme level, studies to analyse general issues and evaluations at FP level Expert Group Report as “top of the iceberg”

15 13/09/2010 FP6 Ex-Post Evaluation Groundwork Monitoring, Project Database (CORDA) Output indicators Self assessments Thematic evaluation studies and reports Horizontal evaluation studies National Impact Assessments Feedbacks, surveys …

16 13/09/2010 FP7 Interim Evaluation Basics FP7 Decision: – Interim Evaluation “no later than” 2010 To cover FP as a whole Specific reviews in some areas – (ERC, RSFF, INFSO …) To be carried out by a group of external experts Meetings from March to September 2010 Final Report expected in October 2010

17 13/09/2010 FP7 Interim Evaluation Expert Group NameFirst NameNationalityGender ACHESONHelenaIEF ANNERBERGRolfSEMChair BEGGIainUKMRapporteur BORRÁSSusanaESF HALLÉNArvidNOM MAIMETSToivoEEM MUSTONENRiittaFIF RAFFLERHartmutDEM SWINGSJean-PierreBE/USAM YLIHONKOKristiinaFIF

18 13/09/2010 FP7 Interim Evaluation Key Questions General objectives achieved? How to improve impact of FP on ERA and other policies? FP7 role in positioning Europe on the global map? Efficiency of novel measures (ERC, JTI,...)? How to better address interdisciplinary “grand challenges”? Simplification measures effective? Progress on issues raised in FP6 evaluation?

19 Contact Dr. Peter Fisch Head of Unit “Evaluation and Monitoring of programmes” European Commission – DG Research A.3 SDME 2/ Brussels