1. FP7 /1 EUROPEAN COMMISSION – DG Research – 12 November 2009 EVALUATION 2009 Orlando, Florida, USA 2 November 2009 Neville Reeve European Commission DG RTD Multi-actor, multi-component evaluation of the EU Framework Programmes

2. FP7 /2 European Commission Council of the European Union European Parliament European Court of Auditors Court of Justice European Union Institutions European Commission and Directorates General Secretariat General DG Research DG Information Society and Media European Commission DG Budget DG Energy & Transport DG External Relations Joint Research Centre DG Enterprise DG Agriculture & Rural dev’t DG Regional Policy DG External Relations DG Competition Managing the Framework Programme

3. FP7 /3 EU research  Framework Programme (FP) is a multi-annual research funding programme implemented at EU level  7th Framework Programme (2006-2013) - €50 billion – 27 Member States – more than 100 participating countries – tens of thousands of transnational co-funded projects

4. FP7 /4 EU Research Framework Programmes Annual Budgets between 1984 and 2013 NB: budgets in current prices. Source: Annual Report 2003, plus FP7 revised proposal

5. FP7 /5 Budgets of the EU Framework Programmes

6. FP7 /6 EU research: changing priorities

7. FP7 /7 FP7 – Indicative breakdown (€ million)

8. FP7 /8 Multi component evaluation system FP8 Ex ante Impact Assessment Annual monitoring 19982000 FP 5 FP 6 2002 200420062008 Thematic level evaluations 5 year assessment National Impact studies FP7 Ex ante Impact Assessment FP 7 FP6 ex post FP7 interim evaluation

9. FP7 /9 Evaluation Actors, Networks & Standards  Actors  Political level – Council, European Parliament  Commission level – Secretariat General  DG level – central units, operational level  Member States evaluation  Networks  DG Research Evaluation Network  European RTD Evaluation Network  Commissions internal Evaluation Network  European Commission evaluation standards

10. FP7 /10 Indicator / issueSub-indicatorMain Data Source Promotion of FP7 1.1Number of information daysAnnual NCP Survey 1.2 Number of attendees at information daysAnnual NCP Survey 1.3Commission organised meetings of NCPsDG RTD Performance of the calls 2.1Success rate (overall) by priority area and funding schemeCORDA 2.2Success rate for different types of organisation by priority area and funding scheme CORDA 2.3Success rate for different types of organisation by priority area and funding scheme & success rates per country CORDA Performance of the proposal evaluation and redress procedure 3.1Overall quality assessment of the proposal evaluators on the FP proposal evaluation process (evaluators survey) Annual Evaluators' Survey 3.2Assessment of quality by the evaluators between the FP evaluation process and other equivalent systems (evaluators survey) Annual Evaluators' Survey 3.3Time to grant agreementCORDA 3.4Percentage of experts reimbursed within the specified 45 daysDG RTD/PMO 3.5Redress cases upheld (i.e. leading to a re-evaluation) – numbers and percentagesDG RTD Quality of on-going research projects 4.1Average results of independent project review process by priority area Data from new reporting system (not existing yet for 2008) 4.2Percentage of projects by priority area covered by reviews Data from new reporting system (not existing yet for 2008) Project performance by outputs 5.1Average number of project publications per project by priority area and funding scheme Data from new reporting system (not existing yet for 2008) 5.2Average number of other forms of dissemination activities per project by priority area and funding scheme Data from new reporting system (not existing yet for 2008) 5.3Average number of different types of intellectual property protection per project by priority area and funding scheme Data from new reporting system (not existing yet for 2008) Indicators for FP Monitoring (1)

11. FP7 /11 FP activity 6.1Total number of active projects by priority areaCORDA 6.2Average financial size of projects by priority area and funding schemeCORDA 6.3Participation by types of organisation by priority area funding schemeCORDA 6.4Participation totals per countryCORDA Achieving gender equality 7.1Number of male and female coordinators in proposalsCORDA 7.2Number of male and female coordinators in projectsCORDA 7.3Gender breakdown (by seniority) of project participantsCORDA 7.4Percentage of male and female members in Advisory Groups and Programme Committees DG RTD Observing sound ethical principles in FP research 8.1Number of projects going through the review process/ % by area/ programme DG RTD 8.2Number of ethics reviews where the result showed insufficient attention had been given in proposal DG RTD 8.3Number of projects stopped as a results of the ethics reviewDG RTD 8.4Number of ethics screeningsDG RTD Performance International Cooperation activities 9.1Total numbers of participations of Third Countries by priority area and funding scheme CORDA 9.2Success of Third Countries in calls by priority area and funding schemeCORDA 9.3EC contribution to Third CountriesCORDA 9.4Number of international outgoing/incoming fellowshipsDG RTD Simplification 10.1Do stakeholders perceive that the FP is getting simpler to use in terms of financial and administrative procedures? Annual NCP Survey Indicators for FP Monitoring (2)

12. FP7 /12 Evaluation Roadmap  FP6 Ex-post evaluation in 2008  FP7 Progress report in 2009  ERC review 2009  FP7 interim evaluation in 2010  Ex ante Impact Assessment (FP8) in 2011/12  Ex evaluation FP7 in 2015

13. FP7 /13 EXPERT GROUP Supporting Experts analyses Studies at Thematic level Studies at FP level Self-assessments; interviews; consultations Impact studies by Member States Statistical analysis of FP implementation and annual Monitoring Reports Evaluation of the 6 th FP - Evidence and Analysis

14. FP7 /14 Evaluation study of FP structuring – (1)  FP6 networks are characterised by a core periphery structure dominated by a small number of closely-knit organisations.  Industry plays less of a coordinating role and is weakly embedded in the FP6 networks with the exception of IST  FP6 has resulted in more upstream than downstream outputs  Performance decreases as project size increases  New Member State participation is dominated by a few actors in Poland, Hungary and the Czech Republic.

15. FP7 /15 Evaluation study of FP structuring (2) - Network structure of the FP6 thematic areas

16. FP7 /16 Evaluation study of FP structuring (3) - Who coordinates whom in FP6 (Universities, Industry, Institutes)

17. FP7 /17 Evaluation study of FP structuring (4) - Who coordinates whom in FP6 (New Member States)

18. FP7 /18 Using bibliometrics to assess participation (1) - Relative citation rate of lead scientists, by FP priority – year 2002, 2004, 2006

19. FP7 /19 Using bibliometrics to assess participation (2) - Lead scientists’ 2–year Relative Citation Rate (RCR), all priorities taken together

20. FP7 /20 http://ec.europa.eu/research/evaluations/

21. FP7 /21

22. FP7 /22

23. FP7 /23 EUROPEAN COMMISSION – DG Research – 2 November 2009 Dr Neville Reeve Directorate-General for Research Unit for Evaluation and Monitoring of Programmes Phone +32-2-298 93 29 Fax +32-2-295 40 82 neville.reeve@ec.europa.eu http://ec.europa.eu/research/evaluations