Evaluation and monitoring of EC-funded research and innovation in ICT Carlos OLIVEIRA DG Information Society and Media (DG INFSO C3)

2 Overview 1) INFSO-C3: Evaluation and Monitoring –Who we are 2) What we do? –Impact assessment –Evaluation –Monitoring 3) Plans for future activities –Data strategy –follow-up studies –…

3 DG INFSO–C3 Evaluation and Monitoring

4 Evaluation and monitoring Performance indicators Calls Monitoring Evaluation Impact assessment FP7 - ICT CIP - ICT/PSP Legislation Other Legislation Financial framework Strategy Policy INFOSO C3 mandate: areas of responsibility …

5 Why evaluate? Better informed decision making Better regulation and coherence with the Strategic Planning Focus Upon Results and Transparency Communication “Focus On Results: Strengthening Evaluation Of Commission Activities” SEC (2000) /07/2000 Communication “Responding to Strategic Needs: Reinforcing the use of Evaluation” SEC(2007)0213

6 Impact Assessment Prospective evaluation (ex-ante) –The rationale for new initiatives –The choice of instruments –The expected impact –The monitoring instruments Budgetary interventions: research and innovation programmes Regulatory: roaming, universal services, public sector information Other policy: ICT for energy efficiency, digitisation and digital preservation Impact assessm ent Legislation Financial framework Strategy Policy

7 Evaluation Interim and ex-post evaluation to which extent the objectives were achieved (effectiveness) efficiency, relevance, coherence, sustainability, utility and/or community added value contribution to broader strategic objectives Evaluation FP7 - ICT CIP - ICT/PSP Legislation Other

8 FP7-ICT Interim Evaluation Legal base –Financial Regulation (art 27 – Sound and Financial Management, art. 60 Deelgation) –FP7 legal base ( art 7 of Decision 1982/2006 FP7 for research, tech. development and demonstration ) Previous evaluations (“ex-post”) –FP6-IST (Aho report) –FP5-ICT (Gago report) Scope –FP7-ICT ICT Theme of Cooperation art. 171 (JTIs) and 169 (AAL) –Relationship FP7-ICT and eInfrastructures (FP7 Capacities) FP7-ICT –Overall budget: 9 billion EUR over the full duration of the FP7 ( )

9 FP7-ICT ICT theme (of Cooperation programme) Specific objectives: Improve competitiveness of European industry Enable Europe to master and shape future developments in ICT Strengthen Europe’s scientific and technology base Ensure global leadership in ICT Stimulate product, service and process innovation and creativity through ICT use Ensure that ICT progress is rapidly transformed into benefits for citizens, businesses, industry and governments Help to reduce digital divide and social exclusion

10 FP7-ICT Interim Evaluation Evaluation questions Quality of research –Are we doing the right kind of research? –Are we mobilising the best people (Europe / worldwide)? Progress towards objectives –ERA? Innovation? Knowledge triangle? –EU policy goals? Quality of implementation; simplification –WP formulation? Mix of instruments? –Rules of participation? –Trust-based relationship?

11 Sources of evidence FP7-ICT Self-assessment –Challenges (technology, societal) Survey + interviews of participants –Nature of the projects, –Perceptions and expectations of participants –Issues / barriers Specific studies –PREDICT (EU R&D in ICT sector) –Portfolio of participants –Non-retained proposals –Patents and publications –Indicators of administrative efficiency (ttc, ttp)

12 Monitoring Monitoring – formative assessment how are we progressing towards the objectives Effectiveness and efficiency of specific processes Examples: Integrated Programme Participation Analysis Publications and patents of EU-funded research in ICT Network analysis Monitoring of calls for proposals Performance indicators Calls Monitoring …

13 IPPA report Integrated Programme Portfolio Analysis –Top beneficiaries by group (HE, research, industry, SMEs) –Analysis by challenge (i.e. research area) –Geographic breakdown (inc. EU and 3rd countries) –Analysis per instrument (collaborative projects IPs and STREPs), NoEs, CSAs

14 IPPA report Integrated Programme Portfolio Analysis

15 Study Performance Indicators assess output performance of EU funded research in ICT using scientometric and bibliometric techniques –N° and relevance of publications (scientific articles, conference papers) –N° Patents –Extent of cooperation (across countries, regions, organisations) –Coverage (FP5, FP6, part of FP7) Who: KITeS Universitá Luigi Bocconi / Prof. Stefano Breschi data collection data cleaning and integration benchmarking methodology

16 Study Performance Indicators data integration: publications

17 Study Performance Indicators data collection - patents

18 What’s next … Lessons learned Systematic data collection (completeness, accuracy) Improved methodological approach A more comprehensive set of indicators …

19 Ongoing reflection data strategy for evaluation Policy goals vs operational objectives => intervention logic Top-down vs bottom-up (macro-economic sectoral analysis, portfolio project analysis, etc.) Efficiency, effectiveness, critical mass? What indicators? Causality? (eg research vs. innovation) Targeted case-studies (not success stories) Data collection / analysis / publication

20 Further output analysis Other concrete project outcomes –Specifications / reference models –Software implementations Human capital –PhD, MSc students –Participants in doctoral schools Standards –industry initiatives leveraged by FP participation Project operations –resources devoted to specific activities (management, training, dissemination) –outcome of annual reviews (eg “continuation”, suspension, “red-flag”) –Administrative efficiency (eg cost of reviews)

21 Thank you Contact: DG Information Society (and Media) INFSO-C3 Evaluation and Monitoring Unit Head of Unit: Anna Krzyzanowska Carlos Oliveira

22 Europa web site