1. Evaluating public RTD interventions: A performance audit perspective from the EU European Court of Auditors American Evaluation Society, Portland, 3 November Evaluating public RTD interventions: A performance audit perspective from the EU Hendrik Fehr, Gareth Roberts European Court of Auditors American Evaluation Society Portland, 3 November 2006

2. Evaluating public RTD interventions: A performance audit perspective from the EU European Court of Auditors American Evaluation Society, Portland, 3 November to discuss.... Background –the European Court of Auditors –the European Union budget What do we mean ‘performance audit’? Current audit: The European Commission’s system for evaluating the impact of RTD interventions –why this audit, why now? –audit methodology –issues arising to date

3. Evaluating public RTD interventions: A performance audit perspective from the EU European Court of Auditors American Evaluation Society, Portland, 3 November European Court of Auditors Independent institution – external auditor of European Union Reports sent to European Parliament and European Council of Ministers 25 Members, based in Luxembourg, organised into 5 groups, covering: –agricultural policies –structural and internal policies –external actions –own resources, banking, admin –co-ordination, quality assurance See www.eca.europa.euwww.eca.europa.eu

4. Evaluating public RTD interventions: A performance audit perspective from the EU European Court of Auditors American Evaluation Society, Portland, 3 November European Union Budget (i)

5. Evaluating public RTD interventions: A performance audit perspective from the EU European Court of Auditors American Evaluation Society, Portland, 3 November European Union Budget (ii) Research 6bn euro (approx)

6. Evaluating public RTD interventions: A performance audit perspective from the EU European Court of Auditors American Evaluation Society, Portland, 3 November Framework Programmes for RTD (i) 19941995199619971998199920002001200220032004200520062007200820092010201120122013 FP4 FP5 FP6 (avg 5bn pa) FP7 (avg 7bn pa) Run-in: developing new programme Shadow: implementation to the end

7. Evaluating public RTD interventions: A performance audit perspective from the EU European Court of Auditors American Evaluation Society, Portland, 3 November Framework Programmes for RTD (iii) directly managed by the European Commission –mainly extra-mural –multi-partner consortia (covering more than 1 EU Member State) –governed by contracts / grant agreements several instruments / types of projects –networks of excellence –integrated projects –specific targeted research projects –specific support actions –etc

8. Evaluating public RTD interventions: A performance audit perspective from the EU European Court of Auditors American Evaluation Society, Portland, 3 November What do we mean by ‘performance audit’? Court’s mandate derived from treaty –provide a statement of assurance as to the reliability of the accounts and the legality and regularity of the underlying transactions, –examine whether the financial management has been sound –submit observations on specific questions and deliver opinions Performance audit: management of research under FP5 audit of effectiveness of instruments audit of evaluation system Financial audits: annual report / opinion Opinions: Rules of Participation for FP7

9. Evaluating public RTD interventions: A performance audit perspective from the EU European Court of Auditors American Evaluation Society, Portland, 3 November What do we mean by ‘performance audit’? According to the Court’s audit manual, a performance audit is –an audit of sound financial management, namely of the economy, efficiency and effectiveness. –to put it simply, examining whether the right things are being done, and whether things are being in the right way

10. Evaluating public RTD interventions: A performance audit perspective from the EU European Court of Auditors American Evaluation Society, Portland, 3 November Overall audit question Does the European Commission have an adequate system for evaluating and assessing the impact of indirect RTD actions? Sub question 1: Are there clear relationships between the policy objectives of the framework programmes and the criteria and indicators used to select, evaluate and assess the impact of RTD projects? Sub question 2: Are evaluation activities carried out in a comprehensive, coherent and independent manner and are relevant and useful results communicated on a timely basis, and are these results taken into consideration for future policy decisions? The focus of the current audit

11. Evaluating public RTD interventions: A performance audit perspective from the EU European Court of Auditors American Evaluation Society, Portland, 3 November Why this audit (now)? because evaluation is relevant in the audit world..... –for transparency and accountability –for good governance and improving programme management –for informing policy decisions assessed as an issue of particular risk because of... –the inherent difficulty in assessing the impact of public R&D interventions given the complexity of relationships between inputs, results and impacts, and, –the increasing levels of European Community funding –the concerns expressed in the Court’s earlier work about the information available about the outputs, outcomes and impacts of Community-funded research actions to contribute to the development of an improved evaluation system for the future

12. Evaluating public RTD interventions: A performance audit perspective from the EU European Court of Auditors American Evaluation Society, Portland, 3 November the audit methodology Work focused on the Commission: analysis of existing evaluation system analysis of objectives and performance criteria analysis of evaluation reports and methodologies deployed Work aimed at understanding good practice: benchmarking visits to USA and Canada literature review stakeholder consultation within the Member States Work to validate credibility of findings expert / stakeholder focus groups contradictory procedure with the Commission

13. Evaluating public RTD interventions: A performance audit perspective from the EU European Court of Auditors American Evaluation Society, Portland, 3 November

14. Evaluating public RTD interventions: A performance audit perspective from the EU European Court of Auditors American Evaluation Society, Portland, 3 November legal and institutional framework intervention logic etc evaluation findings implementing RTD evaluation Issues arising from work to date.......

15. Evaluating public RTD interventions: A performance audit perspective from the EU European Court of Auditors American Evaluation Society, Portland, 3 November Legal and institutional framework What are the main elements of evaluation governance? and what impact do these have?, eg: Legal framework Reporting results and link with budgetary processes Internal evaluation and other standards Quality assurance of the evaluation system –Who is responsible for monitoring the functioning of the evaluation system? –What influence do they have in practice? –Is a distinction between monitoring and evaluation

16. Evaluating public RTD interventions: A performance audit perspective from the EU European Court of Auditors American Evaluation Society, Portland, 3 November Intervention logic etc Is intervention logic clear from the outset? Are objectives SMART? How are performance indicators defined and targets set? How is performance against these targets monitored?

17. Evaluating public RTD interventions: A performance audit perspective from the EU European Court of Auditors American Evaluation Society, Portland, 3 November Implementing RTD evaluation How far is programme evaluation taken as a important priority by top management? Is a robust evaluation strategy in place? If so, how well does it consider how to deal with inherent complexity explain intervention logic take account of the role of evaluation in the policy cycle strike a judicious balance between in-house capacities and external expertise cover the portfolio of issues to be evaluated, program level challenges deploy a mix of quantitative and qualitative methodologies etc Are resources adequate for evaluation needs? –human and financial resources –data collection / IT systems

18. Evaluating public RTD interventions: A performance audit perspective from the EU European Court of Auditors American Evaluation Society, Portland, 3 November Evaluation findings What kind of information does the evaluation system produce? To whom is it disseminated? Do stakeholders consider it to be useful information?

19. Evaluating public RTD interventions: A performance audit perspective from the EU European Court of Auditors American Evaluation Society, Portland, 3 November Comparing ‘performance audit’ and ‘evaluation’ Similarities: Methodologies and techniques Differences: Institutional context Evaluee relationships Reports Do you welcome such an initiative from an audit institution ?

20. Evaluating public RTD interventions: A performance audit perspective from the EU European Court of Auditors American Evaluation Society, Portland, 3 November Contact details Gareth Roberts Auditor Tel: +352 4398 45724 gareth.roberts@eca.europa.eu Hendrik Fehr Head of Division Tel: +352 4398 45503 hendrik.fehr@eca.europa.eu Internal Policies (including Research) Division European Court of Auditors 12 rue Alcide de Gasperi L-1615 Luxembourg http://www.eca.europa.eu