A review of the Value for Money assessment of Science Foundation Ireland Ali Basak Richard O'Carroll MSc EPS Year 2 EC8011 – Assignment 2

Background Science Foundation Ireland (SFI) invests in academic researchers and research teams who are most likely to generate new knowledge, leading edge technologies and competitive enterprises in the fields of science and engineering underpinning three broad areas: Biotechnology, Information and communications technology, Sustainable energy and energy-efficient technologies Activities previously reviewed by Brook in 2005 This evaluation completed by Indecon in 2008

Assessment takes place in context of the Government’s Value for Money and Policy Review Initiative and looks at the SFI’s programmes under three headings: Terms of reference Objectives Impacts Complentarities/ Linkages 1.Examine validity of programmes in meeting the longterm objectives of SPI to create highly visible critical mass of word-class rearch excellence 2.Examine validity of programmes in context of changing circumstances and overall consistency with overall objectives of the SSTI, other relevant policies and EU strategies in the area 3.Determine probable outcomes that could result from discontinuation, reduction or expansion of any programme 4.Review individual objectives of each programme and assess extent to which they are being met 5.Examine separate programmes in terms of comprehensive measurement of inputs and outputs 1.Review effectiveness of operation of individual programmes in terms of outputs and impacts where possible. Also quality of programmes and possible alternatives to them.Review effectiveness of operation of individual programmes in terms of outputs and impacts where possible. Also quality of programmes and possible alternatives to them. 2.Review user satisfaction with management and implementation of programmes by SFIReview user satisfaction with management and implementation of programmes by SFI 1.Review effectiveness of operation ofReview effectiveness of operation of individual programmes in terms of outputs and impacts where possible. Also quality of programmes and possible alternatives to them 2. Review user satisfaction with management and implementation of programmes by SFI

Methodology

Evaluation of programme outputs Review of research outputs Review of Collaboration Activity Review of Commercialisation Activity Outputs of journal-based peer review articles published by SFI funded researchers Outputs in relation to international conference presentations or papers delivered by SFI-funded researchers International collaborations Industry collaborations Views of industry on SFI objectives Data considered: SFI commitments & expenditure; Research activities & outputs; Consultation of stakeholders/experts & surveys

Programme effectiveness & impact Assessment of wider economic impacts Programme Management & Monitoring International Comparative Position of SFI Assessment of research outputs & quality Early citation rates “International” quality of outputs based on appearance in peer-reviewed journals “Productivity" of researchers

Programme effectiveness & impact Assessment of wider economic impacts Programme Management & Monitoring International Comparative Position of SFI Assessment of research outputs & quality Economic benefits of investment in human capital Inward Investment effects – based on discussions with the IDA and a reference to three partnerships between multinationals and Irish third level colleges Commercialisation i.e. actual revenues

Programme effectiveness & impact Assessment of wider economic impacts Programme Management & Monitoring International Comparative Position of SFI Assessment of research outputs & quality Discussion of peer review process User satisfaction with management and implementation based on an extensive survey – note survey response ratios Discussion of approaches to monitoring

Programme effectiveness & impact Assessment of wider economic impacts Programme Management & Monitoring International Comparative Position of SFI Assessment of research outputs & quality Based on discussions with a list of international academics

The concept of “virtual citations”

Conclusions The report concludes that a discontinuation of the programmes would adversely affect the buildup of research capacity It observes that SFI has unusually high staff turnover It points out room for improvement in interaction with other agencies, in particular the HEA It notes that 1.4 billion Euro spent so far and that VFM and financial sustainability will be of increasing importance

Recommendations

Issues Some independent or disinterested input would be useful. Maybe some comparisons with international or European benchmarks for countries of similar size would be an idea. But sure that’d be just mad, Ted. There has been an increase in PhDs but there is no clarity about the counterfactual or control methods. No return on investment can be expected in the short run. This is assessed by recipients of SFI funding only. No serious international bench-marking. Only comparator is Irish programmes not funded by SFI. “Virtual impact” particularly worrying because of this. Methodology merely repeats the objectives so not clear how objectives of review will be met or that they have been.