1. ©Keith G Jeffery/euroCRIS www.eurocris.org Research Evaluation Supported by Automation Keith G Jeffery President, euroCRIS keith.jeffery@stfc.ac.uk euroCRIS Seminar 2010 1

2. ©Keith G Jeffery/euroCRIS www.eurocris.org Why Research Evaluation To protect from erroneous information / knowledge; To demonstrate value for money from public funding; To compare performance of research institutions; To compare judgement of proposal review process in different funding organisations; To promote the research outputs achieved; euroCRIS Seminar 2010 2

3. ©Keith G Jeffery/euroCRIS www.eurocris.org Kinds of Research Evaluation The Research Grants process – Evaluation of proposals submitted for funding – Evaluation of research outputs from funded projects The Research Communication process – Evaluation of scholarly publications submitted The Formal Evaluation – Evaluation of research grant income – Evaluation of research outputs (particularly publications but also products and patents) – Evaluation of contribution(s) to academic communities euroCRIS Seminar 2010 3

4. ©Keith G Jeffery/euroCRIS www.eurocris.org How Research Evaluation Commonly an intensive manual process involving – The funding organisation(s) defining a template – The research institutions struggling to locate, collate and present the information Along the way making strategic decisions on whom and what to include and within which category to place the work – The funding organisations setting up evaluation panels which effectively re-peer-review the submissions Leading to allocation of funding euroCRIS Seminar 2010 4

5. ©Keith G Jeffery/euroCRIS www.eurocris.org Cost of Research Evaluation According to PA Consulting the cost of UK RAE2008 was > £47m – A recent report for JISC in UK estimates that with automation this can be reduced by at least 25% According to PA Consulting the cost of UK submission and monitoring of research grants is £85m p.a. – A recent report for JISC estimates that with automation this can be reduced by 33% euroCRIS Seminar 2010 5

6. ©Keith G Jeffery/euroCRIS www.eurocris.org The Automation The JISC-funded EXRI study recommended CERIF for research information in UK A group representing universities, research funders, Higher education authority, research managers, charities convened and approved the recommendation of CERIF for research information in UK The major automation envisaged is CERIF for – Making returns for research evaluation – Managing the research grant process But there are additional benefits in managing research – Publicity and outreach including to innovators – Research collaborations – Finding reviewers – Strategic decision-making euroCRIS Seminar 2010 6

7. ©Keith G Jeffery/euroCRIS www.eurocris.org The message From the JISC Report: For any given institution the minimal annualised savings of owning a modern CRIS using CERIF or a CERIF wrapper can be expressed as: (X*225)+(y*3200) Where X is the number of researchers to be submitted to the REF and Y is the number of grant applications per annum to Research Councils. Using a typical institution with 1000 researchers and 1000 grant applications this gives the minimal saving as £3.4m per institution per annum Balanced against the savings is the cost of either a CERIF wrapper (£13,000) or CERIF CRIS (£10,450 - £20,880) per annum Which, frankly, is ‘lost in the noise’ of the benefit euroCRIS Seminar 2010 7

8. ©Keith G Jeffery/euroCRIS www.eurocris.org What Automation Many research institutions have a mix of systems: HR, Finance, webpages, institutional repository Some have a Research Information Management function – commonly by linking together heterogeneous systems Some have such a function by using a CRIS - and of those many have a CERIF-CRIS More organisations specify CERIF for their CRIS – ESF, ERC... Several UK universities euroCRIS Seminar 2010 8

9. ©Keith G Jeffery/euroCRIS www.eurocris.org Benefits of Automated Evaluation Research Institution – Reduced cost – Increased accuracy – Information available for other purposes Research Funder – Reduced cost – Increased accuracy – Information available for other purposes euroCRIS Seminar 2010 9

10. ©Keith G Jeffery/euroCRIS www.eurocris.org Benefits of Automated Evaluation A clear transparent, reproducible process for evaluation: – Based on open data that should already exist in a well-managed research institution – Using algorithms not human opinions Thus providing the public with assurance of research output quality and value for money from research funding euroCRIS Seminar 2010 10