Beyond the RAE: New methods to assess research quality July 2008

Purpose of presentation Outline the new approach to assess research quality – known as the Research Excellence Framework (the REF) Consider and discuss any outstanding questions

Context Government funds research through the ‘dual support’ system »HEFCE funds research infrastructure, salaries, blue skies research (known as ‘QR’) »Research Councils fund responsive mode projects HEFCE policy objective for research funding is selectivity »Highest quality research receives the most funding Method to identify the highest quality research has, until now, been the RAE »Most recent of which = RAE 2008 »Results of RAE2008 will be announced on 18 December 2008 »Will inform HEFCE research funding from

Background to the Research Excellence Framework Government has stated that RAE 2008 will be the last such exercise Perception that the RAE was a burden New framework for research assessment and funding will make greater use of quantitative indicators - ‘metrics’ HEFCE consulted in November 2007 on the proposed Research Excellence Framework Consultation was about the high-level process only HEFCE have announced two main changes as a result of the consultation responses: »Will take place in 2010 (not 2009 as proposed) to inform HEFCE research funding from (not as proposed) »Processes for STEM and non-STEM subjects will not be so distinct

New approach to assess research quality For all subjects, the new approach will include some combination of metrics-based indicators, including bibliometrics where appropriate, as well as input from expert panels Exact details and relative weightings to be determined – informed by results of pilot exercise Pilot exercise: commenced July 2008 College is one of 22 institutions selected for the pilot HEFCE contracted Evidence to collect and analyse data Initial databases of staff and research outputs submitted to HEFCE 8 August 2008 Next steps: »validation of data by Evidence & Pilot HEIs (August 08) »Additional records located, added to database (Aug – Sep 08) »Final database produced (31 October 08) »Citation counts established & normalised (31 December 08) »Results of pilot analysed and disseminated (31 March 09)

Bibliometric Indicator Bibliometrics will be the main quality measure for Science Supported by e.g. research income, PhD student numbers Citations will be the bibliometric indicator used Not journal impact factors or e.g. (H)irsch index Source of citation information is to be determined Thomson ISI and SCOPUS will be investigated during the pilot Outputs and citations will be fairly historical 5 – 10 year period up to c.2006 Pilot will use RAE period 01 January 01 to 31 December 2007 Self citations will be removed from citation counts Pilot will model inclusion and exclusion of self citations Citation rates will be normalised by the field average for that type and year of output

Subject Groupings Initial consultation had proposed 6 subject groupings (broadly as per RAE 2008 main panel groups) »Clinical Medicine (RAE 2008 UoAs 1-5) »Health Sciences (RAE 2008 UoAs 6-9) »Subjects Allied to Health (RAE 2008 UoAs 10-13) »Biological Sciences (RAE 2008 UoAs 14-16) »Physical Sciences (RAE 2008 UoAs 17-19) »Engineering and Computer Science (RAE 2008 UoAs 23-29) Will now need to be extended to include ‘non-science’ subjects

Allocation of outputs to subject groups Clinical Medicine Health Sciences Subjects Allied to Health Physical Sciences Engineering and Computer Science Behavioural Sciences Biology Entomology Evolutionary Biology Fisheries Horticulture Mathematical & Computational Biology Mycology Ornithology Plant Sciences Zoology Biological Sciences Oceanogr. & Marine Biol. Adv. Marine Biology European J. Phycology Microbial Ecology Journal of Phycology Aquatic Toxicology Aquatic Microbial Ecol. Freshwater Biology Fish/Shellfish Immunol. Marine Environ. Res. Canadi. J. Fish. Aq. Sci Marine Ecol.-Prog. Ser. J. N. Amer. Benth. Soc. Rev. Fish. Biol./Fisheries J. Exp. Mar. Biol./Ecol. NZ J. Marine & FW. Res. J. Marine Biol Ass. UK Diatom Res. etc Marine & Freshwater Biology Subject Groups Fields Journals Further work needed where journals fall into more than one field Separate process for specific journals which cover multiple subjects (e.g. Nature)

Issues with using citations Inadequate coverage in publication databases E.g. Thomson ISI (Leiden University, Nov 2007) High citation may not indicate good quality Need to ensure publication records and addresses are accurate and current Internal coverage percentage %60-80%40-60%<40% Mol biol & biochemAppl phys & chemMathematicsOther soc sci Biol sci: humansBiol sci: anim & plantsEconomicsHumanities & arts ChemistryPsychol & psychiatEngineering Clinical medicineGeosciences Phys & astronSoc sci in medicine

Allocation of outputs to institutions (HEIs) Many unanswered questions: How to allocate outputs to HEIs? Assign where author employed at time of publication? OR where author employed on a given (census) date? How to treat co-authors? Each output count once OR multiple times? How to handle outputs published by staff at different HEIs? Should author contribution be measured? – How? Should selection form part of the process? Select staff OR submit all? Select outputs OR include all (for selected staff?)

Possible behavioural implications New process could lead to: Publication behaviours: »Preference to publish in particular journals »Impact on other publication types (conferences, internet) or research which produces outcomes which aren’t able to be assessed under the new system »Difficulties establishing new and emerging fields »Reduction in publication volume – desire to publish a few highly cited articles »Increase in single-authored/reduction in co-authored articles (due to exclusion of self-citations) »Formation of citation clubs Staffing behaviours: »Impact on staff who are establishing their research record