1. The Research Excellence Framework RIOJA meeting 7 July 2008 Graeme Rosenberg REF Pilot Manager

2. Research Excellence Framework (REF) The story so far: The Secretary of State asked HEFCE to develop the new framework, working with the other funding bodies and in consultation with the sector We commissioned expert advice on the use of bibliometrics and consulted on key elements of the REF 274 responses (338 UK) Consultation outcomes published on www.hefce.ac.uk/research/ref

3. Consultation outcomes In the light of the feedback we will continue to develop the REF, with two key changes: – a more unified framework: assessment for all subjects to include some combination of metrics- based indicators and input from expert panels – the timetable for designing this more flexible framework is extended by 12 months

4. Key features of the REF A unified framework for research assessment and funding which accommodates differences between disciplines Robust research quality profiles for all disciplines across the UK, benchmarked against international standards Emphasis on identifying and encouraging excellent research of all kinds Greater use of metrics than at present – including bibliometric indicators for all disciplines where these are meaningful Reduced burden on HEIs

5. Timetable Up to spring 2009 Bibliometrics pilot and other development work Spring - summer 2009 Consult on all main features of the REF including operational details of the bibliometrics process By Sept 2009Decide on the main operational features of the framework Calendar year 2010 Undertake bibliometrics exercise in appropriate subjects. Establish expert panels for all subjects. Consult on choice and use of assessment components for each subject group. 2011-12 Metrics begin to inform an element of HEFCE funding in appropriate subjects. 2013 Undertake full assessment process for all subjects including light-touch peer review.

6. Development of the REF Pilot of the bibliometrics indicator Dialogue with the sector on other key elements of the REF: – Light touch peer review – Choosing and using other metrics – Subject structure – Role of expert panels Leading to consultation in mid-2009

7. The use of bibliometrics in the REF Building on expert advice and consultation, we have identified the following key features: – Bibliometrics to be used in those disciplines in which they are meaningful - alongside other data and information – Interpretation by expert panels – To be based on citation rates per paper – not journal impact factors - and taking account of worldwide norms for the field, year of publication and document type – Results to be aggregated for substantial bodies of work; presented as a citation profile

8. The bibliometrics pilot We are now piloting the citation indicator with 22 institutions, to develop and test a number of issues: – Which disciplines? – Which staff and papers should be included? Universal or selective coverage? Are papers credited to the researcher or the institution? – How to collect data – and the implications for institutions – Which citation database(s)? – Refining the methods of analysis – including normalisation fields and handling self citation – Thresholds for the citation profile – Interpretation by expert panels

9. Bibliometrics pilot institutions Bangor UniversityLondon Sch of Hygiene and Trop Med University of BathUniversity of Nottingham University of BirminghamUniversity of Plymouth Bournemouth UniversityUniversity of Portsmouth University of CambridgeQueens University, Belfast University of DurhamRobert Gordon University University of East AngliaRoyal Veterinary College University of GlasgowUniversity of Southampton Imperial College LondonUniversity of Stirling Institute of Cancer ResearchUniversity of Sussex University of LeedsUniversity College London

10. The bibliometrics pilot - timetable May 08-Jun 08 Select HEIs/contractors Aug 08-Oct 08 Data collection Nov 08-early 09 Data analysis Spring 09 Pilot results

11. The bibliometrics pilot – data collection HEIs asked to provide as much data as available on all researchers and publications eligible for the 2008 RAE (in relevant disciplines) To be matched to the Web of Science initially, and supplemented by additional records We will evaluate issues of completeness and accuracy, and seek feedback from the institutions JISC project to document the data systems requirements

12. Some issues for institutions As we develop the REF we will assess the potential impact on the sector (accountability burden, equal opportunities and perceived behavioural incentives) Information management: – Populating databases for the initial bibliometrics exercise – Ongoing management of bibliographic data for multiple purposes Management information and internal resource allocation Relationships between citation data coverage and publication outlets

13. Further information www.hefce.ac.uk/research/ref REF-NEWS mailing list Queries to ref@hefce.ac.ukref@hefce.ac.uk