BIBLIOMETRICS IN THE RAE HEA Workshop, 28/4/10 Charles Oppenheim Department of Information Science

MY CREDENTIALS Have undertaken research on the links between UK Research Assessment Exercise (RAE) results and bibliometrics since the mid 1990s (Token bibliometrician?) member of the Committee advising HEFCE on the use of bibliometrics in the forthcoming Research Excellence Framework (REF), and the pilot use of it to compare to 2008 RAE

THE REF Announced by Gordon Brown in 2006 when he was Chancellor of the Exchequer (so it is clear that the motivation was cost-cutting) Announced that it would be metrics only – details left to HEFCE et al to sort out HEFCE itself was evidently surprised by the announcement

THE PILOT Ran a vast range of citation analyses – which provided the best correlation with actual RAE results? There WAS a correlation (it would have been surprising if there had not been), but it wasn’t consistently strong for any one of the measures tried, and for different subjects, different bibliometric measures correlated best HEFCE then dropped bibliometrics as a primary tool in the REF for many subjects Gordon Brown had lost interest in the topic anyway – he has more pressing things on his mind!

WHY BIBLIOMETRICS? Civil servants clearly felt that this would provide a cheap and reliable method of evaluating research But it is backward looking only and does not evaluate future research strategy

CHEAP AND RELIABLE? I may be to blame for this (sorry….) In a series of articles published since 1997, I demonstrated the statistically significant correlation between RAE results and citation counts – and argued that citation counting could be used as a cheap and reliable substitute for expensive and subjective peer review It’s possible (I don’t know) that Treasury civil servants read my articles and were persuaded by them

IF THIS IS WHAT THE CIVIL SERVANTS DID….. …then they were being naïve I made it clear that to reliably undertake such studies, you needed subject experts to carry out the analyses manually Instead, the Treasury instructed HEFCE to go for a purely algorithmic approach

THE EVIDENCE All studies but one carried out so far have shown a statistically significant correlation between RAE scores and citation counts The one that did NOT show correlation, run by Cranfield University on a range of different subjects, used strange methodology; the usual methodology is correlation between total numbers of citations and RAE scores; Cranfield study used IF Subjects evaluated include: archaeology; business studies; genetics; library and information management; engineering; music; psychology; the REF pilot was in STM subjects only

BIBLIOMETRICS AND SSH SSH traditionally appears in reports, conference proceedings and monographs And is often focussed on national or regional studies Role of journals steadily growing WoK coverage not strong; Scopus is somewhat better; Google Books may eventually offer a significant coverage of monographs; lots of talk about developing a bibliometrics service for SSH, but no action yet

SO IS IT APPLICABLE? If you look at the correlation studies, the answer is a surprising YES! For example, music, as “soft” a subject as you’ll ever get (“background music to a Radio 4 play” was one research output offered in the last RAE), showed highly significant correlation between citation counts and RAE score REF Pilot also showed correlations, though not quite as strong

THE CORRELATIONS ARE HARDLY SURPRISING Citation counts are a measure of impact And impact is closely related to quality Nonetheless, the two concepts are not synonymous

BUT IF THE CIVIL SERVANTS WERE NAÏVE, SO ARE CRITICS OF CITATION ANALYSIS A long familiar catalogue of criticisms, aptly called “fairy tales” by Ton van Raan, head of CWTS in Leiden: –Web of Knowledge has poor coverage of the humanities, computer science, conferences, monographs….. –Poor coverage of non-English language sources –Co-authors only included post-2000 –People with the same surname and initials –Same person using different names, e.g., after marriage –And more – but all have been shown to be statistically insignificant

A KEY POINT No matter how convincing the objective arguments might be, if people don’t “buy into” the concept, there will be problems Most academics simply don’t believe citation counts are an adequate substitute for peer review And that is of course especially true in SSH

WHERE WE HAVE ENDED Civil servants were naïve to think simple citation counts would do the trick on their own Many academics are wrong in believing that citation counts cannot work in their subject area The proposed new REF will be primarily peer review, but panels will be invited to consider bibliometrics to inform and supplement peer review in certain UoAs (mainly STM) Use of citation analysis in other UoAs will require community agreement – most unlikely in arts and social sciences HEFCE will use WoK and/or Scopus; panels will have flexibility on how they will make use of the citation data Citation counts NEVER the sole method of evaluating an output HEFCE working to provide access to citation databases to HEIs to help them select outputs to submit – but the negotiations are proving difficult

BIBLIOMETRICS V PEER REVIEW Bibliometrics – all data is in the public domain, so anyone can replicate and check if they’ve been calculated correctly; numbers are “objective” Peer review – decisions taken behind closed doors Which would you prefer??

BUT…. Will a new Government scrap the REF altogether?? David Willetts has indicated a Conservative Government would do