A Bibliometric Comparison of the Research of Three UK Business Schools John Mingers, Kent Business School March 2014

1. Overview Introduction The Leiden Methodology Criticisms of the Leiden Methodology The Empirical Study – Three UK Business Schools Overview of publications and Citations Differences between Fields Results of the Methodology Conclusions

1.2. Introduction There has been an ever-increasing desire to monitor and measure the quality of a department’s research Apart from peer review, the main method for doing this is bibliometrics, generally based on measuring the citations received by published papers (citations per paper – CPP) However, there are major differences between the citation patterns of different fields and this needs to be taken into account in any comparisons. This means that the data must be normalised with respect to field, and time The Leiden ranking methodology (LRM) is one of the most well developed methodologies for comparing departments. It has been used mainly in the sciences where Web of Science (WoS) data is better. This study is one of the first tests with social science departments.

3. 3. The Leiden Methodology (crown indicator) Leiden methodology (crown indicator) 1.Collect all papers from a department over a specific time window, e.g., 5 years 2.Use the Web of Science to find the citations for each paper (assuming it is included in WoS) 3.Calculate how many citations such a paper would expect to receive given its field and year of publication a.WoS has field lists of all the journals in a particular field. From this we can find the total citations to papers published in that field in that year, and divide by the number of papers giving the field citations per paper (FCS) b.We can calculate a similar figure just for the set of journals the department actually publishes in (JCS) 4.Total the actual number of citations for all papers and the expected number of citations for all papers. Dividing give the crown indicator – ie the average citations per paper relative to the field average (CPP/FCS)

1. 5.The value may be : > 1 – the department is doing better than the field = 1 – it is average < 1 – it is below average Very good departments may be 2 or 3 times the average 6.Can also calculate JCS/FCS which shows if the department is targeting better or worse journals than the field as a whole 7.This methodology has been criticised for its method of calculation – should you sum the citations before dividing, or do the division for each paper and then average The first method can cause biases eg in favour of publications in fields with high citation numbers and it is now accepted that the second method is correct

4. Criticisms of the LRM 1.The use of WoS for collecting citations WoS does not include books either as targets or for their citations Until recently it has not included conference papers It does not include many journals The coverage is differential across disciplines – science is good (> 90%), social science is mediocre (30%-70%), arts and humanities is poor (20%) 2.The use of field lists from WoS which are ad hoc and not transparent 3.Bibliometric methods should not be used by themselves but only in combination with peer review and judgement 4.Are citations a good measure of quality, or just of impact?

5. The Study – Three UK Business Schools 1.Three business schools were selected, primarily because of the availability of the data but they were reasonably representative A.New school but in a very prestigious university B.New school in a traditional university undergoing rapid expansion C.Older school in modern university aiming to become more research intensive TABLE 1 RESEARCH OUTPUTS FOR THREE SCHOOLS Years covered by outputs Staff in the 2008 UK RAE Authors involved in outputs Total outputs Total journal papers Total outputs in Web of Science School A School B School C

TABLE 3 A COMPARISON OF RESEARCH OUTPUTS ACROSS SCHOOLS School ASchool BSchool C ISI WoS subject areasNumber of publications Number of journalsNumber of publications Number of journalsNumber of publications Number of journals Agriculture* Business Business Finance Computer Science* Economics Engineering Environmental Sciences Environmental Studies Ethics Food Science Technology Geography Health Care Sciences & Services Management Mathematics Applied Operations Research & Management Science Pharmacology Pharmacy Planning Development Political Science Public Administration Social Sciences* Others ( <10 Publications/ Field) Journal papers not in WoS

TABLE 4 EXPECTED CITATIONS PER PAPER BY FIELD AND PERIOD BusinessEconomicsManagement % change78%28%60%

TABLE 5 LEIDEN INDICATOR FOR THREE UK SCHOOLS FOR SEVERAL TIME PERIODS PCCPPTotal papers of journals Total cites of journals JCSmCPP/ JCSm CPP/ FCSm MNCSJCSm/ FCSm A ,32415, B , C ,1558, A ,52818, B ,45410, C ,00810, A ,71224, B ,80912, C ,97013, A ,11032, B ,66114, C ,38916, A ,64241, B ,11112, C ,11420,

6. Main Results 1.Raw (un-normalised) CPP: A is always better and is rising. B and C alternate but are also rising especially in (before the RAE) 2. JCS/FCS (quality of journal set): generally around or below 1, showing the journal set is average for the field, but it rises for A (0.99 – 1.23) showing that A is improving the quality of journals. 3.CPP/FCS (the crown indicator): all schools above 1.0 so better than the field average but not hugely (2 or 3). A actually fell before rising. Could this be because they were targeting better journals and competing against a stronger field? 4.MNCS (the alternative calculation): only marginal differences

5.Do we gain anything by the normalization? A has improved the quality of its journal set A appeared to improve over all years, but in fact fell back so this was really a field effect Would allow us to compare schools with very different subject mixes, or departments from different subjects 6.Comparison with 2008 RAE results (GPA out of 4.0): A – 3.05 B – 2.45 C – 2.50 So good agreement

7. Conclusions 1.The Leiden methodology can be applied in the social science to business schools but it requires a huge amount of data collection/ processing. It would have to be automated. 2.The results did provide a limited amount of extra value over raw scores. 3.But there are major problems in using WoS Only 20% of the outputs of the schools could actually be evaluated The WoS field categories are poorly defined 4.At the moment the LRM is NOT suitable for assessing business schools research performance. Perhaps Google Scholar could be used although that is unreliable and has no field categories.