Cognitive distance between expert panels and units of assessment A. I. M. Jakaria Rahman, Raf Guns Loet Leydesdorff and Tim C.E. Engels ECOOM day 20 November 2015

Discipline-specific research evaluation exercises -Improve the quality of scientific research -Research quality of individuals and groups (Nedeva et al., 1996; Rons, et al., 2008; Butler & McAllister, 2011; Lawrenz et al., 2012) -Research proposals (Li & Agha, 2015) Expert Panel composition method Trustworthiness of the assessment (Engels, et al. 2013); Publications are key indicators of expertise (Rybak, Balog, & Nørvåg, 2014) Barycenter method can measure and quantify (Rahman, Guns, Rousseau & Engels 2015). Bibliographic coupling (Wang and Sandström, 2015) 2

Data 9 research groups of Physics department, and Biology department Expert Panels – 6 members each – Entire career publications until the year of assessment SCI-Expanded of Web of Science indexed citable items Panel members had no prior involvement with the research groups Method The cognitive distance between entities decreases as they have more publications in the same or similar Web of Science subject categories and/or journals 3

4 Global map of Science Web of Science Subject Categories map from web-of-science data Leydesdorff, L., Carley, S., & Rafols, I. (2013). Global maps of science based on the new Web-of- Science categories. Scientometrics, 94(2), 589–593

5 Loet Leydesdorff, Ismael Rafols, and Chaomei Chen (2013). Interactive Overlays of Journals and the Measurement of Interdisciplinarity on the basis of Aggregated Journal-Journal Citations, Journal of the American Society of Information Science and Technology 64(12) Global map of Science Journal map from web-of-science data

6 Physics panel members’ publication overlay map Physics groups’ publication overlay map Global map of Science based on Web of Science Subject Categories

7 Groups publications overlay Panel publications overlay Biology Department Global map of Science based on Journals

8 Barycenter = Center of weight Here, Barycenter locations determined by a weighted average of the vertices’ coordinates (Rousseau, 1989; Jin & Rousseau, 2001) according to the number of publications in each journals.

A B N-Dimension/similarity matrix Two dimensions Barycenter calculation A B C D

J2 J1 J3 J4 J5 J6 J7 J9 J8 Panel RG-B RG-A RG-C RG-D RG-E Barycenter Example

Panel RG-B RG-A RG-C RG-E RG-D Barycenter Example d d d d d Euclidean distance

12 Barycenter overlay map of Physics panel, panel members (PM) and research groups based on Web of Science Subject Categories

13 Barycenter overlay map of Physics panel, panel members (PM) and research groups based on Journals

14

Summary Euclidean distances – can be calculated both in two and more dimensions and the results are very similar. Overlay maps visualize the position of panel and group publications Barycenter method allows to - identify the Euclidean distances between the panel and units of assessment - calculate average distances, comparison of distance and visual exploration of the barycenters on the map - identity the relevance of the expertise in a coherent way. 15

Cognitive distance between expert panels and units of assessment Thank you