What Does Responsible Metrics Mean? Bibliometrics in Universities, 1st December 2015 Katie Evans, K.T.Evans@bath.ac.uk Research Analytics Librarian, University of Bath

Responsible metrics “The research community should develop a more sophisticated and nuanced approach to the contribution and limitations of quantitative indicators” “The problem is that evaluation is now led by the data rather than by judgement” Hicks, D., Wouters, P., de Rijcke, S., & Rafols, I., 2015. The Leiden Manifesto for research metrics. Nature. 520 (7548) 429-431 doi:10.1038/520429a Wilsdon, J., et al. (2015). The Metric Tide: Report of the Independent Review of the Role of Metrics in Research Assessment and Management. DOI: 10.13140/RG.2.1.4929.1363

What are the questions? What are the criteria? Leiden Manifesto Principles: Quantitative evaluation should support qualitative, expert assessment Measure performance against the research mission of the institution, group or researcher

What are the questions? What are the criteria? CC-BY Hakan Dahlstrom https://flic.kr/p/7qGQGP Demonstrate Discuss Analyse Iterate

Coverage & transparency Leiden Manifesto Principles Protect excellence in locally relevant research Keep data collection and analytical processes open, transparent and simple Allow those evaluated to verify data and analysis

Coverage Subject Area Coverage Overall 78% Main Panel A – Medical & Life Sciences 97%-99% Main Panel B – Science & Engineering 91% -100% Main Panel C – Social Sciences 41% - 91% Main Panel D - Humanities 17% - 60% University of Bath Submissions 48% - 99% This data comes from: HEFCE (2015). The Metric Tide: Correlation analysis of REF2014 scores and metrics (Supplementary Report II to the Independent Review of the Role of Metrics in Research Assessment and Management). HEFCE. DOI: 10.13140/RG.2.1.3362.4162 and my own calculations for Bath. Very low coverage UoAs are Panel C - Law (41%) and Panel D – Classics (17%).

Coverage

Transparency CC-BY Sharon Mollerus https://flic.kr/p/af54v Individual or small groups – provide underlying publications list for checking Larger groups – offer to share SciVal groups etc.

Interpretation Leiden Manifesto Principles Account for variation by field in publication and citation practices Avoid misplaced correctness and false precision Quantitative evaluation should support qualitative, expert assessment

Source: SCImago Visualize Genetics Medicine Computer Science UK publications 2012-13 in 30 subject categories, average citations, total citations and number of documents (bubble size) Shows variation in citation rates Source: SCImago Visualize

‘Can the REF run on metrics’ Times Higher Education 18th June 2015 Treating different disciplines fairly is one of the main challenges. But even if you have 100% coverage, disciplinary differences still matter. This graph is analysis done by Elsevier and published by the THE on the correlation between citation indicators and REF output results. It ranges from very strong (Biology, Chemistry) to not strong, even weak negative for construction engineering and anthropology. Graph from: Can the research excellence framework run on metrics? An Elsevier analysis explores the viability of a ‘smarter and cheaper’ model. Times Higher Education 18th June 2015. https://www.timeshighereducation.com/can-the-research-excellence-framework-ref-run-on-metrics/ The Metrics Tide Supplementary Report II has more sophisticated (and independent analysis) of correlation between REF output grades and various indicators, see: http://www.hefce.ac.uk/pubs/rereports/Year/2015/metrictide/Title,104463,en.html ‘Can the REF run on metrics’ Times Higher Education 18th June 2015

Supporting interpretation Get researchers using bibliometric tools Discussion Footnotes!

Absolute Subject normalised

Challenges of subject normalising

Assessing individuals Leiden Manifesto Principle Base assessment of individual researchers on a qualitative judgement of their portfolio

Policy Leiden Manifesto Principles Recognize the systematic effects of assessment and indicators Scrutinize indicators regularly and update them

“… develop a clear statement of principles on their approach to research management and assessment, including the role of quantitative indicators” Working on policy in response to Metric Tide recommendation

Not just assessment – explore! CC-BY Thomas Williams https://flic.kr/p/vBnm29