What metrics for environments that nuture researcher development? Dr Gavin Boyce SFHEA The University of Sheffield g.boyce@sheffield.ac.uk @gavinboyce_

2015 Budget “…introduce income-contingent loans of up to £25,000 to support PhDs and research-based masters degrees. These loans will be in addition to existing funding, and designed to minimise public subsidy.” https://www.gov.uk/government/uploads/system/uploads/attachment_data/file/416330/47881_Budget_2015_Web_Accessible.pdf p.49 Need to grasp the narrative and move it from subsidy to investment. While chairing the UKCGE PG funding group, I was consistently trying to combat this narrative by responding to government press releases – welcoming the availability of funding but warning of the need to maintain investment in our research future.

SGHRM report 2014 “Doctoral education was traditionally geared towards the production of a new generation of scientists for universities and the public research system. Here a change has been taking place in recent years.” http://ec.europa.eu/euraxess/pdf/research_policies/SGHRM_IDTP_Report_Final.pdf p.3

SGHRM report 2014 “In Europe the number of doctoral candidates is increasing. It will probably continue to do so in the future, as the strategic goal of the EU to make all member states spend 3% of the GDP for research and development will, if achieved, increase the demand for well-trained researchers.” http://ec.europa.eu/euraxess/pdf/research_policies/SGHRM_IDTP_Report_Final.pdf p.3

The Impact of Doctoral Researchers Enhance profitability Increase productivity Improve reputation Develop staff Generate ideas Inspire and motivate CFE Research (November 2014) –The impact of doctoral careers : http://www.rcuk.ac.uk/RCUK-prod/assets/documents/skills/timodcfullreport.pdf Any ‘near’ metrics are merely proxies for ‘far’ metrics. The actual outcomes. We are all trying to measure the impact of our doctoral researcher alumni (sector, salary etc..) Is the impact of Doctoral Researchers in the workplace ever quantifiable or is just intangible? Measuring this is difficult but also slow! Need to have metrics that work now in order to facilitate strategic funding decisions based upon ROI. Need this if we are to have innovation and development in the sector and in researcher development.

http://ec.europa.eu/euraxess/pdf/research_policies/SGHRM_IDTP_Report_Final.pdf

Institutional Environment Supervisors Department Faculty Peers Physical Space Partners

We are all working toward better researcher development environments but how do we know what is working? Which institution is better? How can we measure it?

The Metric Tide “Peer review, despite its flaws and limitations, continues to command widespread support across disciplines. Metrics should support, not supplant expert judgement.” p.viii, 2015 Wilsdon, J., et al. (2015). The Metric Tide: Report of the Independent Review of the Role of Metrics in Research Assessment and Management. DOI: 10.13140/RG.2.1.4929.1363

REF 2014

Research Environment Overview Research Strategy People, including: Staffing strategy and staff development. Research Students (supported by limited metrics on number of doctoral awards) Income, infrastructure and facilities (supported by metrics on research income) Collaboration and contribution to the discipline. Number of doctoral students? CDT? Quite subjective, view of experts.

Metrics REF 2014 PRES 2015 Submission Rate 2010/11 3yr Submission Rate 2009/10/11/12 ‘Attractive’ environment – attractive to whom. BBSRC’s official line is that they will take ‘submission rate as a clear indicator of quality’ and both the AHRC and ESRC state that they may impose sanctions on individual Research Organisations if their submission rates fall below a certain level.

Results REF v Submission Rate r = -0.02637 REF v 3yr Submission Rate BBSRC’s official line is that they will take ‘submission rate as a clear indicator of quality’ and both the AHRC and ESRC state that they may impose sanctions on individual Research Organisations if their submission rates fall below a certain level.

Nothing here. Tempting to draw conclusions based upon the stand out correlations but given how many correlations we have performed we should expect one or two false positives. Research culture correlations are more tempting than..

……professional development correlations, in theory one’s ability to communicate information effectively to diverse audiences should be more independent of department differences because the skill is often developed outside of the department!

To Discover And Understand. g.boyce@sheffield.ac.uk @gavinboyce_ To Discover And Understand. Starting the debate – we need to drive the narrative – what can we usefully measure now that will demonstrate value for money and potential return on investment. The metrics discussed by Wilsdon are possible (Twitter mentions, publications etc..) but are of less utility perhaps for earl career researchers but the point is that we (the experts) need to agree the appropriate basket of metrics, the appropriate balance between qualitative and quantitative in order that we can evidence return on investment and allow ROI based decisions. And we need this if we are going to argue that research funding is an investment in the future and not a subsidy of the present.